Help with toplevel completion testing

[Simn]

Hi,

I'm working on toplevel completion in the Haxe compiler, but it's really awkward to test without IDE support. Could someone add a quick hack which allows me to press some key combination and make sublime request --display file@pos@toplevel with the current cursor position?

CC @frabbit

[frabbit]

so instead of --display file@0 it should use --display file@0@toplevel?

[clemos]

Hey Simon, I can try to do it on the master branch by sunday. Best, Clément

[Simn]

Exactly!

[frabbit]

you can hack this line:

https://github.com/clemos/haxe-sublime-bundle/blob/master/HaxeComplete.py#L1238 in your local copy and replace it with:

args.append( ("--display", display["filename"] + "@" + str( display["offset"] ) + "@toplevel" ) )

[Simn]

Okay, but how do I invoke that?

[frabbit]

that is just a hack which always uses @toplevel for regular completion.

[Simn]

I understand that, but how do I actually cause run_haxe to be executed? I don't have field. or anything to trigger it on.

[frabbit]

ah ok, i guess that's not such a simple hack

[frabbit]

i can take a look at it this evening.

[Simn]

Nice, thanks!

I've pushed my work in progress here in case you want to test with it: https://github.com/Simn/haxe/tree/toplevel-completion

[clemos]

I've actually been able to push a quick hack on this branch : https://github.com/clemos/haxe-sublime-bundle/tree/toplevel Currently, top level completion is handled "internally" by the bundle, when it detects no "autocompletion" character before the caret (neither of .,(). In this situation, on this branch, we attempt to run haxe with @toplevel, and fallback to internal toplevel completion if no completion is returned (including if there are errors). I've left the haxe command line and haxe output in the console so you can see what's going on under the hood. Please notice that in case of toplevel completion, @display is called at the exact caret position, while in other cases, it's called at the preceding . or ( position. Let me know if I can help further. I really look forward to have compiler based top level completion :D Best, Clément

[Simn]

Ah nice, thanks a lot! I'll check it out.

[Simn]

Alright, I got it working. Thanks a lot, this makes testing it much easier.

Do you have any preferences regarding output format? Right now I'm generating something like this:

 <il>
<i k="static" t="Void -&gt; Unknown&lt;0&gt;">main</i>
<i k="static" t="s : String -&gt; Int">f</i>
<i k="enum" t="E">A</i>
<i k="enum" t="s : String -&gt; E">B</i>
<i k="global" t="haxe.Template">expr_float</i>
<i k="global" t="haxe.Template">expr_int</i>
<i k="global" t="haxe.Template">expr_splitter</i>
<i k="global" t="haxe.Template">expr_trim</i>
<i k="global" t="haxe.Template">globals</i>
<i k="global" t="haxe.Template">splitter</i>
</il>

The idea is that k denotes the kind of the identifier, which is current among "local", "member", "static", "enum" and "global". t represents the type of the identifier for those where that makes sense.

[clemos]

Awesome ! I guess it's fine as-is. t should already be visible, not sure if / how to display k. It would be perfect if the compiler could figure out what kind of completion is needed depending on the context (including finding ., (, etc. ). It would save a lot of dirty parsing / guessing from the IDE. Also, are things like "Jump to declaration", "Find occurences", etc planned ? This would absolutely rock ;) Best, Clément

[Simn]

I have the logic for display modes "usage" (aka find references) and "position" (aka go to declaration) implemented. They were having the same issues as toplevel completion in that I didn't know how to handle our parser side of things, but it should be possible to add them the same way we're adding toplevel now.

[clemos]

Yeah it seems I've been testing it quickly at the time : https://github.com/clemos/haxe-sublime-bundle/commit/d211d346c0e6016841a270dfc81ca542c218be73 Don't remember how it went, though. Let me know, I'd be glad to test all of this on my end :) Best

[clemos]

BTW, pleased to meet you @toplevel :D

[Simn]

I made some progress on the branch, I think it's ready for first tests now. In my messy test project I get an output like this:

<il>
<i k="local" t="Dynamic">e</i>
<i k="member" t="Int">x</i>
<i k="static" t="Void -&gt; Void">main</i>
<i k="package">cpp<i>
<i k="package">sys<i>
<i k="package">haxe<i>
<i k="package">hxargs<i>
<i k="package">as3<i>
<i k="type" p="UInt">UInt</i>
<i k="type" p="StringTools">StringTools</i>
<i k="type" p="String">String</i>
<i k="type" p="Math">Math</i>
<i k="type" p="Map">Map</i>
<i k="type" p="List">List</i>
<i k="type" p="Lambda">Lambda</i>
<i k="type" p="IntIterator">IntIterator</i>
<i k="type" p="EnumValue">EnumValue</i>
<i k="type" p="Enum">Enum</i>
<i k="type" p="DateTools">DateTools</i>
<i k="type" p="Class">Class</i>
<i k="type" p="Array">Array</i>
<i k="type" p="Xml">Xml</i>
<i k="type" p="Type">Type</i>
<i k="type" p="Sys">Sys</i>
<i k="type" p="StringBuf">StringBuf</i>
<i k="type" p="Std">Std</i>
<i k="type" p="Reflect">Reflect</i>
<i k="type" p="EReg">EReg</i>
<i k="type" p="Date">Date</i>
<i k="type" p="State">State</i>
<i k="type" p="LexEngine">LexEngine</i>
<i k="type" p="Config">Config</i>
<i k="type" p="Bug">Bug</i>
<i k="type" p="ArrayAccess">ArrayAccess</i>
<i k="type" p="Iterable">Iterable</i>
<i k="type" p="Iterator">Iterator</i>
<i k="type" p="Dynamic">Dynamic</i>
<i k="type" p="Bool">Bool</i>
<i k="type" p="Null">Null</i>
<i k="type" p="Int">Int</i>
<i k="type" p="Float">Float</i>
<i k="type" p="Void">Void</i>
<i k="type" p="Main">Main</i>
</il>

As a first implementation the IDE can simply grab the node values and display them. My idea regarding the different kinds is to use them for sorting or to display additional information, but that's not very important.

Note that all this only works inside of blocks, which is a restriction that I don't think we'll be able to overcome.

[elsassph]

The k attribute is a fantastic addition - I for one was really waiting for it, although honestly it will break FD completion: you might want to enable this additional info with a define.

[clemos]

I guess I'll just merge my changes to master, since it still works and will fall back to internal toplevel completion until the compiler supports @toplevel. When do you plan to merge it to haxe's master branch so we can call testers ?

[Simn]

This only affects the new toplevel completion, so it doesn't break anything. Where else would you like to have it? Hiding it behind a define sounds like a good idea for other cases.

[clemos]

I'm not sure it's a good idea, but it would be ideal to have deeper class discovery, ie to also have direct access to all available classes ( haxe.Timer, js.JQuery, etc ). Current top level completion shows these classes and it's very convenient although once again somehow tricky to get at the IDE level. Now it depends if this is feasible / fast / efficient to do it at the compiler level.

[Simn]

I don't see how that information really helps because you're gonna get a LOT of classes and have to filter them anyway. It then seems more sensible to only show the toplevel types and use package completion as usual.

We shouldn't do any recursive file system checks because that's gonna slow down completion quite a lot for big projects. It's better if the IDE handles that because it can easily cache information.

[elsassph]

Getting all the project types should be another (quite useful) compiler service - the IDE would be able to cache these results and decide when it's appropriate to update them.

I'm really hoping to get the new k/t/p attributes in the regular completion too. Even more than for toplevel. Pretty please.

[Simn]

I still don't understand for which kind of completion you want that information. Fields already transmit their type, don't they?

[clemos]

You're right: getting all project types would rather be another compiler service. What is p exactly ? Is it the full path ? As for regular . autocompletion, AFAIK it does currently contain t (like String, etc), but not k (like member, enum, type) or p. I think that's what @elsassph means. doc (or d) would also be interresting in toplevel completion, when applicable.

[Simn]

Alright, let's design a proper output for the different completion kinds (fields, toplevel, position, usage, types) and hide that behind a -D completion-api=2 flag to not break any IDEs.

I'll write a suggestion after lunch.

By the way @elsassph, we have an open issue for overload completion which claims to be blocked on FD not supporting "multiple displays when displaying method arguments": https://github.com/HaxeFoundation/haxe/issues/560

Can we resolve that in the process?

[elsassph]

Yes, k/p are missing and would be valuable information in regular completion too. Let's say that toplevel and regular completion should have the same features.

As for overloading that's a current limitation of FD completion and UI: ie. resolving a method/member always returns the first match and not a list of matches. Likewise the UI isn't designed to allow you to switch between different method signatures. We should have a github issue to track this feature requirement.

[Simn]

Here's my first suggestion: https://github.com/HaxeFoundation/haxe/wiki/Completion-API-2.0-proposal

[frabbit]

one thing i would really love to have a nice solution for is aliased typedefs. Because most of the time i have to look at the source code whenever the ide provides just the name of a typedef as a completion hint.

example:

typedef Point = { x : Int, y : Int } 

var x : Point;

for the completion of x i would love to get information about how Point is really defined (ft = followed type) something like this would be nice:

<list>
    <i k="[kind]" t="Point" ft="{ x : Int, y : Int }" p="[position]">[name]</i>
</list>

ideally this would also work for nested aliases like Array<Point> or Void->Point etc.

[elsassph]

I love the spec. Also for FlashDevelop I'm attached to getting the doc, although this is most important for method calltips.

@frabbit problem of typedefs is that signatures can be several thousands of characters so it's not practical. Cf. html API.

[Simn]

I've just pushed the current work in progress to the development branch. Here's an archive with examples for the different modes that show the subtle differences: http://simn.de/haxe/display_test.zip

I didn't make any changes to the normal field completion yet because that's a bit tricky.

[elsassph]

Looks amazing Simon.