support indexing -vcg-ast / resolved mixin code / compiler JSON output

[WebFreak001]

Instead of indexing the raw source files, we could use the existing dub build linting to provide more accurate indexing. This may need extended output formatting in the compiler though.

We build the project already anyway, so it's possibly a good idea to use what the compiler already computes here and not throw it away.

Should enable fancy things such as showing how mixins expand, how templates expand, more accurate find references, etc.

[Reavershark]

With this code:

class A
{
    int field1;
    mixin("int field2;");
}

class B(T)
{
    T field1;
    static if (!is(T == int))
    {
         T field2;
    }
}

A a = new A;
B!int b = new B!int;

Autocompletion for a and b (incorrectly) gives:	Autocompletion for a (mixin expansion)	Autocompletion for b (template expansion)
field2, provided by a string mixin, is missing.	field1 has type T instead of int. field2 shouldn't be shown as B!int doesn't have that field. T shouldn't be shown as that's not a property?

Would love to see this working, but it sounds like there's a long way to go.

The -X flag only seems to output public symbols. It does expand string mixins, but not (mixin) templates. \ The -vcg-ast flag outputs the source code with everything expanded except for string mixins. Line numbers are not preserved and the resulting file does not valid have valid d syntax. \ The -mixin flag outputs only expanded string mixins, with their line numbers.

[WebFreak001]

@ryuukk from #323:

I just noticed it's possible to get AST after all semantic phases with DMD

What if server-d would manage to get that output to retrieve proper type of templats?

Example: https://run.dlang.io/is/h4fjXB (click on AST)

struct MyTemplate(T, V)
{
    T t;
    V v;
}

void main()
{
    MyTemplate!(int, float) test;
}

Will generate:

MyTemplate!(int, float)
{
    struct MyTemplate
    {
        int t;
        float v;
    }

}

So we lookup by the definition MyTemplate!(int, float) and we get a properly constructed type!

Problems: only available after compilation

So the code should be compilable (not usable while editing your code)

Depending on projects, compilation can be slow so this is not a good real-time feature

Whenever a user compile its project, hook the command, add the thing to get the AST ouput, and cache the result

So whenever we generate the auto completion result, we can use this cached data to construct proper types, this way we can turn this into a real-time feature!

The compiler arg is: -vcg-ast

It will create a file name next to the module file: mymodule.d.cg, so this is the file that needs to get parsed

I wonder if it can be made for DMD to output the content to stdout instead of a new file

It seems pretty fast, on my game project, it only add 50miliseconds to compile time

[Reavershark]

On closer inspection, -vcg-ast seems to do everything that would be needed, even mixin evaluation:

https://run.dlang.io/is/tzjaUS

struct SomeStructTemplate(T, V)
{
    T t;
    V v;
}

void someFunctionTemplate(T)(){}

mixin template SomeMixinTemplate()
{
    int i = 1;
}

alias inst1 = SomeStructTemplate!(int, float); // Works
alias inst2 = someFunctionTemplate!(int); // Works
alias inst3 = mixin("someFunctionTemplate!(float)"); // Works
mixin("alias inst4 = someFunctionTemplate!(bool);"); // Mixin string is kept, but template is instantiated!
// alias inst5 = SomeMixinTemplate!(); // Can't do this
mixin SomeMixinTemplate!(); // Works (template with no args and its instantiation are distinguishable)

We could apply this to every module in a project + its import paths and store the info in some cache. Dependencies should compile fine, but some modules that are actively worked on might not. To solve that we could:

1. Keep using older info in cache (Maybe serve-d does this by default?).
2. Maybe try to dissect the file, removing erroneous code and trying to evaluate template instantiations then, but that sounds very difficult (but maybe serve-d already does something similar somewhere?). This should be doable for imported templates at least.

A way I can think of implementing this is by optionally sending information about template instantiations etc look to dcd? Maybe using dmd-as-a-library should be used? (Calling half of main, then doing some analysis passes and exiting) I don't know how automatic compilation is happening at the moment. Then again the user might want to compile with specific dub settings applied such as import paths, compiler flags...

-vcg-ast is implemented in dmd here: https://github.com/dlang/dmd/blob/a0d3c94edfed7ad5de6c6deaf18be67170e89e84/compiler/src/dmd/mars.d#L545