How do we deal with macros?

[nolanderc]

Macros can appear almost anywhere. For example, in tests/glsl-samples/well-formed/glslang/tokenPaste.vert we have the following line:

float bothpaste(foo, 719);

At first this looks like a function definition, but it is actually macro expansion with:

#define bothpaste(a, b) a##b

The current implementation ignores macro expansion entirely, and just treats it as if it were normal code. This keeps both the tokenizer and parser simple, but has the added benefit of keeping the parse tree a 1:1 correspondence with the source code.

To properly implement macros we would ideally want to include both the pre-expanded tokens, as well as the expanded tokens in the same tree. One possible approach here is to add a special .macro syntax node, which contains the tokens before expansion (used in formatting and hover info, but ignored everywhere else), and then have the expanded tokens in the tree as usual, but with 0-width token spans pointing to the first character of the macro. Spans being 0-width ensures that tokens cannot be hovered, but still have a position for goto-definition.

The parse tree for the above snippet would then look something like the following:

file
  declaration
    identifier "float"@0..5
    macro
      identifier "bothpaste"@6..15
      ( @15..16
      identifier "foo" @16..19
      , @19..20
      number @21..24
      ) @24..25
    identifier "foo123"@6..6
    ; @25..26

[automaticp]

I'll try to contribute more to the discussion here later as to how to try to tackle this, but for now I also want to bring up some other (arguably even more problematic) cases. It's fairly common to see a pattern similar to:

#ifndef SOME_VALUE
#define SOME_VALUE 12
#endif

Which gives a default value to SOME_VALUE that can be overriden by pasting, ex. #define SOME_VALUE 16 at the top of the file in a custom preprocessing step defined by the user in their app before compilation of the shader.

Another one is conditional compilation:

#if ENABLE_FEATURE
uniform float conditionally_present;
#endif

// ...

void main() {
#if ENABLE_FEATURE
    frag_color = vec4(conditionally_present, 1.0);
#else 
    frag_color = vec4(1.0);
#endif
}

And it's nightmarish friends:

#if ENABLE_FEATURE
uniform float parameter;
#else
const vec4 parameter = vec4(1.0); // same name
#endif

#if ENABLE_FEATURE
#define NIGHTMARE(x) (x + x)
#else
#define NIGHTMARE(x) (x - x)
#endif

#if ENABLE_FEATURE
#define NIGHTMARE_WITHIN(x) (x * x)
#endif

#if ENABLE_FEATURE
#define NIGHTMARE(x) NIGHTMARE_WITHIN(x + x)
#else
#define NIGHTMARE(x) (x / x)
#endif

And the "best part" is that ENABLE_FEATURE is probably not even defined anywhere in the file, because it's supposed to be inserted pre-compilation by the application, or if it is defined, then it's probably guarded like the first example, again, with the option to override.

Preprocessor is a stateful machine and we don't even know all the initial state, as opposed to, say, C or C++ where these flags are at least a part of a build description, which helps the tooling there. We probably need to think what extent of this macro-madness we intend to support at all.

[nolanderc]

It’s a thorny problem all around, and no obvious solutions.

In my mind, a local optimum is to just give the user as many completions as possible by assume both branches of every #if and #else is taken. That way, at least the user isn’t frustrayed about missing completions.

Another alternative is to define our custom macro, which allows the user to set some defaults:

#ifdef __GLSL_ANALYZER__
#define ENABLE_FEATURE
#endif

But I’m not sold on that idea: it’s a bunch of extra complexity, and clutters files with glsl_analyzer specifix details, for not that much gain.

I would rather just say that we don’t handle all kinds of exotic preprocessor usecases, but at least give a good user experience for the most common.

[mkoncek]

clangd (the LSP server for C and C++) reads a file named compile_flags.txt (https://clangd.llvm.org/design/compile-commands) placed in the root of the project. Those can contain additional include paths, setting the standard version define macros, anything that you can pass to your compiler on the CLI. Rust analyzer is getting a similar approach from what I have heard. Maybe you can do the same? Read a file like .glsl_analyzer.txt.