Define a way for code generators to indicate the origin of generated code

[bwilkerson]

When a user sees a reference to generated code it would be useful for them to be able to navigate to the location that caused the code to be generated. It might be more useful than being able to navigate to the generated code, which is what server currently supports. The problem is that server doesn't have any way to know what the code generator used when it decides to generate the code.

Consider adding a way for code generators to inform the analysis server of the location on which generated code is based and support to the analysis server that would use this information for navigation.

One possible way to do this would be to introduce an annotation:

@BasedOn('package:my_package/file.dart', 'MyClass.property')
Object someGeneratedProperty;

One downside of this is that the annotation would need to be defined somewhere, and that would potentially require packages to add a dependency that they otherwise wouldn't need. The meta package is fairly widely used, so it might be a good candidate for where to add such an annotation if we decide to go that way.

An alternative would be to define some kind of comment syntax, but that's likely to be more error-prone.

(This issue is a follow-up to a thread in the analyzer-discuss group.)

[bwilkerson]

@davidmorgan @jakemac53 @natebosch @grouma @matanlurey

[simolus3]

I'm also interested in such feature, but I prefer the more general way that would just specify an offset and length in a file. I'm generating Dart sources based on content in a non-Dart file, so I don't have a name available.

Maybe we can have multiple options, with one of them being a Dart member reference? When providing raw offsets that update quickly, the annotation could provide information about a matching analyzer plugin. The plugin would then provide real-time updates when the source changes, so something like

@RedirectNavigation.withPlugin('plugin_name', 'some_identifier', defaultLocation: ...)
Object somePropertyWithPlugin; // can provide updates on edits, useful when not targeting a Dart file
@RedirectNavigation.toDart('uri', 'Class.member')
Object anotherProperty; // for code generators without a plugin, will be good enough

The plugin could then send a map of identifiers to source locations via a notification. The analysis server could use that to patch the target in real time. For code-generators that don't have an analyzer plugin, users would still get acceptable results if they re-run their generator often enough.

[jakemac53]

Note that many code generators cannot add dependencies at all because they generate to part files.

[rrousselGit]

Note that many code generators cannot add dependencies at all because they generate to part files.

They usually have an annotation package though.

So their annotation package can do:

export 'package:meta/meta.dart';

class MyAnnotation {
...
}

[jakemac53]

Exporting other packages is highly discouraged. Your api is now effectively unioned with the package:meta api and you are thus tightly coupled to it. To be technically correct you would have to pin an exact version dependency on meta and copy its changelog etc, and publish in lock step with it.

Even if you wanted to do that if users use show or any import prefixes etc you can also easily get broken.

[bwilkerson]

Yeah, there are definitely issues with using an annotation. Are there better ways to approach the problem?

[rrousselGit]

A comment using a specific format maybe?

/** @Navigation(...) */
int value;

[jakemac53]

Most of the typical drawbacks of a comment wouldn't apply to code generators (they don't need code completion or have to worry so much about typos etc). So a comment approach might be reasonable.

[lrhn]

This sounds like something that should be handled using source map files. As I understand it (big caveat) a source map explicitly links code in one file to code in another file (not necessarily in the same language, but it can be). There is a comment format for specifying the source-map file in the generated file.

[natebosch]

Using source maps would require the codegen author to track offsets into the output file as it is generating it. We could add some infrastructure around that to try to mitigate the cost for codegen authors, but I believe the overall complexity of that system would be higher than writing into the output file directly.

Something like code_builder could probably have such infrastructure. I'm not sure how much of the problem that solves - there is a good amount of codegen happening with string concatenation. In the source_gen scenario the codegen author can't know the offset into the final file since the output from multiple builders gets combined.

[jakemac53]

Note that source_maps would also have bad invalidation semantics if they were ever imported from other build steps during the build. If they are only needed for analysis_server that might not be an issue though.

[lrhn]

My point is the source maps is an existing standard for solving this exact problem, and it's already supported by (some) IDEs and debugging tools. Let's not invent a new custom format to solve the same problem. If code generators need help generating source maps, then by all means let's help them with that.

Even if it turns out it only works well when compiling to JavaScript, we are compiling to JavaScript, and we are already generating source maps when doing so. Making those source maps link back to the original source, not the generated source, would still be convenient.

[natebosch]

One drawback to any scheme which maps back to an offset instead of a higher level concept is that it won't work if anyone is generating code which would want to map back to a file in another package. I don't know if this use case comes up. It wouldn't make sense to publish something to pub pointing to an offset in a dependency, whereas pointing to a class in a package or import would be stable across non-breaking versions.

[rrousselGit]

Writing source maps is definitely too complex for me.\ Generating the source map would be more difficult than making the code-generator itself.

Another problem is that most code-generators relies on source_gen's SharedPartBuilder.\ This means that multiple code-generators may output to the same Dart file – which makes handling offsets relatively difficult.