Support multi-option contexts

pq commented 10 months ago

The Dart Analyzer treats the presence of an analysis_options.yaml file as a signal to create an analysis context, a discrete, hermetic environment for analysis. In many cases this is unneeded and the memory overhead of elements that could be shared but are duplicated is significant, contributing to poor IDE performance.

Plan: Decouple options from contexts, allowing one context to contain many analysis options files without any element model and file state duplication. To the user, there will be no change in semantics with reduced memory pressure and improved performance.

Follow-ups:

[x] #56352

Blocking:

[x] https://github.com/dart-lang/sdk/issues/55580
[x] https://github.com/dart-lang/sdk/issues/55413
[x] #55300
[x] https://github.com/dart-lang/sdk/issues/55252
[x] #54858
[x] https://github.com/dart-lang/sdk/issues/54849
[x] https://github.com/dart-lang/sdk/issues/54816
[x] #54714
[x] #54814
[x] #54791
[x] #54770
[x] #54312
[x] https://github.com/dart-lang/sdk/issues/53873
- [x] deprecate uses (and update callers)
- [x] remove API
[x] https://github.com/dart-lang/sdk/issues/54376
[x] https://github.com/dart-lang/sdk/issues/54043
[x] https://github.com/dart-lang/sdk/issues/54074
[x] #54124
[x] #54310

Multi-Option Contexts (future)

Single-Option Contexts (today)

It's interesting to note the less than dramatic difference in memory usage.

Likely this can be at least partly explained by the nature of the saved contexts (in all 3 cases they have very few libraries) but the fact that this is essentially a cold-start capture makes these numbers imprecise anyway. Much more interesting will be to measure memory footprint over time (and with more substantial nested contexts).

Testing

For the curious, to enable mutli-option contexts in an SDK, set singleOptionContexts to false

and do a build.

rrousselGit commented 6 months ago

I think the .analysisOption may have been deprecated a bit early.

If .getAnalysisOptionsForFile(file) is still experimental, we don't have a stable replacement for .analysisOptions

pq commented 6 months ago

Sorry for the confusion @rrousselGit. This is taking a bit longer than hoped (lots of moving parts).

Out of curiosity, how are you using analysis options objects? In plugins?

rrousselGit commented 6 months ago

In analyzer_plugins, yes. In custom_lint specifically, a wrapper around analyzer_plugin to make it more dev friendly and improve support for monorepo in terms of memory usage.

pq commented 6 months ago

Thanks! If it isn't too much trouble, could you tell me how you're using analysis options in custom_lint?

(fyi @srawlins.)

srawlins commented 5 months ago

Any updates on this effort, @pq , as a P1 feature?

pq commented 5 months ago

Thanks for the ping. This is currently blocked behind #54858. Will be revisiting that in the next week.

pq commented 5 months ago

Investigating current blocked (https://github.com/dart-lang/sdk/issues/55252) now.

pq commented 5 months ago

Fix for #55252 cleared up the pkg bots 🎉. Waiting on flutter try results. 🤞

knopp commented 5 months ago

I switched the singleOptionContexts flag to false, rebuilt the SDK, but I get same amount of context for our mono-repo project. Is this supposed to make any difference in monorepo projects?

pq commented 5 months ago

Is this supposed to make any difference in monorepo projects?

@keertip

pq commented 5 months ago

The singleOptionContexts flag will only make contexts due to nested analysis options files unnecessary so it's sort of orthogonal to monorepos.

For example, with singleOptionContexts set to true the following will produce two contexts

my_package/
  analysis_options.yaml
  lib/
  test/
    analysis_options.yaml

With it set to false, you should see only one.

If you don't have any nested options files, you won't see a difference.

bwilkerson commented 5 months ago

On the other hand, it's a necessary part of providing good monorepo support. Without this, every package in a monorepo would likely still have it's own analysis context, which would largely defeat the gains we expect to see from the monorepo support.

knopp commented 5 months ago

Thank you for the confirmation. I'll patiently wait for pub workspace support.

bwilkerson commented 4 months ago

Any updates?

pq commented 4 months ago

We're waiting for https://dart-review.googlesource.com/c/sdk/+/362442 to fully make it through a flutter roll before we can try to re-enable this. (See: https://github.com/dart-lang/sdk/issues/55413)

pq commented 4 months ago

https://github.com/dart-lang/sdk/commit/22da6ed1ccc84fa5fb31433ba4465542733195fe flips the bit but we've reverted related changes twice so I think we want to let it bake for a few more days at least before declaring victory.

pq commented 4 months ago

Looking at the latest engine roll (https://github.com/flutter/flutter/pull/147373) which brings Dart in @https://github.com/dart-lang/sdk/commit/b5f51d8868191b5358ccb11f07e77bf3fd3bd09b, it looks like this change has safely made it's way into Flutter. Here's hoping this landing sticks. 🤞

pq commented 3 months ago

With https://github.com/dart-lang/sdk/issues/55580 fixed, all known issues have been addressed.

dart-lang / sdk