Open srawlins opened 4 months ago
It's quite possible that some of the base classes might include support that isn't needed by the tests that a plugin author would need to write. It might be worthwhile to start by thinking about what the requirements are and what those pieces of support actually depend on. Moving the rest of the support out to mixins might reduce the number of dependencies.
It's also the case that plugin authors will need to be able to do some things that we currently don't have support for.
For example, here are some initial thoughts.
Plugin authors need to be able to test both lints and fixes.
For both of those use cases the tests need to be able to create the code being tested and any supporting code in an in-memory disk layout. That includes being able to create analysis options files, package config files, and any (mock) packages that the test code needs to be able to depend on (including packages only available to the plugin author).
For lint tests they need to be able to specify which file(s) to analyze and the diagnostics that the lint is expected to produce. They'll also need to be able to register the lint rule being tested.
For fix tests they need to do every they need for testing lint rules as well as confirming the result of applying the fix to the appropriate files.
They probably don't need the printers that we use for printing resolution output, the findNode
and findElement
utilities, and a reasonable portion of the assert
methods, among other things.
Also, are you thinking of analyzer_utilities
as a stepping stone to a published package of plugin test utilities, or are you thinking that we'd eventually publish analyzer_utilities
? I'd prefer the former, but you might know of a reason why that isn't the best option.
Also, are you thinking of
analyzer_utilities
as a stepping stone to a published package of plugin test utilities, or are you thinking that we'd eventually publishanalyzer_utilities
?
Either. I hadn't thought of the differences.
Something that may make this task difficult is that much of ContextResolutionTest (and others I mentioned) depends on package:analyzer. It uses MemoryByteStore
, AnalysisContextCollection
, MockSdkLibrary
, Folder
, etc. I can't remember the cyclic dependency rules with certainty. But I think that two packages cannot depend on each other (directly or indirectly) through their dependencies
. Two packages can only depend on each other if at least one of the dependencies is via dev_dependencies
. And I think that also means that at least one of the dependency chains must involve a dev_dependency
instead of a dependency
, yes?
Anyways, I think that means that analyzer_utilities
can have a dependency on analyzer
, but then analyzer
cannot have a dependency on analyzer_utilities
, meaning that no code in pkg/analyzer/lib/src/test_utilities
can depend on package:analyzer_utilities
. That code would have to move first.
Another anecdote pointing to the need to solve this:
I recently moved selection.dart
from analysis_server
to analysis_server_plugin
. @bwilkerson pointed out I should move the test as well, selection_test.dart
. When I tried doing this, I saw that this would require moving text_expectations.dart
as well (or copying it). But text_expectations.dart
(TextExpectationsCollector
) is required by half a dozen other tests in analysis_server
. So I could copy that over to analysis_server_plugin
, in it's lib/
directory (something like lib/src/test_utilities
) but then the analysis_server_plugin
package would need a dependency (not a dev_dependency) on both package:test
and packag:test_reflective_loader
. ☹️ I think this is super smelly and not adhering to the purpose of the analysis_server_plugin
package.
So I think we need an analysis_server_plugin_testing
(or analyzer_testing
or pick your better name) package. Above, @bwilkerson wrote:
Also, are you thinking of
analyzer_utilities
as a stepping stone to a published package of plugin test utilities, or are you thinking that we'd eventually publishanalyzer_utilities
? I'd prefer the former [...]
I wrote, "Either. I hadn't thought of the differences." Now I'm thinking a fresh, test-oriented package (like, the name is also test-oriented) is warranted. We wouldn't need to publish this package for a while. Maybe not until plugin authors need it.
Quick background / example: I would like to move
ResolveNameInScopeTest
from the analyzer package to the linter package. But it depends onAbstractLinterContextTest
(also inpkg/analyzer/test
) which depends on PubPackageResolutionTest (also inpkg/analyzer/test
). So I cannot move any tests until we have some shared base classes. Or I can copy swaths of code.Creating shared testing code between the analyzer package, linter package, analysis_server and analysis_server_plugin packages is necessary for the plugins story. CC @bwilkerson @scheglov @pq
ContextResolutionTest
It would take significant investigation to determine how many test utility classes and mixins would be prudent to move. But as a starting place, I think it would be of huge benefit to move
ContextResolutionTest
intopackage:analyzer_utilities
. How do we get there?ResourceProviderMixin
ResourceProviderMixin
(mixed intoContextResolutionTest
) is declared inpackage:analyzer/src/test_utilities
, so we're already in a good place there. Should be easier to move than other things, as it cannot depend on code inpkg/analyzer/test
. This might be the best starting place.ResolutionTest
ContextResolutionTest
also mixes inResolutionTest
, so this is also a pre-requirement. Unfortunately it lives inpkg/analyzer/test
, and is in a file which imports eight other files frompkg/analyzer/test
. Yikes.PubPackageResolutionTest
PubPackageResolutionTest
is also an important base class, as it is the base for supporting analysis options files and package configs.