Proposal: Automatically deduce imported file locations

[nex3]

Right now a user of the dev compiler needs to explicitly pass every single file to be included a module on the command line. This is a big UX drain; it makes compiling modules by hand very painful.

I propose that instead the files passed on the command line just be considered "seeds" that define the root nodes of a tree of files to include. All files transitively imported, exported, or parted by the seeds are included in the module, unless they're already in a summary file that's passed to the dev compiler.

This will also make it easier to work with packages that include some files that use dart:io and some that do not. Right now even an automated tool has to painstakingly deduce which files are actually reachable from the code it cares about compiling, in order to avoid accidentally trying to compile a file that imports dart:io. If the dev compiler followed imports on its own, this wouldn't be necessary.

[jmesserly]

Question on this. Would we want to generate a single output file or per-package like @vsmenon 's global compile script does, or perhaps disallow reaching outside of the current package? (I think Vijay hit some issues where passing too many files started to stress the compiler, especially if they cross package boundaries.)

[jmesserly]

BTW overall I really like this idea 👍

[nex3]

I think it should just be up to the user how many files get pulled in. Artificial boundaries like that will just make things more confusing.

I also think that you should expect users to want to compile large chunks at once, especially before we have a thorough story in place for a unified build system that tools can tap into. The test runner, for example, wants to compile all the code in lib/ and in all packages in a single go. It would really suck if we had to split that up into separate compilations to work around implementation constraints in the compiler.

[jmesserly]

yeah, maybe worth another chat about the test runner... I'm a little worried if we try and compile essentially the whole program... DDC will end up with worse performance than dart2js =( (not because DDC codegen is all that slow, but because Analyzer is slower than dart2js front end). To have a responsive, great feeling user-experience we depend on being able to reuse compiled output for code that isn't frequently changed. That's a large benefit to developers who don't have to wait for the system to recompile code they already compiled.

e.g. if the user scenarios are like "tweak lib/ stuff and see if it affects tests" and "tweak tests as developing them" then the ideal modules are:

• all package dependencies
• lib/
• test/

another scenario might be editing a package that is a dependency_override. That would be another good candidate for its own module.

grouping all other packages together makes sense though. They should only change when "pub upgrade" is invoked so presumably, not terribly often.

[jmesserly]

if it would help, we could do a lightweight build system that checks if pubspec.lock is in sync with the last "packages.js" build and if not, rebuild it. That might get us 80% there for 5% work :)

[nex3]

Until we have some sort of build system to hook into, I don't think it makes sense do any sort of sharing across different test runs. As you suggest, we could introduce something ad hoc for test specifically, but there's not a good place to put it and I don't like the idea of adding a new potentially-conflicting file that requires its own .gitignore without figuring out how it'll integrate into a broader build system.

So for the short term, I'm only thinking about sharing modules within the scope of a single pub run test invocation, where it makes sense to compile all the code that's not specific to an individual test file at once. Hopefully when the cost of the monolithic compilation is amortized over a large number of entrypoints this will be comparable to dart2js.

[jmesserly]

Hopefully when the cost of the monolithic compilation is amortized over a large number of entrypoints this will be comparable to dart2js.

Oh, nice! so we're still able to get some benefit from it if you have a lot of tests. That's nice.

[jmesserly]

I'm also removing the "part" validation (it was picky about supplying all of the parts). it doesn't seem worth it.

[jmesserly]

I think I've got the core of this implemented. Right now I'm trying to disentangle our test build script (test/codegen_test.dart) which is not building the packages that tests depend on properly.

[jmesserly]

https://codereview.chromium.org/2234343003/