Bring --update-goldens from Flutter to pure Dart

[alexeyinkin]

In Flutter, there is a concept of golden tests and the following support for it:

• flutter_test package has autoUpdateGoldenFiles global variable that indicates whether all actual values should be written as a new expectations.
• flutter test command supports --update-goldens flag to set that variable to true.
• flutter_test package has MatchesGoldenFile matcher that checks this variable and either expects the golden value or overwrites it.

All of that is useful in pure Dart as well.

One case I came across was testing code generators. The standard way for that is source_gen_test package. It currently requires to inline the expected generated code, and I have prepared an update that allows that to be extracted to golden files. Unfortunately, in pure Dart there is no way to inform the test that it needs to overwrite goldens, so I am considering an ugly workaround with an environment variable:

SOURCE_GEN_TEST_UPDATE_GOLDENS=1 dart test my_test.dart

It would be much cleaner to write:

dart test --update-goldens my_test.dart

Other uses of this feature include testing all kinds of serializers, format converters, etc.

[jakemac53]

My general thoughts:

• I think there is a general need here for command line configuration, we have had a similar request before as well. I don't know exactly what this would look like but I would much rather support this in a general purpose way.
• The environment variable is an OK workaround for now. It can only be read from VM tests, but only VM tests can touch the file system so that is kind of a good thing. So we don't have to rush into a solution, given a workaround is available.
• Unless we move all the support for goldens testing into this package, I don't think it makes sense to support a flag like this.
• We could possibly explore implementing all the goldens stuff here, but I would rather see it in a separate package (package:goldens_test or similar?).

[natebosch]

• We could possibly explore implementing all the goldens stuff here, but I would rather see it in a separate package (package:goldens_test or similar?).

Yes, this is what I would prefer.

• The environment variable is an OK workaround for now. It can only be read from VM tests, but only VM tests can touch the file system so that is kind of a good thing. So we don't have to rush into a solution, given a workaround is available.

+1 - I agree that a better API for passing in arguments to main or something along those lines would be cleaner, and would be nice to have in general. Even with that feature I think I'd expect the implementation to live in a separate package, and I don't think that feature is going to be high enough priority to get above the line anytime soon, so an environment variable seems like good tradeoff.