Test vector generation from scenario tests

[ZenGround0]

Generate test vectors from scenario tests

• [x] 1. Introduce a test vector type that can generate valid test vectors conforming to the schema
• [x] 2. Build test vectors inside test vm ApplyMessage calls
• [x] 3. Serialize and write test vectors to file behind an environment variable flag
• [x] 4. Verify that test vectors correctly validate against lotus implementation

[anorth]

Whoah, hold up. I thought this was about adding some simple golden file approaches into our scenario tests, as recommended by the Jan 2021 post-mortem. At no point until now did I realise that anyone was talking about the full machinery of the https://github.com/filecoin-project/test-vectors project. That is a complex mechanism that was designed in response to different requirements and constraints, and is aimed at testing a different thing (the VM). There might be some advantage to unification, but IMO that would require its own proposal and analysis, because it will have costs.

I'll admit to a certain amount of not-invented-here syndrome in my reaction here (as also influenced the creation of the test-vectors project after the chain-validation project) but I think we should aim to add a minimal golden-file mechanism to our existing scenario test set-up.

• The test-vectors project is complex, especially in integration, and as a result of this is in disuse since November 2020
• We already have basic golden files in use for a few specific tests around reward calculation
• We can't depend on the test-vectors project from here, so maintaining compatibility (even if the test-vectors project gets a new injection of effort – which is not planned) will involve either extensive cross-repo refactoring to re-arrange dependencies, or ongoing manual effort to simulate that with parallel changes. It's big friction, and I'm seeing the most likely outcome as more technical debt

[anorth]

@ZenGround0 and I talked about this and we're aligned on work continuing to leverage the format defined by the test vectors project (which will be replicated here, rather than depended on, for now). It's relatively easy for us to produce conformance vectors in addition to the golden files this way, making the the project useful to other implementations.

This would be a step towards a longer-term plan of reviving test-vectors by moving much of the generation into the specs-actors project itself. This project now has a VM which, with only a little more development, could serve as a reference VM and solve the ugly dependency problems that have left the test-vectors project behind.