check in generated code as end-to-end regresison test

[izgzhen]

with this, we can track precisely what has changed because of what. Also, we can throw anyone curious about the output from Cozy a simple link to GitHub

[Calvin-L]

I think this is too fragile to be useful.

• In simple mode, Cozy's output is deterministic. However, the output is also very bad, so there isn't much point in sharing it.
• With full synthesis, Cozy's output is nondeterministic. In particular, it will often choose different names for variables.

Right now the test_examples.sh script uses simple mode, so this works. However, this will stop working if we ever want to test synthesis.

[izgzhen]

• In simple mode, Cozy's output is deterministic. However, the output is also very bad, so there isn't much point in sharing it.

It is a good point. But the diff of trivial output can be useful to understand what has changed within Cozy.

With full synthesis, Cozy's output is nondeterministic. In particular, it will often choose different names for variables.

I think it is possible to make Cozy deterministic with a simple change? For example, we can assign a maximum step size for the main improvement loop to avoid using imprecise timeout. Also, variable can be made deterministic by choosing a different generator.

Making it deterministic is also useful for day-to-day debugging/reproduction.

[izgzhen]

In simple mode, Cozy's output is deterministic. However, the output is also very bad, so there isn't much point in sharing it.

Regarding this -- again from my perspective I am still curious about a synthesizer's output even if it is a trivial one or bad or anything, as long as I am informed about its status.

[izgzhen]

ping @Calvin-L