search

microsoft / TypeChat

TypeChat is a library that makes it easy to build natural language interfaces using types.
https://microsoft.github.io/TypeChat/
MIT License
8.06k stars 379 forks source link

Implement some snapshot testing for our Python schema translation #165

Closed DanielRosenwasser closed 3 months ago

DanielRosenwasser commented 6 months ago

Currently we have a few tests that can be run manually, but we have no testing infrastructure to make sure things don't regress.

I would advocate for something like snapshot/baseline testing which we use within TypeScript.

What we do on TypeScript is have outputs always checked against a "reference" folder which is treated like the source of truth. Newer outputs are emitted to a "local"/"current" folder which can be compared to the reference folder.

When generated files are different from what's in the reference folder, the test runner fails. These folders can be diffed easily using a tool like Beyond Compare or WinMerge. When changes are considered desirable, we have have a way of accepting the baselines using a command (or they can be manually copied over).

I'm not as familiar with testing in the Python ecosystem. syrupy and pytest-snapshot seem to be a plugins for pytest which had a recent release. There's also turnt which is all about capturing compiler outputs and the like.

DanielRosenwasser commented 3 months ago

https://github.com/microsoft/TypeChat/pull/186 and https://github.com/microsoft/TypeChat/pull/209 have sufficiently implemented snapshotting and CI.