I recently came across the concept of snapshot testing. Maybe this could be of use here in an inverse fashion. Let me explain :smile:
Snapshot testing is done in a way, that the first time a test is run on a certain kind of object I do not want to fully describe by hand, a snapshot is taken of that object and all its properties etc. I'm thinking of it in terms of json serialization. Every time the test is run in the future the object is then compared against the snapshot.
This by itself does not solve our problem of not having to download the dataset each time. But a solution could be to locally store an example from the dataset and compare it against a snapshot of a previous version of it. To force the test to be successfully run, we could implement a pseudo-test, which checks if the example-to-be-tested has been rewritten for this commit, i.e. updated with the new dataset code. It fails if not, forcing the user to locally update the example. One could also make the pseudo test more fine grained, s.t. it only runs/fails if special dataset code has been changed.
In general I think that would be a good way of starting to implement a workflow on how to work with testable data: Forcing a dataset to have locally generatable descriptors, that can be tested.
I recently came across the concept of snapshot testing. Maybe this could be of use here in an inverse fashion. Let me explain :smile:
Snapshot testing is done in a way, that the first time a test is run on a certain kind of object I do not want to fully describe by hand, a snapshot is taken of that object and all its properties etc. I'm thinking of it in terms of json serialization. Every time the test is run in the future the object is then compared against the snapshot.
This by itself does not solve our problem of not having to download the dataset each time. But a solution could be to locally store an example from the dataset and compare it against a snapshot of a previous version of it. To force the test to be successfully run, we could implement a pseudo-test, which checks if the example-to-be-tested has been rewritten for this commit, i.e. updated with the new dataset code. It fails if not, forcing the user to locally update the example. One could also make the pseudo test more fine grained, s.t. it only runs/fails if special dataset code has been changed.
In general I think that would be a good way of starting to implement a workflow on how to work with testable data: Forcing a dataset to have locally generatable descriptors, that can be tested.