search

gintool / gin

GI in No Time - a Simple Microframework for Genetic Improvement
MIT License
43 stars 20 forks source link

Expensive test cases #90

Open GiovaniGuizzo opened 1 year ago

GiovaniGuizzo commented 1 year ago

I am dealing with a few tests in a project that is used to manage graphs. As you can imagine, they take a long time when they are testing specific graph traversal algorithms.

The issue here is that the default timeout we enforce is too low for those tests. Even when I double the timeout, these tests still fail from time to time. Increasing the parameter even more would mean that all tests have the same timeout, which would increase the time needed by Gin to perform the search in case of infinite loops.

It is not a bug exactly, but I was thinking that it could be improved in the future to avoid such a thing. Perhaps get the maximum time needed by a given test case in the original program over the 10 repetitions and dynamically setting the timeout for the variants to, say, 2 x MAX or a given constant, whichever is bigger. That way we could dynamically treat such test cases.

justynapt commented 1 year ago

At some point there was functionality to set timeout for individual tests in gin.util.Sampler. There is still some code there that has been commented out (although looks like it's doing smth slightly different) and a helper method in gin.test.UnitTestResultSet (again commented out). I think we removed it for simplicity at some point, but I would imagine it would be easy to re-add.