search

quintel / mechanical_turk

Automatic tester for etengine
1 stars 0 forks source link

Run specs for random scenarios #145

Open michieldenhaan opened 4 years ago

michieldenhaan commented 4 years ago

The ETM has various 'laws' that should hold under any circumstances. E.g. supply and demand of the hydrogen, collective heat and ccus mekkos should always balance, inputs and outputs of (most) nodes should be equal etc.

I think it would be very useful if we have something like Turk::Scenario.new(random, n=10) that generates n random scenarios. With random I mean:

to make testing for these 'laws' more robust. Currently, we have to explicitly specify all of the above. As a result we typically only test a few very specific situations.

Curious to hear your thoughts, @ChaelKruip, @antw

antw commented 4 years ago

The random element makes me a little uneasy, since it's quite possible that the scenario might set inputs which have nothing to do with the law being tested. It's possible that one of these laws might be broken, but for us to miss it because the wrong random numbers are generated. It might be that a law is only broken when a certain combination of inputs are set, and with 811 inputs the probability of happening upon that combination is quite low.

However I don't have a better suggestion, so I've added this to the backlog.

michieldenhaan commented 4 years ago

Thanks! I fully agree with your concerns. My worry is that we currently only test one or two predefined settings for most 'laws'. Supplementing this with X random combinations that change every night may allow us to find flaws that would otherwise have gone unnoticed. But it is definitely not fool proof.