search

fact-project / shifthelper

So we can sleep at night.
4 stars 0 forks source link

Evaluate shifthelper on historical data #143

Open dneise opened 7 years ago

dneise commented 7 years ago

Adrian requested the following study, which I am going to perform. It is not directly related with shifthelper code or features or docu, but I wanted to keep you all informed, so I though I'd open an issue about it:

The task is this:

See how often the shifthelper would have called and for which reason in the past. So we better understand what the requirements really are to the people going to be called.

This should also work as a kind of functional test on the shifthelper, as in: it should have called on 2013.10.14 around 20:11h UTC ... would it really do that? Just on a bigger data set than just this particular minute.

How?

We wanted to test the real thing as much as possible. So while one could take out the checks from checks.py or even just the conditions on conditions.py, we'd rather want to test the real thing.

In order to test it on historical data, we just need to feed it fake historical data instead of real current data. I am planning to implement a fake_smart_fact_crawler which gets its data from aux-files and has the exact same interface as the current smart_fact_crawler. But I need to be able to tell this fake_sfc, what time its answer should be based on, while not hampering with the interface. For this we are planning to try libfaketime or rather faketime ... I tried it on my laptop and it did work ...

So the fake_smart_fact_crawler with simply use the current system time to find the right aux-file, open it and deliver the correct row. This makes no sense at all, when the system time is not compromised, or one has no aux-files containing aux-data for "right now and the next ten minutes". So testing this fake_smart_fact_crawler without libfaketime is not going to work.

So far the plan. And now I'm starting to look into it. In case you have other ideas, and I'm sure you will, don't hesitate to come out with them.

dneise commented 7 years ago

Okay so I found 21 values the shifthelper actually needs from the smart_fact_crawler. I will only fake these and not all of them.