Evaluation Improvements

aidan-curtis / firehose

We are trying to put out big fires

GNU General Public License v3.0

3 stars 2 forks source link

Evaluation Improvements #10

Closed williamshen-nz closed 2 years ago

williamshen-nz commented 2 years ago

https://github.com/aidan-curtis/firehose/blob/main/cell2fire/evaluate_model.py

[x] Right now the evaluation script doesn't support random ignition points. Support random ignition points
[x] Support all the args that train script does.
[x] Support multiple eval iterations and collecting data into JSON or whatever.
[x] Jupyter notebook for plotting and making life easy
[x] Update gym env to output data we want

williamshen-nz commented 2 years ago

If we are doing random ignition points, we want all algorithms to be evaluated on the same random ignition point. We do not support this functionality at the moment.

[x] Support using same random ignition points across multiple algorithms.
- We could take the JSON output from one run and use those ignition points.
- Or have a standalone script to generate random ignition points as a JSON and feed it into the evaluation script

This is not critical, we will use reward curve as measure of performance

williamshen-nz commented 2 years ago

Done in https://github.com/aidan-curtis/firehose/commit/2d926d48cc7e375e3dfa241216e0720f638f5f1f

We generate a JSON with random ignition points beforehand. Then we can run the evaluation script individually for each algorithm.

Then we plot them as required.