illinoisdata / DeepOLA

7 stars 5 forks source link

Save intermediate query results and durations for accuracy plot #134

Closed mIXs222 closed 1 year ago

mIXs222 commented 1 year ago

Currently run_query only saves the last df query result. We would want intermediate results to measure accuracy over partition/time. These changes are

  1. Save possibly multiple intermediate dfs into multiple CSVs
  2. Save durations for these results into JSON (for parsing/inspecting convenience)

Add some error handling too, e.g. the panic message is more clear when outputs directory doesn't exist.