Closed fredshone closed 2 years ago
Thanks @fredshone -- I'm approving, but have also left a couple thoughts on adding some fail safes for cases where people supply bad values for plot_type
.
I've cleaned up test fixtures.
At this time I am not in favour of being strict about the benchmark kwargs that get passed. There will obviously be room for miss-spelling of keys and bad values, but i consider the benchmark module to be WIP and don't want to engineer anything at this point. One bit of good news is that kwargs that aren't specifically caught will be appended to output file names. So will help with debugging.
this is a monster (sorry about the merge Andrew):
trip_durations_comparison
euclidean_distance_comparison
toeuclidean_distance_breakdown_comparison
duration_comparison
toduration_breakdown_comparison
link_vehicle_speeds_comparison
scripts/example_configs_smoke_tests.sh
to the build (this elara runs all confiogs found in example_configs)output_directory_override
, but this is only exposed for smoke tests (so that they can write to a temporary directory) so not documentedEDIT
Trip durations comparisons (aka google trips queries):
Configured via the config as follows:
The expected format for the new benchmark data (csv) is:
Where agent is the agent id and seq the trip sequence. Duration is in seconds!
I also wanted to support the case where we had colleccted data about trip durations but some agents may have changed modes in sim. If we wanted to only compare agent trips that still use the same mode, we can do as follows:
In which case we should additionally provide a mode column:
This ensures, for example, that if chris has shifted to bike mode for seq 1, that this comparison is ignored. The default for mode_consistent is false.
I am intending to add standalone docs for the benchmarks in a future PR