Closed dchaley closed 2 months ago
It's possible, but pretty janky to do this in the benchmark.py script. We're starting to diverge substantially in terms of functionality between the multi-step versions, and the benchmark.py version.
The main(/only?) advantage of benchmark.py is that it doesn't push files to/from cloud storage in between steps. So, this change is likely to regress performance by introducing the intra-step copies.
benchmark.py uploads to BQ using the old schema (assuming one instance runs everything).
The batch results schema was updated to have parameters per phase (preprocess, predict, postprocess).
We need to change the benchmark.py process to upload the new schema. Hopefully we have all the data we need.
If that's not possible, we need to bite the bullet and unify the implementations.