These are changes I've made locally to get benchmarks to run.
Most COS requests during and after the annotate step will now retry. I've consolidated the logic into utils and made sure that exceptions are always logged, and rethrown if there's a repeated failure.
Also, many requests now pass the stream to msgpack.load instead of downloading the entire object to a buffer before converting it. In theory this could save some memory, but I haven't tested if that's actually the case.
The cpyMSpec version has been pinned to 0.3.5. This is needed for the tests in Experiment 1 to pass. I've also added an assertion to isocalc_wrapper.py to ensure that the correct version is used, because it's otherwise very difficult to debug why the test is failing.
Experiment 3 has been changed to use the "huge3" dataset
These are changes I've made locally to get benchmarks to run.
utils
and made sure that exceptions are always logged, and rethrown if there's a repeated failure.msgpack.load
instead of downloading the entire object to a buffer before converting it. In theory this could save some memory, but I haven't tested if that's actually the case.cpyMSpec
version has been pinned to 0.3.5. This is needed for the tests in Experiment 1 to pass. I've also added an assertion toisocalc_wrapper.py
to ensure that the correct version is used, because it's otherwise very difficult to debug why the test is failing.