Open jchodera opened 2 years ago
Yes, considering benchmarking is becoming bigger we do want to have better (more efficient and friendly) and smarter ways to deal with these situations. I wonder if we want to have a whole new benchmarking
module for perses, considering how this is evolving.
Seems like it is becoming even more beneficial to have a whole module for the benchmarks part of perses https://github.com/choderalab/perses/pull/1050#discussion_r903025368
Is this for performance benchmarking (ns/day, time for a single calculation) or accuracy benchmarking?
Is this for performance benchmarking (ns/day, time for a single calculation) or accuracy benchmarking?
This is for accuracy benchmarking. That is, running the systems in the protein-ligand-benchmark
dataset and checking the plots and errors.
There appear to be some issues with how the
run_benchmarks.py
script is written: If each script is supposed to handle a transformation, which may cause issues if this code runs concurrently for multiple transformations:Presumably, we want to break this into multiple stages, or find some way to appropriately construct and execute the dependency graph on the cluster:
Alternatively, we can make sure that every transformation acts fully independently until the analysis stage at the end using diffnet.
We'll want to find a more clever way to refactor this in the next release so we can support benchmarking multiple targets at the same time as well.