Folding MVP Final Week Sprint

Here are the details for the final week's sprint on Folding. Folding is currently in a place that can operate such that is has high success rates for validator hp search, but validators in general are not busy, and there are open questions about how miners are asked for work. Here are some key components

gro-based energy calculations (exploit resistant)
Validator scheduler & DB
Hyperparameter tuning (timeout, wait_time, sample_size, max_concurrent_pdbs) for optimal workload (adaptive?)
Miner process pool executor for multiple jobs
Some restructuring of data directories (low priority?)
Logging and persistence of final results
Data viz
Loads of experiments

We can try and break these down into tasks for each member of the team.

@steffencruz

[ ] Validator scheduler and DB management.
- Validators to ping miners to see if they are alive
- Validators can process/wait more than 1 pdb at a time
- Validators can query miners at some pre-defined interval to acquire .gro files

@mccrindlebrian

[ ] Mining infrastructure
- Miner process pool executor (aka: being able to pull out pdbs out of the queue and not always having to wait for another Pdb request from a validator), and spinning up an N number of child processes based on the capacity of their CPU. Need to check if this design is a problem with GPU-compiled Gromacs?
- Being able to be interrupted during simulation, return requested data, and continue simulation (with or without a new ckpt file?)
- Therefore a part of this is storing which validators get what pdb? I guess this is handled within the child process itself so we don't need to explicitly keep track of this?
[ ] Logging and persistence of final results
- This infra is already in place, I just need to add more to the miner side. This requires some refactoring of the run_commands function

@schampoux

[ ] Gro-based energy calculations
- Implement a methodology on the validator side that can apply the necessary steps to compute the next step (or next N steps) of any energy file via the -rerun method.
[ ] Data visualization
- We can currently plot data if given in the reward stack (we log this to wandb) but it would also be good to see if we can do this on the wandb side (if the data is available)
- More importantly, we want to see the protein! Find a way to show the protein given the necessary files. This can be separate from wandb

@RodrigoLPA

[ ] We need to you to run your parallelization pdb script and gather all the statistics.
- Number of successful pdbs
- What are the properties of these pdbs? Distribution of size/ other metrics of complexity. It looks like simple pdbs might be out of the question since they optimize too quickly. We need to know how large this space is.

Everyone

We need to run a boat load of experiments, specifically:

[ ] Run with multiple validators (3) on different machines
[ ] Run with at least 10 miners

I think this is a good opportunity for @RodrigoLPA to learn how to spin up miners and validators, use pm2, and manage across multiple machine. I am happy to support here with 1 on 1 time getting him up to speed

macrocosm-os / folding