macrocosm-os / folding

Decentralized Protein Folding Bittensor Subnet
https://www.macrocosmos.ai/sn25
MIT License
14 stars 16 forks source link

Handling Large File Buildup from Simulations #51

Closed schampoux closed 4 months ago

schampoux commented 5 months ago

This issue will uncover the cause for large file buildup in the validator directories, as well as strategies to mitigate excessive file growth from the simulations.

schampoux commented 5 months ago

Each miner is responsible for roughly 33MB of output for each forward pass. These files include:

The first 6 files are ready to be analyzed by the reward stack, and the rest of the files are used to create them. These files can be reduced depending on what the reward stack contains. for example, if we only want to use the data form the production run, we can eliminate some of the other files.

The remaining files in the validator directory (12MB) include everything used to get the simulations going. These files include:

Fortunately, They take up less space and they are more valuable.