It looks like right now our re-running of benchmarks to update weights in subtensor is fairly haphazard, is not occurring on a regular basis, and is not being done on consistent / reference hardware. We need something that will allow us to run this automatically on reference hardware, preferably every time we deploy to testnet/finney.
AC:
[ ] Sync with @rajkaramchedu and @opentensor/medulla to ensure that minimum recommended hardware specs for running a validator are publicly documented and updated on some recurring basis, like annually
[ ] Sync with @opentensor/medulla to set up a custom CI runner (notSubtensorCI) that would be solely for running benchmarking jobs. It should match our minimum recommended hardware specs for running a validator.
[ ] Set up a CI action that runs the benchmarking suite on the specified branch using this custom CI runner, and commits the weight changes in a new commit to the branch. Concurrency settings must be configured such that only one benchmarking job can run on the runner at a time, to avoid tainting the results of any particular run
[ ] Verify the github action works
[ ] Decide how we want the github action to be triggered -- manually, on push to testnet / finney, on PR for those..?
sam0x17
commented
3 months ago
It looks like right now our re-running of benchmarks to update weights in subtensor is fairly haphazard, is not occurring on a regular basis, and is not being done on consistent / reference hardware. We need something that will allow us to run this automatically on reference hardware, preferably every time we deploy to testnet/finney.
AC:
SubtensorCI) that would be solely for running benchmarking jobs. It should match our minimum recommended hardware specs for running a validator.testnet/finney, on PR for those..?