Right now the re-running of benchmarks to update weights in subtensor is fairly haphazard, is not occurring on a regular basis, and is not being done on consistent / reference hardware, to alleviate this I recommend:
[ ] Set up a custom gh runner specifically for running benchmarking jobs that matches the minimum hardware we would want for a validator. When creating the self-hosted runner, set --max-jobs 1 so that only one benchmark job can run at a time
[ ] Document the recommended/minimum validator hardware spec somewhere (if we don't already)
[ ] Set up a CI action that runs the benchmarking suite on the custom runner. This action would be able to add a commit committing the updated weights
CI job wpossibly would be triggered as part of the tagging/release process, or manually triggered, or possibly even whenever something goes into main, pros and cons to each of those
The end goal is that subtensor's on-chain benchmarking weights always reflect the current code on consistent reference hardware without any lift from the team
Right now the re-running of benchmarks to update weights in subtensor is fairly haphazard, is not occurring on a regular basis, and is not being done on consistent / reference hardware, to alleviate this I recommend:
--max-jobs 1so that only one benchmark job can run at a timeCI job wpossibly would be triggered as part of the tagging/release process, or manually triggered, or possibly even whenever something goes into main, pros and cons to each of those
The end goal is that subtensor's on-chain benchmarking weights always reflect the current code on consistent reference hardware without any lift from the team