Staging

[mccrindlebrian]

Features:

• Remove async nature of wandb logging because this seems to have unintended issues with wandb getting stuck
• Add tenacity retry to try and set weights 3 times before moving on.

Bug Fixes:

• Move the initial sync of the validator to the outer loop to avoid deleting previously saved weights
• Remove the sync inside of the base validator to avoid having miner weights go to 0 on each restart.

Extras

• Increase block time for weight setting from 100 to 250 to help smooth out miner weights