added back useful wandb metrics

[surcyf123]

The previous commit is problematic because now one can't pull a run by the netuid, created timestamp, among many other helpful config details (reward model weighting, reward models off, etc.). The sensitive data I noticed will still be removed from the config. Please review quick so we don't lose valuable data!

[p-ferreira]

Hello @surcyf123, ty for taking the time to push this commit.

The rationale behind the idea of shrinking the config in the wandb came from the need to omit some particular information from the validators config that could be exploited, considering that some key data are already logged as tags or by wandb automatically.

Unfortunately, it seems that the alternatives for wandb data retrieval are quite limited due to current state of their API. (More details on issue #136)

So I agree with you that it will be interesting to add more data to be logged in the metrics, such as config.netuid and config.wandb.

Now, about the other mentioned filters:

• created timestamp: this was never manually logged in wandb config, wandb logs this as default and the change implemented on #132 didn’t changed that
• reward model weighting: The reward model weighting is being logged like it was before, #132 didn’t changed that.
• reward models: The reward models were not part of the config, #132 didn’t changed that.

The change you propose is to include all the config data with the exception of the following:

• 'wallet_name', 'wallet_hotkey', 'neuron_full_path', 'neuron_name', 'User', 'axon_port', 'subtensor_chain_endpoint', 'neuron_reward_path', 'openai_api_key’

On that end, I have one question:

• openai_api_key: Even though this is set to be excluded, could you please provide us with more information on why would you need an openai_api_key on the validator in the first place?

If you are ok with the proposed changes of #136 that plans to bring back the ****netuid and wandb**** data back to the config, you can drop this PR and follow that issue. Else, you are more than welcome to continue this conversation so we bring a solution that works for everybody. Any details of your use case will be greatly appreciated.

[surcyf123]

Hey @p-ferreira, thanks so much for the in-depth and thoughtful response.

I agree with you about the idea about needing to omit the sensitive wandb data that has the potential to be exploited. For this reason, the PR still removes the sensitive data from the config. I think all other config arguments can be valuable for analysis. The most crucial one being netuid and the tags (which includes validator version, validator hotkey, and the reward models used). Other than that, after pulling and sorting all wandb config details, the majority are unnecessary, but interesting to see.

For example, we can see a couple validators are running with non-default neuron.answer_timeout at 20 or 30 seconds. Or that some validators have runs with reward.reciprocate_weight or reward.diversity_weight set to lower than default, or at 0, with higher weighting on the rlhf reward model.

Regarding your question about the openai_api_key being used within the config: This was in the config dataset that I pulled because people are running and logging mining runs within the wandb data in addition to validator runs. I noticed that some runs had an api key within the config. I can share the dataset of all previous config details if it would be of interest to you while considering this PR. I also noticed a lot of other mining flags that people have used to log mining runs on wandb. But by exlucluding their api key we can allow them to do this safely.

I will continue the convesation on #136 as I noticed I may have some insight to share for the wandb api pulling script.

[p-ferreira]

I will close this PR as the implementation of this issue was shifted to #136 and #137 Let me know in that issue/PR if there is any consideration on your end related to the implemented solution.