Add support for W&B

[mathpluscode]

Subject of the feature

It can be very helpful to allow users to upload their training results to W&B https://wandb.ai/site in addition to tensorboard.

This feature should be activated using either config / env variable.

[NMontanaBrown]

I could do this, I have experience with WandB?

[NMontanaBrown]

TODOs:

• [x] Add config option where if Wandb key exists, it will be parsed into the train script.
• [x] If the config is incorrect, chuck warning, train as normal
• [x] If config Wandb exists and is correct, initialise callback
• Docs
  • [x] Example on how to add Wandb config to current configs. Link to Wandb docs.
  • [ ] Changelog
  • Tests

[NMontanaBrown]

Not sure how we are going to test that WandB is logging a run though. @mathpluscode @YipengHu any ideas?

[mathpluscode]

Not sure how we are going to test that WandB is logging a run though. @mathpluscode @YipengHu any ideas?

We will need to mock these functionalities.

[NMontanaBrown]

Not sure how we are going to test that WandB is logging a run though. @mathpluscode @YipengHu any ideas?

We will need to mock these functionalities.

How? Any links? Not even sure we can mock this.

[mathpluscode]

Not sure how we are going to test that WandB is logging a run though. @mathpluscode @YipengHu any ideas?

We will need to mock these functionalities.

How? Any links? Not even sure we can mock this.

Here's an example mocking gcloud functions https://stackoverflow.com/questions/62803901/python-unittest-mock-google-storage-how-to-achieve-exceptions-notfound-as-side

The main idea is just to pretend that we've called the wandb functions. The package would be https://docs.python.org/3/library/unittest.mock.html. This is the common choice instead of using pytest.

For further details, I do not understand these 100% so better to google it ;)

[NMontanaBrown]

Maybe cose #476 because this is available in WB.

[YipengHu]

Maybe cose #476 because this is available in WB.

@mathpluscode ?

[mathpluscode]

Maybe cose #476 because this is available in WB.

@NMontanaBrown What WB has are basic methods like grid search or random search or Bayesian. Ofc it's a good starting point, but I think we do not have to close that issue, as:

1. adding W&B for logging and adding W&B for hyper parameters can be two different things. Specially hyper parameter tuning requires changes of configs I believe.

2. we may want to look at some more advanced tuning techniques:)

[NMontanaBrown]

Ok, in any case @YipengHu and I talked about closing this ticket soon because WandB should provide tests for their successful running, not sure worth mocking.

[mathpluscode]

Ok, in any case @YipengHu and I talked about closing this ticket soon because WandB should provide tests for their successful running, not sure worth mocking.

@NMontanaBrown you can skip these test requirements for now?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.