search

captain-pool / hydra-wandb-sweeper

WandB sweeps integration with Hydra sweeper
47 stars 6 forks source link

RAY or SageMaker ? #10

Open charleswilmot opened 3 months ago

charleswilmot commented 3 months ago

Oh captain my captain,

As it seems, hydra plays well with RAY_AWS, but W&B prefers SageMaker.

Do you have some experience with hydra + W&B + (RAY|SageMaker) ? Do you have an advice to give on that front ?

Thank you for the good work

captain-pool commented 3 months ago

Hi @charleswilmot , thanks for the question. Unfortunately, this library is no longer maintained, so it is not up to date with the recent changes made to hydra or wandb or ray. When I was building this library, I was primary targetting this for ray when invoked this way: https://hydra.cc/docs/plugins/ray_launcher/#ray-launcher. However, I am not 100% sure if it would still automatically support ray, since there were some further commits which added support to SLURM cluster launcher, and I did not get the chance to write proper tests to make sure nothing breaks. My guess, it should support. But in case it does not, you most likely would need to add a custom launcher which wraps around the main ray launcher to add support. Example: https://github.com/captain-pool/hydra-wandb-sweeper/blob/main/example/hydra_plugins/launchers/my_submitit_launcher.py .

All you would need to do, is: pass hydra/sweeper=wandb hydra/launcher=ray in your config or in the command line during invocation.

I am curious about the current state as well, let me know if it works!

Thanks, Adrish

charleswilmot commented 2 months ago

Hi Adrish,

Thank you very much for your answer. At this stage, I am discussing with my colleagues what might be the best approach for us. Should we decide to use the hydra-wandb-sweeper, I'll make sure to report back our findings in this thread.

Cheers