Closed bwdeng20 closed 2 months ago
@bwdeng20 thank you for dropping by and thank you for the toy repo. I am not a user of wandb, nor have an account. So I am not able to reproduce your issue.
From what I see in the code, there is no reason yet for me to believe that there is a bug in jsonargparse, rather than a problem in your code. The instructions in wandb are not specific to jsonargparse. And it doesn't explain how the sweeps work internally.
How is it supposed to work? How are the modified parameters of the sweep given to jsonargparse? Is it via sys.argv
, via environment variables, something else? The only thing I can deduce from the screenshot is that the configs/nc.yaml file is loaded. This file is fixed and no init_args
are given, thus it makes sense that the values are the defaults of the FakeResultGenerator
class.
Thanks for the swift response. Maybe I should change the tags from bug
to question
.
Solidify a set of experiment configurations (in this case ) into a configuration file (in this case configs/nc.yaml). At this point, running python script.py --configs/nc.yaml
will use the default parameter values in configs/nc.yaml for an experiment. When we want to tune some hyperparameters (in this case, the construction parameters mu
and sigma
of FakeResultGenerator
), first define reading the default experiment configuration nc.yaml
in tuning config sweeps/swp_nc.yaml
by
# a snippet from sweeps/swp_nc.yaml
command:
- python
- ${program}
- --config
- configs/nc.yaml
and then define the following fields according to the wandb sweep guidelines.
parameters:
gen.init_args:
parameters:
mu:
values: [ 4., 5. ]
sigma:
values: [ 2., 2. ]
We hope the hyperparameter values sampled by wandb sweep
can be used by jsonargparser
to replace the corresponding values defined in configs/nc.yaml
or FakeResultGenerator
defaults to run different experiments, given the wandb sweep commands like
wandb sweep sweeps/swp_nc.yaml # to intialize sweep and get `entity/project/sweep_ID`
wandb agent entity/project/sweep_ID --count 2 # count is the number of hyperparameter tuning trials
I tried to find relevant clues, but I couldn't find any helpful documentation or source code, so I'm sorry that I'm not clear on that.
I tried to find relevant clues, but I couldn't find any helpful documentation or source code, so I'm sorry that I'm not clear on that.
This is kind of key. If the wandb docs are not clear, then the problem is wandb and you should contact them to understand what needs to be done. It isn't a bug in jsonargparse.
Just supposing, imagine that it works by running the command appending more arguments, i.e. sys.argv
. The first run of the sweep would be the command plus the first values in the parameters. jsonargparse would expect the command to be like
python ${program} --config configs/nc.yaml --gen.init_args.mu 4. --gen.init_args.sigma 2.
In your code you could print the value of sys.argv
to see this. But it seems wandb does not work this way, or there is something you are not doing correctly. Still, it is the wandb people the ones who should help.
Closing this since it seems to be a topic for wandb. But feel free to comment further if there is need.
🐛 Bug report
To reproduce
You need a wandb account.
Clone my toy repo and then follow the official wandb sweep instructions like the following.
Then you can see something like
This image shows the terminal output from the first sweep experiment. Note the hyperparameter values in the red rectangles. We can see that
jsonargparser
is not affected by the hyperparameter values (i.e., mu=5, sigma=2) chosen by wandb sweep . Instead, it still uses the default values (i.e., mu=0, sigma=1). This means that adjusting hyperparameters with wandb sweep is not compatible withjsonargparser
.Expected behavior
The args parsed by
jsonargparser
can be changed by wandb sweep.Environment
4.27.2
3.10
pip install jsonargparse[all]
):pip install jsonargparse[all]
Ubuntu22.04
0.16