Closed janvainer closed 2 years ago
Can you show Encoder1, Encoder2 and Decoder1/2 in your example?
@idoby thank you for response! I adjusted the example to use Encoder and Decoder abstractions.
Looks like argparse doesn't support this easily, which kind of makes sense because this style is not very unix-like. A more standard command line would look like:
python script.py --some_unrelated_args \
--encoder=rnnencoder --args_related_to_rnnencoder \
--decoder=convdecoder --args_related_to_convdecoder
Would that be a good solution @janvainer ? It's not too hard to implement this with SimpleParsing but it is a little hacky. I suggest we add support for this kind of "polymorphic" command line in SP. What do you think @lebrice ?
I implemented something like this in my project in a few lines but it would be nice if SP had built-in support.
Yeah this would be great, especially if its less hacky that what I suggested! :) As long as I can do
@dataclass
class Global:
letter: Union[A, B] = A()
number: Union[One, Two] = One()
some_other_arg: int = 5
I am very happy :D
Greeting @janvainer , thanks for posting!
This is the eternal thorn in my foot, and it's quite possible IMO that this issue of Argparse was one of the biggest motivations for other approaches like Hydra's.
I've worked on this kind of thing a LOT. There is no neat way of doing this in SP, and I don't quite see a clean way forward.
I implemented something like this in my project in a few lines but it would be nice if SP had built-in support.
Talk is cheap @idoby . Show me the code! :stuck_out_tongue:
As for your question @janvainer , the best solution I've found so far is to add every combination of nested subparsers.
It sounds ugly, but it's actually pretty neat. The only thing you have to watch out for is the order in which the arguments are passed. It's also a bit buggy with lists and other types of fields that generate a value of nargs
"*".
I called this feature add_independant_subparsers
. There's a proof-of-concept for this in the lebrice/multiple_subparsers
branch, which I'll probably make a PR for soon.
Here is an example:
from typing import ClassVar, Tuple
from simple_parsing.helpers.independent_subparsers import add_independent_subparsers
from dataclasses import dataclass
from simple_parsing import ArgumentParser
from pathlib import Path
import os
@dataclass
class DatasetConfig:
""" Configuration options for the dataset. """
image_size: ClassVar[Tuple[int, int, int]]
# Number of samples to keep. (-1 to keep all samples).
n_samples: int = -1
# Wether to shuffle the dataset.
shuffle: bool = True
@dataclass
class MnistConfig(DatasetConfig):
""" Configuration options for the MNIST dataset. """
image_size: ClassVar[Tuple[int, int, int]] = (28, 28, 1)
n_samples: int = 10_000 # some random number just for sake of illustration.
shuffle: bool = True
foo: str = "foo_hey"
@dataclass
class ImageNetConfig(DatasetConfig):
""" Configuration options for the ImageNet dataset. """
image_size: ClassVar[Tuple[int, int, int]] = (28, 28, 1)
n_samples: int = 10_000_000 # some random number just for sake of illustration.
shuffle: bool = False
# Path to the imagenet directory.
path: Path = os.environ.get("IMAGENET_DIR", "data/imagenet")
@dataclass
class ModelConfig:
""" Configuration options for the Model. """
# Learning rate.
lr: float = 3e-4
@dataclass
class SimpleCNNConfig(ModelConfig):
""" Configuration options for a simple CNN model. """
lr: float = 1e-3
@dataclass
class ResNetConfig(ModelConfig):
""" Configuration options for the ResNet model. """
lr: float = 1e-6
def main():
parser = ArgumentParser(description=__doc__)
add_independent_subparsers(
parser,
dataset={"mnist": MnistConfig, "imagenet": ImageNetConfig},
model={"simple_cnn": SimpleCNNConfig, "resnet": ResNetConfig},
)
args = parser.parse_args()
dataset_config: DatasetConfig = args.dataset
model_config: ModelConfig = args.model
print(f"Args: {args}")
print(f"Dataset config: {dataset_config}")
print(f"Model config: {model_config}")
$ python examples/multiple_subparsers/multiple_subparsers_example.py --help
usage: multiple_subparsers_example.py [-h] <dataset>|<model> ...
optional arguments:
-h, --help show this help message and exit
dataset or model:
<dataset>|<model>
mnist Configuration options for the MNIST dataset. (help)
imagenet Configuration options for the ImageNet dataset. (help)
simple_cnn Configuration options for a simple CNN model. (help)
resnet Configuration options for the ResNet model. (help)
$ python examples/multiple_subparsers/multiple_subparsers_example.py mnist --help
usage: multiple_subparsers_example.py mnist [-h] [--n_samples int] [--shuffle bool] [--foo str] <model> ...
Configuration options for the MNIST dataset. (desc.)
optional arguments:
-h, --help show this help message and exit
model:
<model>
simple_cnn Configuration options for a simple CNN model. (help)
resnet Configuration options for the ResNet model. (help)
MnistConfig ['command_0']:
Configuration options for the MNIST dataset.
--n_samples int some random number just for sake of illustration. (default: 10000)
--shuffle bool (default: True)
--foo str (default: foo_hey)
$ python examples/multiple_subparsers/multiple_subparsers_example.py mnist resnet
Args: Namespace(dataset=MnistConfig(n_samples=10000, shuffle=True, foo='foo_hey'), model=ResNetConfig(lr=1e-06))
Dataset config: MnistConfig(n_samples=10000, shuffle=True, foo='foo_hey')
Model config: ResNetConfig(lr=1e-06)
Things to watch out for:
$ python examples/multiple_subparsers/multiple_subparsers_example.py mnist resnet --shuffle
usage: multiple_subparsers_example.py [-h] <dataset>|<model> ...
multiple_subparsers_example.py: error: unrecognized arguments: --shuffle
$ python examples/multiple_subparsers/multiple_subparsers_example.py mnist --shuffle resnet
usage: multiple_subparsers_example.py mnist [-h] [--n_samples int] [--shuffle bool] [--foo str] <model> ...
multiple_subparsers_example.py mnist: error: argument --shuffle: Boolean value expected for argument, received 'resnet'
Here's the correct way to pass the shuffle
argument for the dataset, for example:
$ python examples/multiple_subparsers/multiple_subparsers_example.py mnist --shuffle=True resnet
Args: Namespace(dataset=MnistConfig(n_samples=10000, shuffle=True, foo='foo_hey'), model=ResNetConfig(lr=1e-06))
Dataset config: MnistConfig(n_samples=10000, shuffle=True, foo='foo_hey')
Model config: ResNetConfig(lr=1e-06)
It's not a perfect solution, and there are probably bugs lurking in there, but it should hopefully be useful to you or to others, even in it's current form.
Hope this helps. If you want to use it, I suggest copying the contents of this function here, or waiting until I've merged this to master with a PR.
Well, it's a hack, but here goes.
I was testing out different optimizers and hyperparameters, and didn't like that my experiments had this configuration code in them, so I made this (details omitted for clarity):
def optimizer(model_params, train_config: TrainConfig) -> torch.optim.Optimizer:
if train_config.optimizer_type.lower() == 'adam':
adam_config = poly_config(AdamConfig, 'adam_')
return torch.optim.Adam(model_params, lr=train_config.lr, weight_decay=train_config.weight_decay,
betas=adam_config.betas)
elif ...:
train_config is a dataclass that includes the optimizer_type field that is read by SP from the command line, then poly_config injects the appropriate class and reparses the command line at runtime. There's nothing stopping me from making this more robust with automatic mappings etc. But this allows me to have a hierarchy of *Config types for each optimizer, where client code only ever sees the appropriate concrete type at runtime (e.g. AdamConfig, SGDConfig, etc). The line where I use this class to create a torch adam object is just to show the point of this.
The resulting command line is:
python myscript.py --lr=0.0001 --optimizer_type=adam --adam_betas=0.9 0.9
If you don't supply adam_betas you will either get a default value or an error if there's no default value. Once I automate this, I will also be able to throw an error if any arguments are leftover when the whole parsing procedure is done (i.e the user specified --adam_betas in an invalid context like not having specified Adam first).
While it is a hack, doing hacks on top of argparse seems to be SP's job. However, since SP subclasses AP and provides the same interface, I'm not sure multiple parsing passes are possible to implement in SP without breaking AP's contract. I'll need to think about this some more.
On second thought, it seems better for SP to stick to simple parsing and let my library do all the hacks on top of that.
@lebrice Thank you for the response! Much appreciated.
Do you think there is a clean way how to leverage the Union
typing to have the multiple subcommand options as fields in a parent config?
Ie
@dataclass
class Global:
letter: Union[A, B] = A()
number: Union[One, Two] = One()
some_other_arg: int = 5
If not I'll simply use what you are suggesting, it does what I want in the end and is simple :) Any ETA on merging this?
Edit: after a second thought, it may be a bit confusing that there is missing a keyword for the subparser options - either script.py --dataset=mnist
or script.py dataset mnist
would be more informative than just script.py mnist
. WDYT?
Thanks for the response @idoby !
I've also used this kind of approach before (also for types of optimizers) and it worked well for a while, until I wasted a bunch of jobs in an HPO sweep because I misspelled an argument, and an error wasn't raised explicitly.
There are ways around this however. For instance, in your setup, If you had the poly_config
function modify a global variable containing all the leftover arguments, taking away some arguments each time, then, at the point where all the args are supposed to have been parsed (e.g. once the models and optimizers have been created), you could raise an error if there are any leftover unused arguments.
Do you think there is a clean way how to leverage the Union typing to have the multiple subcommand options as fields in a parent config?
Sure, it would be possible to do, but it wouldn't be that easy. We'd need to collect all the subparser fields on all dataclasses at a given level (i.e. all the dataclasses passed to parser.add_arguments
for the same parser) and then add them after all the other non-subparser arguments.
I'm also not sure if this would need to be recursive, i.e. collect all the subparser fields of all subparser fields, etc.
it may be a bit confusing that there is missing a keyword for the subparser options - either script.py --dataset=mnist or script.py dataset mnist would be more informative than just script.py mnist
Agreed 100%, but it isn't possible to pass a subparser name as an argument in Argparse, as far as I can tell. If someone figures out a way to do this with regular argparse that isn't too hacky, I'd be very interested.
One idea could be to replace --model=resnet
in the argv that we receive with just resnet
and then pass it down to argparse.
That would also be super hacky, but it might work?
Let me know what you think! :)
I've also used this kind of approach before (also for types of optimizers) and it worked well for a while, until I wasted a bunch of jobs in an HPO sweep because I misspelled an argument, and an error wasn't raised explicitly.
Yeah, that's why I want to raise the error explicitly and this requires a process that can keep track of the state across multiple parse operations, which my library is doing anyway (and SP isn't and IMO shouldn't).
I'll get around to this at some point, maybe after NeurIPS...
By the way, I'm not opposed to adding this kind of mechanism directly in Simple-Parsing @idoby !
For instance, it could be an idea to not create argparse subparsers when using the dataclass Union syntax. (e.g. model: Union[ResNetConfig, VggConfig]
)
Instead, we could replace a model
subparser with a --model
argument, make parser.parse_args()
call parser.parse_known_args()
internally, and then do the kind of trick you mention directly when parsing the arguments, to resolve the type of each field, then parse the remaining args, etc. until all fields are resolved.
Then, if there are still leftover arguments, we'd raise an error just like argparse would.
We'd just have to design it first, and make it transparent to the user as to what exactly is going on.
I think it would be nice, but I'm not sure it fits well with the argparse mental model. I do think that subparsers were not meant to do what SP is doing with them and this solution could be a better approach. Subparsers are meant for simple git-like commands, they are not a main feature of argparse.
Made a discussion for this feature, with the pseudocode of how I plan to do this: https://github.com/lebrice/SimpleParsing/discussions/134
What I am trying to do:
I am training an encoder-decoder neural network and I have multiple different encoders and decoders. Each encoder and decoder has a separate dataclass config.
I want to be able to specify from the commandline which encoder to use by specifying its config name. Once I specify that, I also want to be able to adjust the selected config fields. So far I was able to accomplish this subparsers:
I have something like
And I simply parse it just like in the subparsers example. I then specify the encoder keyword and then it allows me to specify args specific to the selected architecture.
The problem is, if I want to do the same also for decoder, subparsers are not available anymore and I get
Ideal state
How can I solve my problem with simple_parsing? My ideal syntax would probably be to use the
Union
type in the dataclasses and call it like so:Solutions & workarounds
There are quite many suggestions in this SO thread. The issue can be solved with multiple subparsers and user-side argv parsing. The following code allows me to call my script like so:
And get
The code:
The result looks correct and leverages simple_parsing argument name resulotions etc, so its quite convenient. Some caveats and ugliness:
args.encoder.encoder
, but it would be nicer to haveargs.encoder
and get the final class straight awayWhat would be better:
args.glob1.encoder
callsThe configs could look like this:
and we would get the same output like in the example.
Other stuff I tried:
I also tried to use
Enum
to store the configs for encoder types and it allows me to select correct config, but I cannot adjust the selcet config params. I tried to usechoice
but it did not accept dataclass as an argument.