[Feature Request] Simple way to create one cli to access multiple sub-cli's

[AyushExel]

🚀 Feature Request

This is more of a usage question rather than a feature request. I'm one of the maintainers of Yolov5 and I've been exploring hydra to manage all our args and hyper-params. So far I think hydra team has done amazing work on this library. I was able to simplify a lot of things. Now I want to package the repo and provide a cmd-line entry point for all the separate tasks managed by hydra. Let me explain with an example. I have multiple tasks that are all powered by hydra. Each task has 3 possible operations. Let's look at only 2 for simplicity- training and inference I can run them like this

python train.py args.epochs=2 ... # edit default params via hyda
python infer.py ...

Now I want to package this repo and create a cli interface that can access all these tasks and support the same ease of use as hydra. Something like this

yolov5 train  --args.epochs=2 # should be same as python detect/train.py --args.epochs=2 ..
yolov5 infer --args.conf=0.2

Up until now, I would use fire cli to create console entry points. But combining fire with hydra doest work. If I try to initialize fire with a function decorated by hydra main, it expects cfg as manual cmd-line input rather than hydra reading the default config.

# train.py
@hydra.main(...):
def run(cfg):
    ...
fire.Fire({
        'classify/train': yolov5.train.run,
        }) # Throws error that cfg must be passed

So, is there a clean and simple way of supporting this use case? Ideally, I'd like to use only hydra to manage this but open to combine other tools if absolutely necessary.

[AyushExel]

hey @Jasha10, maybe you know a workaround?

[Jasha10]

Hi @AyushExel, The @hydra.main function internally uses argparse (see this function). Argparse is driven by sys.argv. This means you can wrap a @hydra.main-decorated function however you like as long as sys.argv is set properly before you call the function. For example, you could try the following:

...

@hydra.main(...)
def hydra_train(cfg): ...

@hydra.main(...)
def hydra_infer(cfg): ...

def my_entry_point(...):
    if should_train:
         sys.argv = args_for_training
         hydra_train()
    elif should_infer:
         sys.argv = args_for_inference
         hydra_infer()

I am not an expert in the fire CLI, but I suspect that fire also works the same way: it probably depends on sys.argv. In principle, you could use this technique (of overwriting sys.argv) to either wrap your hydra apps in a fire app or to wrap a fire app inside of a hydra app. In either case, you'd overwrite sys.argv before calling the inner app.

I'll mention the hydra-zen library's launch function, which exposes a similar API to @hydra.main but without using sys.argv. This might be more convenient to use.

An alternative to using @hydra.main or hydra_zen.launch would be to use Hydra's compose API. That being said, migrating from the @hydra.main API to the compose api can be challenging (as currently the compose API does not have great support for the ${hydra:...} resolver or for multirun mode). Also, if you use compose then you miss out on @hydra.main's command-line completion feature.

I have multiple tasks that are all powered by hydra.

For the sake of completeness, I'll mention a final option: you could combine your multiple tasks into a single hydra task.

yolov5 mode=train foo=bar ...
yolov5 mode=infer foo=bar ...

Then in your @hydra.main function, you would inspect the cfg object to see if the user wants to train or to infer.

I hope this helps.

[AyushExel]

Okay thanks for responsing. I'll try to build the solution based on your suggestion

[Jasha10]

Cross-link to related Stack-overflow question: https://stackoverflow.com/questions/73971359/how-to-create-cli-to-access-multi-cli-commands-using-hydra

[AyushExel]

@Jasha10 hey is there any way to accept unlisted arguments? Something like kwargs - where all the unlisted args are accessible under a specified namesapce. Like - cgf.IDENTIFIER.arg1 ..

[Jasha10]

There's the plus syntax (+) for adding to the config object:

$ cd hydra/examples/tutorials/basic/your_first_hydra_app/1_simple_cli
$ python my_app.py +foo=bar ++baz=qux
foo: bar
baz: qux

The difference between + and ++ is that ++baz=qux will succeed even if there is already a baz key in the config, while +foo=bar will only succeed if the key foo is not yet present.

Ref: Modifying the Config Object

[AyushExel]

Perfect! thankyou

[AyushExel]

@Jasha10 hey I have another question. In the master config we have a device which accepts a string. It can be either empty, cpu, a string of GPU ids.. Like `device=''/'cpu'/'1,2,3,4' This works well for us currently but while passing a comma-separated string we have to follow a certain syntax in the cli..:

So I thought of supporting the list input as suggested by the hydra prompt. But passing list doesn't work. I always get this: no matches found: device=[1,2,3,4] It happens with evey key for which I try to pass a list.

[omry]

your shell is interpreting that value. try to quote it. This is documented here

$ python foo.py 'device=[1,2,3,4]'