Thoughts on lightning CLI / jsonargparse?

[turian]

I'm a hydra fanboy, but a recent discussion on the lightning slack suggested to me that lightning CLI / jsonargparse might be an even better choice for lightning.

I haven't investigated deeply but the one feature that stuck out was that it automagically saves your hyperparams for later.

Just curious your thoughts on this too, if you've used them.

[tshu-w]

I prefer LightningCLI since it's built-in in Pytorch Lightning. Here is a lightning-template motivated by ashleve fantastic project. The limitation I encountered so far is that LightningCLI doesn't support sweep, but I can implement it quickly with jsonargparse and ray tune.

[ashleve]

automagically saves your hyperparams for later

@turian could you please refer me to some docs because I couldn't find what this means.

Unfortunately, I do not have much experience with lightning cli.

What I like about hydra (which I think lightningCLI doesn't offer, please correct me if I'm wrong) is:

• automatic and customizable output dir
• direct control over object instantiation
• convenient version control of specific combinations of hyperparameters with yaml submodules containing # @package _global_ at the top
• multi-run and easy to use baseline hyperparameter search like optuna plugin

What I'm missing the most is:

• a well thought out way to prevent issues with output dir in DDP mode - feels like this should be achievable with some environment variables
• ability to define submodes, like what LightningCLI does with train / test / fit / etc.
• more command line features like extending config lists

[tshu-w]

Nice comparison! I thought jsonargparse can "direct control over object instantiation" either If I understand correctly.

[ashleve]

What I meant you can freely use hydra.utils.instantiate(cfg) in any part of your code, which is useful because sometimes you might need to control the order of instantation or pass extra parameters available only at some point of the pipeline, e.g. hydra.utils.instantiate(cfg.model, vocab_size=datamodule.vocab_size)

Is this also possible with jsonargparse? (I'm not familiar with it)

[tshu-w]

To some extent, I think, jsonargparse does instantiate for you, which means you cannot "directly" control. However, jsonargparse provides link_arguments with apple_on='instantiate' and compute_fn, will automatically determine the instantiate order based on the links (must be a directed acyclic graph).

[turian]

@ashleve I'm not the export on this, but if you follow the slack link in my original comment, you can see the thread. (Just run slack in the browser if you don't wanna install it.)

@mauvilsa and @jlperla seem to understand the technologies very well, and I found his arguments persuasive enough that I'd like to try them for another project, just to get my feelings on them. I've invited them also to drop in on this comment thread in case they would like to share their feedback.

[turian]

@tshu-w https://github.com/tshu-w/lightning-template is cool! How different is it from lightning-hydra-template ? Would it be possible to merge the two?

[tshu-w]

@turian The biggest difference is whether to use hydra or lightningcli (jsonargparse) to manage and launch experiments, which provide similar but distinct functionality, as summarized by @ashleve. I don't see the possibility of merging for now, as it's essentially just using different tools to do the same thing, but can borrow functionality from each other.

[mauvilsa]

I don't like much the hydra vs. LightningCLI comparison. I would say that hydra is closer to jsonargparse or other configuration packages/frameworks. One goal of LightningCLI was that for many use cases, completely avoid the need for people to choose and learn how to implement configurability with one of the many existing frameworks. Implementing a CLI with LightningCLI in many cases is one line of code or very few, which already provides many features. If someone wants more advanced features, then certainly there might be a need to learn a bit about jsonargparse, but still much less demanding.

Something could be implemented on top of hydra with similar goals to LightningCLI and then it would be a more appropriate comparison. This template could be considered something on top of hydra. However, there is a big difference between a template and a feature part of a well maintained library. Templates can be very useful, in particular for newbies who wouldn't know how to structure their code from scratch. This specific template is certainly very useful to learn how to use hydra and pytorch-lightning. As a newbie to hydra, it is much easier learning from this than going to the hydra documentation. A downside of a template might be that people end up implementing some things again and again, which goes against the "don't repeat yourself" principle. Blocks of code that are always the same or very similar, is convenient to generalize and add to a library. This way the code becomes well tested, with a well defined interface and upgrade process including deprecation periods. Improvements to a template are less likely to reach code that used an older version of the template. An example could be automatic save of the config with some guarantees of reproducibility. LightningCLI has it and for hydra+pytorch-lighting it shouldn't be in a template.

For someone who is already very familiar with hydra, continuing to use hydra instead of LightningCLI could be preferable. But still I would encourage everyone to look at LightningCLI to understand what it provides and maybe suggest improvements.

[mauvilsa]

automagically saves your hyperparams for later

@turian could you please refer me to some docs because I couldn't find what this means.

Unfortunately, I do not have much experience with lightning cli.

@ashleve read the first section in https://pytorch-lightning.readthedocs.io/en/1.6.5/common/lightning_cli.html#lightningcli

What I like about hydra (which I think lightningCLI doesn't offer, please correct me if I'm wrong) is:

* automatic and customizable output dir

Indeed LightningCLI does not work like this.

* direct control over object instantiation

As mentioned before, see link_arguments

* convenient version control of specific combinations of hyperparameters with yaml submodules containing `# @package _global_` at the top

I am not entirely sure what this means, I guess due to my lack of experience with hydra. With LightningCLI for each run of fit, all of the configuration (model, data, trainer, callbacks) is automatically saved. And it is saved in a way that there is a guarantee of reproducibility. There is no need to add anything. This is enabled by default with the option to disable.

* multi-run and easy to use baseline hyperparameter search like optuna plugin

From what I understand this is already possible with https://github.com/Lightning-AI/lightning-hpo. Not part of LightinigCLI but on top of it. Hopefully this will be improved significantly, getting similar features to LightningCLI. For example automatic save of the sweep settings. Also see https://github.com/mauvilsa/ray-tune-cli