[Bug] Basic example on Jupyter doesn't work

[GlidingRaven]

🐛 Bug (?)

Description

I can't reproduce the Basic example on Jupyter notebook.

Checklist

• [x] I checked on the latest version of Hydra
• [ ] I created a minimal repro (See this for tips).

To reproduce

Config: (conf/config.yaml)

db:
  driver: mysql
  user: omry
  pass: secret

Run a cell:

import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="conf", config_name="config")
def my_app(cfg : DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()

Output:

---------------------------------------------------------------------------
SystemExit                                Traceback (most recent call last)
<ipython-input-7-e492f95800ae> in <module>()
      8 
      9 if __name__ == "__main__":
---> 10     app()
     11 # %tb

C:\Users\11\AppData\Roaming\Python\Python36\site-packages\hydra\main.py in decorated_main(cfg_passthrough)
     50                     task_function=task_function,
     51                     config_path=config_path,
---> 52                     config_name=config_name,
     53                 )
     54 

C:\Users\11\AppData\Roaming\Python\Python36\site-packages\hydra\_internal\utils.py in _run_hydra(args_parser, task_function, config_path, config_name)
    307     from .hydra import Hydra
    308 
--> 309     args = args_parser.parse_args()
    310     if args.config_name is not None:
    311         config_name = args.config_name

C:\Users\11\AppData\Roaming\Python\Python36\site-packages\argparse.py in parse_args(self, args, namespace)
   1726         if argv:
   1727             msg = _('unrecognized arguments: %s')
-> 1728             self.error(msg % ' '.join(argv))
   1729         return args
   1730 

C:\Users\11\AppData\Roaming\Python\Python36\site-packages\argparse.py in error(self, message)
   2390         """
   2391         self.print_usage(_sys.stderr)
-> 2392         self.exit(2, _('%s: error: %s\n') % (self.prog, message))

C:\Users\11\AppData\Roaming\Python\Python36\site-packages\argparse.py in exit(self, status, message)
   2378         if message:
   2379             self._print_message(message, _sys.stderr)
-> 2380         _sys.exit(status)
   2381 
   2382     def error(self, message):

SystemExit: 2

System information

• Hydra Version : 1.1.1
• Argparse Version : 1.4.0
• Python version : Python 3.6.1 :: Anaconda 4.4.0 (64-bit)
• Virtual environment type and version : virtual
• Operating system : Windows 10

Additional context

Is it possible to run the Hydra on Jupyter?

[Jasha10]

Hi @GlidingRaven, The issue here is related to argparse and sys.argv. Hydra's @hydra.main function relies on argparse to read from sys.argv, and in Jupyter notebooks this is problematic.

I'd recommend using the compose API when working with Jupyter. See this example app using a Jupyter notebook.

It would also be possible to manually patch sys.argv before calling my_app, though this is not so elegant.

[pozimekSG]

For those who want to write scripts with Hydra that work both inside Ipython and in the terminal, the following should work without sacrificing any CLI functionality:

import hydra
from omegaconf import DictConfig, OmegaConf
from hydra import initialize, compose

def is_ipython():
    try:
        shell = get_ipython().__class__.__name__ #DO NOT IMPORT get_ipython
        return True
    except NameError:
        return False

def run_with_config(main_fn, cfg_name = "default_config"):
    if is_ipython():
        with initialize(".", None):
            cfg = compose(cfg_name)
        main_fn(cfg)
    else:
        decorator = hydra.main(version_base=None, config_path=".", 
                               config_name=cfg_name)
        decorator(main_fn)()

def main(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    run_with_config(main)

Note that the config_path passed to hydra.main has to be relative to the location of the main_fn, whereas the config_path passed to initialize is relative to the script where run_with_config is defined. This becomes relevant if you refactor the non-main functions into another script.

[chuntian236]

Try this: import sys sys.argv=['self.py']

(Source: https://stackoverflow.com/questions/13657199/use-sys-argv-inside-ipython3-notebook)