search

DiamondLightSource / httomo

High-throughput tomography pipeline
https://diamondlightsource.github.io/httomo/
Other
5 stars 4 forks source link

Fix incorrect YAML templates path construction in YAML checker #302

Closed yousefmoazzam closed 5 months ago

yousefmoazzam commented 5 months ago

There's a subtle bug in the YAML checker that is not found unless the YAML checker is executed from a current working directory other than the httomo root directory.

There shouldn't be double quotation marks around __file__ in order to refer to the file in the following: https://github.com/DiamondLightSource/httomo/blob/f53709ac6350898dcfe874961241d98e3a722f84/httomo/yaml_checker.py#L336

Doing os.path.abspath("__file__") is trying to refer to a file called __file__ in the current working directory, which is going to change depending on where the user is navigated to in the terminal.

If I run the YAML checker in the httomo root dir, it works:

(/dls/science/users/twi18192/conda-envs/httomo) [twi18192@pc0074 httomo (jessica-refactor-task-runner)]$ pwd
/dls/science/users/twi18192/httomo
(/dls/science/users/twi18192/conda-envs/httomo) [twi18192@pc0074 httomo (jessica-refactor-task-runner)]$ python -m httomo check tests/samples/pipeline_template_examples/pipeline_gpu1.yaml 
Checking that the YAML_CONFIG is properly indented and has valid mappings and tags...
Sanity check of the YAML_CONFIG was successfully done...

Checking that the first method in the pipeline is a loader...
Loader check successful!!

parent_dir is /dls/science/users/twi18192/httomo
templates_dir is /dls/science/users/twi18192/httomo/yaml_templates
parent_dir is /dls/science/users/twi18192/httomo
templates_dir is /dls/science/users/twi18192/httomo/yaml_templates

YAML validation successful!! Please feel free to use the `run` command to run the pipeline.

But if I cd into the tests/ directory and try to run the checker, it fails:

(/dls/science/users/twi18192/conda-envs/httomo) [twi18192@pc0074 httomo (jessica-refactor-task-runner)]$ cd tests/
(/dls/science/users/twi18192/conda-envs/httomo) [twi18192@pc0074 tests (jessica-refactor-task-runner)]$ pwd
/dls/science/users/twi18192/httomo/tests
(/dls/science/users/twi18192/conda-envs/httomo) [twi18192@pc0074 tests (jessica-refactor-task-runner)]$ python -m httomo check samples/pipeline_template_examples/pipeline_gpu1.yaml 
Checking that the YAML_CONFIG is properly indented and has valid mappings and tags...
Sanity check of the YAML_CONFIG was successfully done...

Checking that the first method in the pipeline is a loader...
Loader check successful!!

parent_dir is /dls/science/users/twi18192/httomo/tests
templates_dir is /dls/science/users/twi18192/httomo/tests/yaml_templates
Traceback (most recent call last):
  File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/dls/science/users/twi18192/httomo/httomo/__main__.py", line 4, in <module>
    main()
  File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/dls/science/users/twi18192/httomo/httomo/cli.py", line 45, in check
    return validate_yaml_config(yaml_config, in_data)
  File "/dls/science/users/twi18192/httomo/httomo/yaml_checker.py", line 392, in validate_yaml_config
    do_methods_exist = check_methods_exist_in_templates(conf)
  File "/dls/science/users/twi18192/httomo/httomo/yaml_checker.py", line 136, in check_methods_exist_in_templates
    template_yaml_files = _get_template_yaml(conf, packages)
  File "/dls/science/users/twi18192/httomo/httomo/yaml_checker.py", line 341, in _get_template_yaml
    assert os.path.exists(templates_dir), "Dev error: expected YAML templates dir to exist"
AssertionError: Dev error: expected YAML templates dir to exist
yousefmoazzam commented 5 months ago

Changing the line

parent_dir = os.path.dirname(os.path.abspath("__file__"))

to

parent_dir = Path(__file__).parent.parent

fixes the issue for me, so maybe we can do this change (or something similar) at some point.

yousefmoazzam commented 5 months ago

Fixed by merge of #201 into the gpuloop branch