search

ratt-ru / QuartiCal

CubiCal, but with greater power.
MIT License
8 stars 4 forks source link

Invalid/incorrect arguments may produce unclear error messages #180

Closed landmanbester closed 1 year ago

landmanbester commented 2 years ago

Describe the bug I ran into the above error while trying to reproduce https://github.com/ratt-ru/QuartiCal/issues/179. The full back trace

goquartical reproduce_issue179.yaml
2022-07-20 10:58:40.239 | INFO     | quartical.config.parser:parse_inputs:109 - User defined config file: reproduce_issue179.yaml
2022-07-20 10:58:40.266 | ERROR    | __main__:<module>:33 - An error has been caught in function '<module>', process 'MainProcess' (355742), thread 'MainThread' (140131489408832):
Traceback (most recent call last):

> File "/home/bester/.venv/qcal/bin/goquartical", line 33, in <module>
    sys.exit(load_entry_point('quartical', 'console_scripts', 'goquartical')())
    │   │    └ <function importlib_load_entry_point at 0x7f72e79d01f0>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>

  File "/home/bester/software/QuartiCal/quartical/executor.py", line 32, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7f72e7514f40>
    └ <function _execute at 0x7f718f0f58b0>

  File "/home/bester/software/QuartiCal/quartical/executor.py", line 41, in _execute
    opts = parser.parse_inputs()
           │      └ <function parse_inputs at 0x7f7197356040>
           └ <module 'quartical.config.parser' from '/home/bester/software/QuartiCal/quartical/config/parser.py'>

  File "/home/bester/software/QuartiCal/quartical/config/parser.py", line 122, in parse_inputs
    config = oc.merge(config, *additional_config)
             │  │     │        └ [{'input_ms': {'path': 'ms1_primary_subset.zarr', 'data_column': 'DATA', 'sigma_column': 'SIGMA_SPECTRUM', 'weight_column': N...
             │  │     └ {'input_ms': {'path': '???', 'data_column': 'DATA', 'sigma_column': None, 'weight_column': None, 'time_chunk': '0', 'freq_chu...
             │  └ <staticmethod object at 0x7f72e0d98ac0>
             └ <class 'omegaconf.omegaconf.OmegaConf'>

  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 268, in merge
    target.merge_with(*configs[1:])
    │      │           └ ({'input_ms': {'path': '???', 'data_column': 'DATA', 'sigma_column': None, 'weight_column': None, 'time_chunk': '0', 'freq_ch...
    │      └ <function BaseContainer.merge_with at 0x7f72e0de9160>
    └ {'input_ms': {'path': 'ms1_primary_subset.zarr', 'data_column': 'DATA', 'sigma_column': 'SIGMA_SPECTRUM', 'weight_column': No...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 490, in merge_with
    self._format_and_raise(key=None, value=None, cause=e)
    │    └ <function Node._format_and_raise at 0x7f72e0dd53a0>
    └ {'input_ms': {'path': 'ms1_primary_subset.zarr', 'data_column': 'DATA', 'sigma_column': 'SIGMA_SPECTRUM', 'weight_column': No...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7f72e0dc70d0>
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/_utils.py", line 793, in format_and_raise
    _raise(ex, cause)
    │      │   └ ValidationError('Invalid value assigned: AnyNode is not a ListConfig, list or tuple.\n    full_key: output.products\n    refe...
    │      └ ValidationError('Invalid value assigned: AnyNode is not a ListConfig, list or tuple.\n    full_key: output.products\n    refe...
    └ <function _raise at 0x7f72e0dc7040>
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ValidationError('Invalid value assigned: AnyNode is not a ListConfig, list or tuple.\n    full_key: output.products\n    refe...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 488, in merge_with
    self._merge_with(*others)
    │    │            └ ({'input_ms': {'path': 'ms1_primary_subset.zarr', 'data_column': 'DATA', 'sigma_column': 'SIGMA_SPECTRUM', 'weight_column': N...
    │    └ <function BaseContainer._merge_with at 0x7f72e0de91f0>
    └ {'input_ms': {'path': 'ms1_primary_subset.zarr', 'data_column': 'DATA', 'sigma_column': 'SIGMA_SPECTRUM', 'weight_column': No...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 512, in _merge_with
    BaseContainer._map_merge(self, other)
    │             │          │     └ {'input_ms': {'path': 'ms1_primary_subset.zarr', 'data_column': 'DATA', 'sigma_column': 'SIGMA_SPECTRUM', 'weight_column': No...
    │             │          └ {'input_ms': {'path': 'ms1_primary_subset.zarr', 'data_column': 'DATA', 'sigma_column': 'SIGMA_SPECTRUM', 'weight_column': No...
    │             └ <staticmethod object at 0x7f72e0ddef10>
    └ <class 'omegaconf.basecontainer.BaseContainer'>
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 397, in _map_merge
    dest_node._merge_with(src_node)
    │         │           └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': 'residual', 'columns': 'RESIDUALX'...
    │         └ <function BaseContainer._merge_with at 0x7f72e0de91f0>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': [], 'columns': None, 'flags': True...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 512, in _merge_with
    BaseContainer._map_merge(self, other)
    │             │          │     └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': 'residual', 'columns': 'RESIDUALX'...
    │             │          └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': [], 'columns': None, 'flags': True...
    │             └ <staticmethod object at 0x7f72e0ddef10>
    └ <class 'omegaconf.basecontainer.BaseContainer'>
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 399, in _map_merge
    dest.__setitem__(key, src_node)
    │    │           │    └ 'residual'
    │    │           └ 'products'
    │    └ <function DictConfig.__setitem__ at 0x7f72e0d810d0>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': [], 'columns': None, 'flags': True...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 316, in __setitem__
    self._format_and_raise(key=key, value=value, cause=e)
    │    │                     │          └ 'residual'
    │    │                     └ 'products'
    │    └ <function Node._format_and_raise at 0x7f72e0dd53a0>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': [], 'columns': None, 'flags': True...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7f72e0dc70d0>
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/_utils.py", line 873, in format_and_raise
    _raise(ex, cause)
    │      │   └ ValidationError('Invalid value assigned: AnyNode is not a ListConfig, list or tuple.')
    │      └ ValidationError('Invalid value assigned: AnyNode is not a ListConfig, list or tuple.\n    full_key: output.products\n    refe...
    └ <function _raise at 0x7f72e0dc7040>
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ValidationError('Invalid value assigned: AnyNode is not a ListConfig, list or tuple.\n    full_key: output.products\n    refe...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 310, in __setitem__
    self.__set_impl(key=key, value=value)
    │                   │          └ 'residual'
    │                   └ 'products'
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': [], 'columns': None, 'flags': True...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 320, in __set_impl
    self._set_item_impl(key, value)
    │    │              │    └ 'residual'
    │    │              └ 'products'
    │    └ <function BaseContainer._set_item_impl at 0x7f72e0de9280>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': [], 'columns': None, 'flags': True...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 602, in _set_item_impl
    self.__dict__["_content"][key]._set_value(value)
    │    │                    │               └ 'residual'
    │    │                    └ 'products'
    │    └ <attribute '__dict__' of 'Node' objects>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'overwrite': False, 'products': [], 'columns': None, 'flags': True...
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/listconfig.py", line 612, in _set_value
    raise e
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/listconfig.py", line 608, in _set_value
    self._set_value_impl(value, flags)
    │    │               │      └ None
    │    │               └ 'residual'
    │    └ <function ListConfig._set_value_impl at 0x7f72e0d90700>
    └ []
  File "/home/bester/.venv/qcal/lib/python3.8/site-packages/omegaconf/listconfig.py", line 640, in _set_value_impl
    raise ValidationError(msg)
          │               └ 'Invalid value assigned: AnyNode is not a ListConfig, list or tuple.'
          └ <class 'omegaconf.errors.ValidationError'>

omegaconf.errors.ValidationError: Invalid value assigned: AnyNode is not a ListConfig, list or tuple.
    full_key: output.products
    reference_type=BaseConfig_output
    object_type=BaseConfig_output

Data description (if applicable) I've left the data and parset on oates in /home/bester/for_jon

To Reproduce run goquartical on the provided parset file

Expected behavior I expected t reproduce https://github.com/ratt-ru/QuartiCal/issues/179.

Version This is in a fresh python3.8 virtualenv on the stimelation branch

$ pip freeze
aiobotocore==2.3.4
aiohttp==3.8.1
aioitertools==0.10.0
aiosignal==1.2.0
antlr4-python3-runtime==4.9.3
appdirs==1.4.4
asciitree==0.3.3
astLib==0.11.7
astro-kittens==1.4.6
astro-tigger-lsm==1.7.2
astropy==5.1
async-timeout==4.0.2
attrs==21.4.0
bokeh==2.4.3
botocore==1.24.21
certifi==2022.6.15
charset-normalizer==2.1.0
cli-ui==0.17.0
click==8.1.3
cloudpickle==2.1.0
codex-africanus==0.3.2
colorama==0.4.5
Columnar==1.4.1
commonmark==0.9.1
configparser==5.2.0
contextlib2==21.6.0
dask==2022.7.0
dask-ms==0.2.10
decorator==5.1.1
dill==0.3.5.1
distributed==2022.7.0
docopt==0.6.2
fasteners==0.17.3
frozenlist==1.3.0
fsspec==2022.5.0
future==0.18.2
HeapDict==1.0.1
idna==3.3
iniconfig==1.1.1
Jinja2==3.1.2
jmespath==1.0.1
llvmlite==0.38.1
locket==1.0.0
loguru==0.6.0
MarkupSafe==2.1.1
msgpack==1.0.4
multidict==6.0.2
multiprocess==0.70.13
munch==2.5.0
numba==0.55.2
numcodecs==0.10.0
numpy==1.22.4
omegaconf==2.2.2
packaging==21.3
pandas==1.4.3
partd==1.2.0
pathos==0.2.9
Pillow==9.2.0
pluggy==1.0.0
pox==0.3.1
ppft==1.7.6.5
psutil==5.9.1
py==1.11.0
pydantic==1.9.1
pyerfa==2.0.0.1
Pygments==2.12.0
pyparsing==3.0.9
pytest==7.1.2
python-casacore==3.5.0
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
-e git+ssh://git@github.com/ratt-ru/QuartiCal.git@07f07fd53e3185f660f12744a37f1311bd606de7#egg=quartical
requests==2.28.1
rich==12.5.1
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
s3fs==2022.5.0
schema==0.7.5
scipy==1.8.1
six==1.16.0
sortedcontainers==2.4.0
stimela==2.0rc2
tabulate==0.8.10
tblib==1.7.0
tbump==6.9.0
tomli==2.0.1
tomlkit==0.11.1
toolz==0.12.0
tornado==6.1
typing_extensions==4.3.0
Unidecode==1.3.4
urllib3==1.26.10
wcwidth==0.2.5
wrapt==1.14.1
xarray==2022.3.0
yarl==1.7.2
zarr==2.12.0
zict==2.2.0
JSKenyon commented 2 years ago

This is user error, although the error message could suck less. Both output.products and output.columns expect a list (even if it only has one element). Could you please try again, with either of the following:

output:
  products: [residual]
  columns: [RESIDUALX]

or

output:
  products:
    - residual
  columns:
    - RESIDUALX
landmanbester commented 2 years ago

Oops, my bad. Too much haste and too little speed. Thanks

JSKenyon commented 1 year ago

Fixed in #266.