search

ratt-ru / QuartiCal

CubiCal, but with greater power.
MIT License
8 stars 4 forks source link

omegaconf.errors.ConfigKeyError: Key 'reweighting_flag_threshold' not in 'Solver' #207

Closed Kincaidr closed 2 years ago

Kincaidr commented 2 years ago

Hi,

When using the Quartical main branch i get:

2022-10-11 16:05:49.611 | ERROR    | __main__:<module>:11 - An error has been caught in function '<module>', process 'MainProcess' (4108909), thread 'MainThread' (140154637940544):
Traceback (most recent call last):

> File "/home/kincaid/stimela_venv/bin/goquartical", line 11, in <module>
    load_entry_point('quartical', 'console_scripts', 'goquartical')()
    └ <function load_entry_point at 0x7f784af1ad30>

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/executor.py", line 32, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7f784ae64670>
    └ <function _execute at 0x7f77a9c9d670>

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/executor.py", line 41, in _execute
    opts = parser.parse_inputs()
           │      └ <function parse_inputs at 0x7f77b18f1160>
           └ <module 'quartical.config.parser' from '/vault-ike/kincaid/Software/QuartiCal/quartical/config/parser.py'>

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/config/parser.py", line 122, in parse_inputs
    config = oc.merge(config, *additional_config)
             │  │     │        └ [{'input_ms': {'path': '/vault-ike/kincaid/xova_analysis_new_bda/xova_bda/Abell3376_bda_0.84.ms', 'data_column': 'DATA', 'wei...
             │  │     └ {'input_ms': {'path': '???', 'data_column': 'DATA', 'sigma_column': None, 'weight_column': None, 'time_chunk': '0', 'freq_chu...
             │  └ <staticmethod object at 0x7f7801f95cd0>
             └ <class 'omegaconf.omegaconf.OmegaConf'>

  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 268, in merge
    target.merge_with(*configs[1:])
    │      │           └ ({'input_ms': {'path': '???', 'data_column': 'DATA', 'sigma_column': None, 'weight_column': None, 'time_chunk': '0', 'freq_ch...
    │      └ <function BaseContainer.merge_with at 0x7f7801fea940>
    └ {'input_ms': {'path': '/vault-ike/kincaid/xova_analysis_new_bda/xova_bda/Abell3376_bda_0.84.ms', 'data_column': 'DATA', 'sigm...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 490, in merge_with
    self._format_and_raise(key=None, value=None, cause=e)
    │    └ <function Node._format_and_raise at 0x7f780204bc10>
    └ {'input_ms': {'path': '/vault-ike/kincaid/xova_analysis_new_bda/xova_bda/Abell3376_bda_0.84.ms', 'data_column': 'DATA', 'sigm...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7f7802048940>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/_utils.py", line 820, in format_and_raise
    _raise(ex, cause)
    │      │   └ ConfigKeyError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    refere...
    │      └ ConfigKeyError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    refere...
    └ <function _raise at 0x7f78020488b0>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/_utils.py", line 798, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ConfigKeyError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    refere...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 488, in merge_with
    self._merge_with(*others)
    │    │            └ ({'input_ms': {'path': '/vault-ike/kincaid/xova_analysis_new_bda/xova_bda/Abell3376_bda_0.84.ms', 'data_column': 'DATA', 'wei...
    │    └ <function BaseContainer._merge_with at 0x7f7801fea9d0>
    └ {'input_ms': {'path': '/vault-ike/kincaid/xova_analysis_new_bda/xova_bda/Abell3376_bda_0.84.ms', 'data_column': 'DATA', 'sigm...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 512, in _merge_with
    BaseContainer._map_merge(self, other)
    │             │          │     └ {'input_ms': {'path': '/vault-ike/kincaid/xova_analysis_new_bda/xova_bda/Abell3376_bda_0.84.ms', 'data_column': 'DATA', 'weig...
    │             │          └ {'input_ms': {'path': '/vault-ike/kincaid/xova_analysis_new_bda/xova_bda/Abell3376_bda_0.84.ms', 'data_column': 'DATA', 'sigm...
    │             └ <staticmethod object at 0x7f7801feb160>
    └ <class 'omegaconf.basecontainer.BaseContainer'>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 397, in _map_merge
    dest_node._merge_with(src_node)
    │         │           └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'reweighting_flag_threshold': 0.0, 'threads'...
    │         └ <function BaseContainer._merge_with at 0x7f7801fea9d0>
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 512, in _merge_with
    BaseContainer._map_merge(self, other)
    │             │          │     └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'reweighting_flag_threshold': 0.0, 'threads'...
    │             │          └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
    │             └ <staticmethod object at 0x7f7801feb160>
    └ <class 'omegaconf.basecontainer.BaseContainer'>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 430, in _map_merge
    dest[key] = src._get_node(key)
    │    │      │   │         └ 'reweighting_flag_threshold'
    │    │      │   └ <function DictConfig._get_node at 0x7f7801ffedc0>
    │    │      └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'reweighting_flag_threshold': 0.0, 'threads'...
    │    └ 'reweighting_flag_threshold'
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 310, in __setitem__
    self._format_and_raise(
    │    └ <function Node._format_and_raise at 0x7f780204bc10>
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7f7802048940>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/_utils.py", line 820, in format_and_raise
    _raise(ex, cause)
    │      │   └ ConfigAttributeError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    ...
    │      └ ConfigKeyError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    refere...
    └ <function _raise at 0x7f78020488b0>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/_utils.py", line 798, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ConfigKeyError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    refere...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 308, in __setitem__
    self.__set_impl(key=key, value=value)
    │                   │          └ 0.0
    │                   └ 'reweighting_flag_threshold'
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 318, in __set_impl
    self._set_item_impl(key, value)
    │    │              │    └ 0.0
    │    │              └ 'reweighting_flag_threshold'
    │    └ <function BaseContainer._set_item_impl at 0x7f7801feaa60>
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 543, in _set_item_impl
    self._validate_set(key, value)
    │    │             │    └ 0.0
    │    │             └ 'reweighting_flag_threshold'
    │    └ <function DictConfig._validate_set at 0x7f7801fee8b0>
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 180, in _validate_set
    target = self._get_node(key) if key is not None else self
             │    │         │       │                    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
             │    │         │       └ 'reweighting_flag_threshold'
             │    │         └ 'reweighting_flag_threshold'
             │    └ <function DictConfig._get_node at 0x7f7801ffedc0>
             └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
    │    │             └ 'reweighting_flag_threshold'
    │    └ <function DictConfig._validate_get at 0x7f7801fee700>
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
    │    └ <function Node._format_and_raise at 0x7f780204bc10>
    └ {'terms': ['K'], 'iter_recipe': [100], 'propagate_flags': True, 'robust': False, 'threads': 1, 'convergence_fraction': 0.99, ...
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7f7802048940>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/_utils.py", line 900, in format_and_raise
    _raise(ex, cause)
    │      │   └ ConfigAttributeError("Key 'reweighting_flag_threshold' not in 'Solver'")
    │      └ ConfigAttributeError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    ...
    └ <function _raise at 0x7f78020488b0>
  File "/home/kincaid/stimela_venv/lib/python3.8/site-packages/omegaconf/_utils.py", line 798, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ConfigAttributeError("Key 'reweighting_flag_threshold' not in 'Solver'\n    full_key: solver.reweighting_flag_threshold\n    ...

omegaconf.errors.ConfigKeyError: Key 'reweighting_flag_threshold' not in 'Solver'
    full_key: solver.reweighting_flag_threshold
    reference_type=Solver
    object_type=Solver

Command:

goquartical input_ms.path=Abell3376_timechannel_t11_c4.ms input_ms.data_column=DATA input_ms.time_chunk=128 input_ms.freq_chunk=0 input_ms.is_bda=False input_ms.group_by=['SCAN_NUMBER','FIELD_ID','DATA_DESC_ID'] input_ms.select_fields=[] input_ms.select_ddids=[] input_ms.select_uv_range=[0.0,0.0] input_model.recipe=MODEL_DATA input_model.beam_l_axis=X input_model.beam_m_axis=Y input_model.invert_uvw=True input_model.source_chunks=500 input_model.apply_p_jones=True output.gain_directory=gains.qc output.log_directory=logs.qc output.overwrite=True output.products=['corrected_data'] output.columns=['SELFCAL_DATA'] output.flags=True output.apply_p_jones_inv=True mad_flags.enable=False mad_flags.threshold_bl=5.0 mad_flags.threshold_global=5.0 mad_flags.max_deviation=5.0 solver.terms=['K'] solver.iter_recipe=[100] solver.propagate_flags=True solver.robust=False solver.reweighting_flag_threshold=0.0 solver.threads=1 solver.convergence_fraction=0.99 solver.convergence_criteria=1e-06 solver.reference_antenna=0 dask.workers=1 dask.scheduler=threads K.type=phase K.solve_per=antenna K.direction_dependent=False K.time_interval=4 K.freq_interval=0 K.interp_mode=reim K.interp_method=2dlinear K.respect_scan_boundaries=True K.initial_estimate=False
landmanbester commented 2 years ago

That functionality only exists in the stimelation branch. You can just take the solver.reweighting_flag_threshold=0.0 out of the command and it should work

Kincaidr commented 2 years ago

Thanks!

o-smirnov commented 2 years ago

On a side note @JSKenyon, we might need to open an issue for friendlier exception reporting... I've been fighting the same battle in stimela2, and I think I"m on top of it now, so let's compare notes...