AssertionError: assert high > low # by construction

[Sinah-astro]

I,ve been getting the following error with my target fields:

2021-12-20 22:11:17 | INFO | preprocess:transcribe_recipe | The following model sources were obtained from --input-model-recipe: 
   Columns: {'MODEL_DATA'} 
   Sky Models: None
2021-12-20 22:11:17 | INFO | ms_handler:read_xds_list | Antenna table indicates 62 antennas were present for this observation.
2021-12-20 22:11:18 | INFO | ms_handler:read_xds_list | Polarization table indicates 4 correlations are present in the measurement set - ['XX', 'XY', 'YX', 'YY'].
2021-12-20 22:11:18 | INFO | ms_handler:read_xds_list | Field table indicates phase centre is at (0.7604908605304458 -1.0289239539282171).
2021-12-20 22:11:22 | WARNING | reads:_group_datasets | Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 0 of column FLAG_CATEGORY in /home/manaka/PSZ277/msdir/PSZ2G277-PSZ2G277_76_51_74-corr.ms/table.f18'
2021-12-20 22:11:29 | INFO | ms_handler:write_xds_list | Outputs will be written to FLAG, FLAG_ROW, CCORRECTED_DATA, WEIGHT_SPECTRUM.
2021-12-20 22:11:29 | WARNING | ms:descriptor | Unable to infer 'flagcat' dimension from input variables
2021-12-20 22:12:31 | SUCCESS | executor:_execute | 74.22 seconds taken to build graph.
2021-12-21 00:28:42 | ERROR | goquartical:<module> | An error has been caught in function '<module>', process 'MainProcess' (124401), thread 'MainThread' (140707655604032):
Traceback (most recent call last):

> File "/home/manaka/software/quartical/qenv/bin/goquartical", line 8, in <module>
    sys.exit(execute())
    │   │    └ <function execute at 0x7ff8d5839f80>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/quartical/executor.py", line 27, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7ff90c71db90>
    └ <function _execute at 0x7ff8d583d050>
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/quartical/executor.py", line 136, in _execute
    scheduler=dask_opts.scheduler)
              │         └ 'threads'
              └ Dask(threads=0, workers=1, address=None, scheduler='threads')
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/base.py", line 570, in compute
    results = schedule(dsk, keys, **kwargs)
              │        │    │       └ {'num_workers': 0}
              │        │    └ [[[[[('write~FLAG-PSZ2G277-PSZ2G277_76_51_74-corr.ms-5a4738cc31746c4581d212e14e4cbbae', 0, 0, 0)]]], [('write~FLAG_ROW-PSZ2G2...
              │        └ {('write~param-8a461a0ebecaacaa48afce59f6b2dc64', 0): (subgraph_callable-9d62660d-f78f-4402-b01d-cd549e601985, ('array-2972d9...
              └ <function get at 0x7ff909b37050>
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/threaded.py", line 87, in get
    **kwargs,
      └ {}
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/local.py", line 507, in get_async
    raise_exception(exc, tb)
    │               │    └ <traceback object at 0x7ff8d549fe10>
    │               └ AssertionError()
    └ <function reraise at 0x7ff909d93e60>
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/local.py", line 315, in reraise
    raise exc
          └ AssertionError()
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/local.py", line 220, in execute_task
    result = _execute_task(task, data)
             │             │     └ {('compute_chisq-e2d6cc587a6b6a38e924cb77e1f69fbb', 0, 0): array([[1.41702608e+08, 2.43421348e+04, 2.43288418e+04, ...,
             │             │            ...
             │             └ (subgraph_callable-8de704ad-5c51-474f-a562-04af8f67ce3e, (<function concatenate_axes at 0x7ff8fa029290>, [('compute_chisq-e2d...
             └ <function _execute_task at 0x7ff909ee9050>
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           │      │                │               └ ((<function concatenate_axes at 0x7ff8fa029290>, [('compute_chisq-e2d6cc587a6b6a38e924cb77e1f69fbb', 0, 0)], [1]), (<function...
           │      │                └ {('compute_chisq-e2d6cc587a6b6a38e924cb77e1f69fbb', 0, 0): array([[1.41702608e+08, 2.43421348e+04, 2.43288418e+04, ...,
           │      │                       ...
           │      └ <function _execute_task at 0x7ff909ee9050>
           └ subgraph_callable-8de704ad-5c51-474f-a562-04af8f67ce3e
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/optimization.py", line 969, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
           │    │   │    │    │    │                │    │       └ (array([[1.41702608e+08, 2.43421348e+04, 2.43288418e+04, ...,
           │    │   │    │    │    │                │    │                 2.96196387e+04, 3.00168496e+04, 3.07717520e+04],
           │    │   │    │    │    │                │    │               ...
           │    │   │    │    │    │                │    └ <member 'inkeys' of 'SubgraphCallable' objects>
           │    │   │    │    │    │                └ subgraph_callable-8de704ad-5c51-474f-a562-04af8f67ce3e
           │    │   │    │    │    └ <member 'outkey' of 'SubgraphCallable' objects>
           │    │   │    │    └ subgraph_callable-8de704ad-5c51-474f-a562-04af8f67ce3e
           │    │   │    └ <member 'dsk' of 'SubgraphCallable' objects>
           │    │   └ subgraph_callable-8de704ad-5c51-474f-a562-04af8f67ce3e
           │    └ <function get at 0x7ff909ee90e0>
           └ <module 'dask.core' from '/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/core.py'>
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
             │             │     └ {'__dask_blockwise__0': array([[1.41702608e+08, 2.43421348e+04, 2.43288418e+04, ...,
             │             │               2.96196387e+04, 3.00168496e+04, ...
             │             └ (CPUDispatcher(<function compute_gbl_mad_and_med at 0x7ff8de79f7a0>), '__dask_blockwise__0', '__dask_blockwise__1')
             └ <function _execute_task at 0x7ff909ee9050>
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           │      │                │               └ ('__dask_blockwise__0', '__dask_blockwise__1')
           │      │                └ {'__dask_blockwise__0': array([[1.41702608e+08, 2.43421348e+04, 2.43288418e+04, ...,
           │      │                          2.96196387e+04, 3.00168496e+04, ...
           │      └ <function _execute_task at 0x7ff909ee9050>
           └ CPUDispatcher(<function compute_gbl_mad_and_med at 0x7ff8de79f7a0>)
  File "/home/manaka/software/quartical/qenv/lib/python3.7/site-packages/numba/np/arraymath.py", line 1224, in _select_two
    assert high > low  # by construction

AssertionError: assert high > low  # by construction

I tried to look up a solution or cause but to no success. Any ideas what could be causing this and how to fix it? The config file: quartical1.txt

[JSKenyon]

Hi! I am back at work. This error is likely coming from the MAD flagging on a fully flagged chunk. This could be one of two things:

• The input data is fully flagged in the chunk. I will figure out a fix for this case, but it should be trivial.
• The input data has been overflagged by QuartiCal. This can happen if the solver has failed to find solutions and commonly happens when starting a chain with delays (K). This is due to the fact that until such time as the phases are approximately aligned, the delay is unlikely to converge. I am looking at trying to make QuartiCal smarter in this regard.

In terms of your path going forward, could you please check are report the percentage of flagged data in your measurement set? This will tell us whether or not QuartiCal has ruined your flags.

[JSKenyon]

135 should fix this problem. I will leave this open until you can confirm @Sinah-astro. Please be sure to update to the latest main.

[JSKenyon]

Closing for now @Sinah-astro. Please comment if you run into the issue again.