ratt-ru / QuartiCal

CubiCal, but with greater power.
MIT License
8 stars 4 forks source link

BDA error: ValueError(f"No GCD was found for {arr}. #185

Closed Kincaidr closed 2 years ago

Kincaidr commented 2 years ago

Hi,

I am trying the bda option on Quartical: input_ms.is_bda=True for my bda MS with this command:

goquartical input_ms.path=Abell3376_bda_0.7.ms input_ms.data_column=DATA input_ms.is_bda=True input_ms.weight_column=WEIGHT_SPECTRUM input_ms.time_chunk=128 input_ms.freq_chunk=0  input_ms.group_by=['SCAN_NUMBER','FIELD_ID','DATA_DESC_ID'] input_ms.select_fields=[] input_ms.select_ddids=[] input_ms.select_uv_range=[0.0,0.0] input_model.recipe=MODEL_DATA input_model.beam_l_axis=X input_model.beam_m_axis=Y input_model.invert_uvw=True input_model.source_chunks=500 input_model.apply_p_jones=True output.gain_directory=gains.qc output.log_directory=logs.qc output.overwrite=True output.products=['corrected_data','corrected_residual'] output.columns=['CORRECTED_DATA','CORRECTED_RESIDUAL'] output.flags=True output.apply_p_jones_inv=True output.net_gain=False mad_flags.enable=False mad_flags.threshold_bl=5.0 mad_flags.threshold_global=5.0 mad_flags.max_deviation=5.0 solver.terms=['K'] solver.iter_recipe=[100] solver.propagate_flags=True solver.robust=False solver.reweighting_flag_threshold=0.0 solver.threads=1 solver.convergence_fraction=0.99 solver.convergence_criteria=1e-06 solver.reference_antenna=0 dask.workers=1 dask.scheduler=threads K.type=phase K.solve_per=antenna K.direction_dependent=False K.time_interval=4 K.freq_interval=0 K.interp_mode=reim K.interp_method=2dlinear K.respect_scan_boundaries=True K.initial_estimate=True

I am getting this error:

2022-07-31 13:29:01 | INFO | preprocess:transcribe_recipe | The following model sources were obtained from --input-model-recipe: 
   Columns: {'MODEL_DATA'} 
   Sky Models: None
2022-07-31 13:29:01 | INFO | ms_handler:read_xds_list | Antenna table indicates 60 antennas were present for this observation.
2022-07-31 13:29:01 | INFO | ms_handler:read_xds_list | Polarization table indicates 2 correlations are present in the measurement set - ['XX', 'YY'].
2022-07-31 13:29:01 | INFO | ms_handler:read_xds_list | Field table indicates phase centre is at (1.5822391417032844 -0.6969439072790133).
2022-07-31 13:29:01 | ERROR | goquartical:<module> | An error has been caught in function '<module>', process 'MainProcess' (31557), thread 'MainThread' (140597231429440):
Traceback (most recent call last):

> File "/home/kincaid/venv/stimela_venv/bin/goquartical", line 33, in <module>
    sys.exit(load_entry_point('quartical', 'console_scripts', 'goquartical')())
    │   │    └ <function importlib_load_entry_point at 0x7fdf56c0b1f0>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/executor.py", line 32, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7fdf56bcd310>
    └ <function _execute at 0x7fde25490af0>

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/executor.py", line 98, in _execute
    data_xds_list, ref_xds_list = read_xds_list(model_columns, ms_opts)
                                  │             │              └ BaseConfig_input_ms(path='Abell3376_bda_0.7.ms', data_column='DATA', sigma_column=None, weight_column='WEIGHT_SPECTRUM', time...
                                  │             └ {'MODEL_DATA'}
                                  └ <function read_xds_list at 0x7fde2d662940>

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/data_handling/ms_handler.py", line 118, in read_xds_list
    data_xds_list, utime_chunking_per_data_xds = process_bda_input(
    │                                            └ <function process_bda_input at 0x7fde26e74550>
    └ [<xarray.Dataset>
      Dimensions:          (row: 30474, chan: 1024, corr: 2, uvw: 3)
      Coordinates:
          ROWID            (row) int3...

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/data_handling/bda.py", line 106, in process_bda_input
    upsample_size = int(da.round(da.sum(interval_col)/gcd).compute())
                        │  │     │  │   │             └ dask.array<lambda, shape=(1,), dtype=float64, chunksize=(1,), chunktype=numpy.ndarray>
                        │  │     │  │   └ dask.array<copy, shape=(30474,), dtype=float64, chunksize=(30474,), chunktype=numpy.ndarray>
                        │  │     │  └ <function sum at 0x7fde2fd00ee0>
                        │  │     └ <module 'dask.array' from '/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/array/__init__.py'>
                        │  └ <function round at 0x7fde2fcbb700>
                        └ <module 'dask.array' from '/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/array/__init__.py'>

  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/base.py", line 315, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                │       │                       └ {}
                │       └ dask.array<round_, shape=(1,), dtype=float64, chunksize=(1,), chunktype=numpy.ndarray>
                └ <function compute at 0x7fdf53da59d0>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/base.py", line 603, in compute
    results = schedule(dsk, keys, **kwargs)
              │        │    │       └ {}
              │        │    └ [[('round_-bde940d8f778b4e2bf10a5eed6c59444', 0)]]
              │        └ {('round_-bde940d8f778b4e2bf10a5eed6c59444', 0): (subgraph_callable-b17c8e47-e6a2-44e4-90e7-a69af564823e, ('sum-aggregate-464...
              └ <function get at 0x7fdf53b55940>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/threaded.py", line 89, in get
    results = get_async(
              └ <function get_async at 0x7fdf54011f70>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
    │               │    └ <traceback object at 0x7fde2438c180>
    │               └ ValueError('No GCD was found for [  7.99661697  15.99319267  15.99323273  15.99323368  23.98980999\n  23.98985004  23.989851 ...
    └ <function reraise at 0x7fdf54011e50>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/local.py", line 319, in reraise
    raise exc
          └ ValueError('No GCD was found for [  7.99661697  15.99319267  15.99323273  15.99323368  23.98980999\n  23.98985004  23.989851 ...
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/local.py", line 224, in execute_task
    result = _execute_task(task, data)
             │             │     └ {('sum-aggregate-4642adb0a871e3f5f8fad6a3773896ff',): 1585249.333393862, ('values-48fdc8ee938e7de3d83de494a3c17f3b', 0): arra...
             │             └ (subgraph_callable-b17c8e47-e6a2-44e4-90e7-a69af564823e, ('sum-aggregate-4642adb0a871e3f5f8fad6a3773896ff',), ('values-48fdc8...
             └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           │      │                │               └ (('sum-aggregate-4642adb0a871e3f5f8fad6a3773896ff',), ('values-48fdc8ee938e7de3d83de494a3c17f3b', 0))
           │      │                └ {('sum-aggregate-4642adb0a871e3f5f8fad6a3773896ff',): 1585249.333393862, ('values-48fdc8ee938e7de3d83de494a3c17f3b', 0): arra...
           │      └ <function _execute_task at 0x7fdf54303670>
           └ subgraph_callable-b17c8e47-e6a2-44e4-90e7-a69af564823e
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/optimization.py", line 990, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
           │    │   │    │    │    │                │    │       └ (1585249.333393862, array([  7.99661697,  15.99319267,  15.99323273,  15.99323368,
           │    │   │    │    │    │                │    │                 23.98980999,  23.98985004,  23.989...
           │    │   │    │    │    │                │    └ <member 'inkeys' of 'SubgraphCallable' objects>
           │    │   │    │    │    │                └ subgraph_callable-b17c8e47-e6a2-44e4-90e7-a69af564823e
           │    │   │    │    │    └ <member 'outkey' of 'SubgraphCallable' objects>
           │    │   │    │    └ subgraph_callable-b17c8e47-e6a2-44e4-90e7-a69af564823e
           │    │   │    └ <member 'dsk' of 'SubgraphCallable' objects>
           │    │   └ subgraph_callable-b17c8e47-e6a2-44e4-90e7-a69af564823e
           │    └ <function get at 0x7fdf54303700>
           └ <module 'dask.core' from '/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py'>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
             │             │     └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
             │             └ (<function apply at 0x7fdf54363700>, <function round_ at 0x7fdec6d4c040>, [(<function apply at 0x7fdf54363700>, <function _en...
             └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           │      │                │               └ (<function round_ at 0x7fdec6d4c040>, [(<function apply at 0x7fdf54363700>, <function _enforce_dtype at 0x7fdebbcea4c0>, ['__...
           │      │                └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
           │      └ <function _execute_task at 0x7fdf54303670>
           └ <function apply at 0x7fdf54363700>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 119, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
                  │             │  │          └ [(<function apply at 0x7fdf54363700>, <function _enforce_dtype at 0x7fdebbcea4c0>, ['__dask_blockwise__0', (<function process...
                  │             │  └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
                  │             └ [(<function apply at 0x7fdf54363700>, <function _enforce_dtype at 0x7fdebbcea4c0>, ['__dask_blockwise__0', (<function process...
                  └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 113, in _execute_task
    return [_execute_task(a, cache) for a in arg]
            │                │               └ [(<function apply at 0x7fdf54363700>, <function _enforce_dtype at 0x7fdebbcea4c0>, ['__dask_blockwise__0', (<function process...
            │                └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
            └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 113, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
            │             │  │          └ (<function apply at 0x7fdf54363700>, <function _enforce_dtype at 0x7fdebbcea4c0>, ['__dask_blockwise__0', (<function process_...
            │             │  └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
            │             └ (<function apply at 0x7fdf54363700>, <function _enforce_dtype at 0x7fdebbcea4c0>, ['__dask_blockwise__0', (<function process_...
            └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           │      │                │               └ (<function _enforce_dtype at 0x7fdebbcea4c0>, ['__dask_blockwise__0', (<function process_bda_input.<locals>.<lambda> at 0x7fd...
           │      │                └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
           │      └ <function _execute_task at 0x7fdf54303670>
           └ <function apply at 0x7fdf54363700>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 119, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
                  │             │  │          └ ['__dask_blockwise__0', (<function process_bda_input.<locals>.<lambda> at 0x7fde243a6f70>, '__dask_blockwise__1')]
                  │             │  └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
                  │             └ ['__dask_blockwise__0', (<function process_bda_input.<locals>.<lambda> at 0x7fde243a6f70>, '__dask_blockwise__1')]
                  └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 113, in _execute_task
    return [_execute_task(a, cache) for a in arg]
            │                │               └ ['__dask_blockwise__0', (<function process_bda_input.<locals>.<lambda> at 0x7fde243a6f70>, '__dask_blockwise__1')]
            │                └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
            └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 113, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
            │             │  │          └ (<function process_bda_input.<locals>.<lambda> at 0x7fde243a6f70>, '__dask_blockwise__1')
            │             │  └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
            │             └ (<function process_bda_input.<locals>.<lambda> at 0x7fde243a6f70>, '__dask_blockwise__1')
            └ <function _execute_task at 0x7fdf54303670>
  File "/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           │      │                │               └ ('__dask_blockwise__1',)
           │      │                └ {'__dask_blockwise__0': 1585249.333393862, '__dask_blockwise__1': array([  7.99661697,  15.99319267,  15.99323273,  15.993233...
           │      └ <function _execute_task at 0x7fdf54303670>
           └ <function process_bda_input.<locals>.<lambda> at 0x7fde243a6f70>

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/data_handling/bda.py", line 104, in <lambda>
    gcd = uintervals.map_blocks(lambda x: np.array(arr_gcd(x)),
                                       │  │  │     │       └ array([  7.99661697,  15.99319267,  15.99323273,  15.99323368,
                                       │  │  │     │                 23.98980999,  23.98985004,  23.989851  ,  31.98642731,...
                                       │  │  │     └ <function arr_gcd at 0x7fde26f30dc0>
                                       │  │  └ <built-in function array>
                                       │  └ <module 'numpy' from '/home/kincaid/venv/stimela_venv/lib/python3.8/site-packages/numpy/__init__.py'>
                                       └ array([  7.99661697,  15.99319267,  15.99323273,  15.99323368,
                                                 23.98980999,  23.98985004,  23.989851  ,  31.98642731,...

  File "/vault-ike/kincaid/Software/QuartiCal/quartical/utils/maths.py", line 51, in arr_gcd
    raise ValueError(f"No GCD was found for {arr}.")

ValueError: No GCD was found for [  7.99661697  15.99319267  15.99323273  15.99323368  23.98980999
  23.98985004  23.989851    31.98642731  31.98646736  31.98646832
  39.98308468  39.98308563  47.979702    47.97970295  55.97627926
  55.97631931  55.97632027  63.97289658  63.97293663  63.97293758
  71.96951389  71.96955395  71.9695549   79.96613121  79.96617126
  79.96617222  87.96274853  87.96278858  87.96278954  95.95936584
  95.9594059   95.95940685 103.95598316 103.95602322 103.95602417
 111.95260048 111.95264053 111.95264149 119.94925785 119.9492588
 127.94583511 127.94587517 127.94587612 135.94245243 135.94249249
 143.93906975 143.93910885 143.9391098  151.93568707 151.93572712
 159.93230438 159.93234348 159.93234444 167.9289217  167.9289608
 167.92896175 175.92553902 175.92557812 175.92557907 183.92215538
 183.92219543 183.92219639 191.9187727  191.91881275 191.91881371
 199.91539001 199.91543007 199.91543102 207.91200733 207.91204739
 207.91204834 215.90862465 215.9086647  215.90866566 223.90524197
 223.90528202 223.90528297 231.90185928 231.90189934 231.90190029
 239.89851665 239.89851761 247.89509392 247.89513397 247.89513493
 255.89171124 255.89175129 263.88832855 263.88836861 271.88494587
 271.88498592 279.88156319 279.88160324 287.8781805  287.87822056
 295.87479782 295.87483692 295.87483788 303.87141514 303.87145424
 303.87145519 311.86803246 311.86807156 319.86464977 319.86468887
 327.86126709 327.86130619 335.85788441 335.85792351 343.85450172
 343.85454082 351.85111904 351.85115814 351.8511591  359.8477354
 359.84777641 367.84435272 367.84439278 367.84439373 375.84097004
 375.84101009 383.83758736 383.83762741 391.83420467 391.83424473
 399.83082199 399.83086205 407.82743931 407.82747936 415.82405663
 415.82409573 415.82409668 423.82067394 423.82071304 423.820714
 431.81729126 431.81733036 431.81733131 439.81390858 439.81394768
 447.81052589 447.81056499 455.80714321 463.80376053 471.80037785
 479.79699516 487.79361248 495.79022884 503.78684616 511.78346348
 519.7800808  527.77669811 543.76993275 551.76655006 567.7597847
 575.75640202 583.75301933 591.74963665 599.74625397 615.73948765
 623.73610497 647.72595692 655.72257423 663.71919155 671.71580887
 679.71242619 687.7090435  711.69889545 719.69551277 735.68874741
JSKenyon commented 2 years ago

This error is related to how QurtiCal currently handles the BDA case. It attempts to implicitly restore the time axis to some intermediary regular grid. The reason that this is currently failing is probably the occurrence of several slightly different time values e.g. [71.96951389 71.96955395 71.9695549].

The easiest way for me to confirm this if for you to point me to a copy of the data somewhere at Rhodes. Then I can reproduce and debug. It may be that I just need to slightly lower the strictness of the GCD (greatest common divisor) code. There is another possible solution but it is more complicated and I only want to try it as a last resort.

Kincaidr commented 2 years ago

The data is on vault-ike over here:

/vault-ike/kincaid/xova_analysis/xova_real/xova_bda , MS is called Abell3376_bda_0.7.ms.

The original pre-bda MS is in the folder before at: /vault-ike/kincaid/xova_analysis/xova_real and is called Abell3376_raw-Abell3376East-corr.ms

JSKenyon commented 2 years ago

@bennahugo @sjperkins It is worth noting that this touches on a point which I mentioned back in the depths of time. Currently, QuartiCal is trying to figure out the original/intermediate interval values (i.e. the interval which perfectly divides into the all the data in a given chunk). However, this is error-prone as even small variations in the values in the interval column (see above), can make determining this value very difficult. For now I am going to introduce a bit of a hack and round the values. The logic is that if data which contains only one sampling interval (another BDA assumption for now), any BDA row will have some integer multiple of that native interval value. This is more complicated if you have a single scan which contains data dumped with different intervals, but let us cross that bridge when we come to it. The upshot is that is may be nice if xova could provide the original sampling interval, perhaps in the column keywords/xarray attrs.

@Kincaidr I am going to test out this hack and see if I run into other problems. Will let you know when I have fixes in place.

sjperkins commented 2 years ago

The upshot is that is may be nice if xova could provide the original sampling interval, perhaps in the column keywords/xarray attrs.

I'm not sure we can assume the original data had a uniform INTERVAL. How about a subtable (BDA_ORIGINAL_GRID) with the original indexing columns?

JSKenyon commented 2 years ago

The upshot is that is may be nice if xova could provide the original sampling interval, perhaps in the column keywords/xarray attrs.

I'm not sure we can assume the original data had a uniform INTERVAL. How about a subtable (BDA_ORIGINAL_GRID) with the original indexing columns?

We definitely cannot assume that it had a uniform interval in general, but I do and will continue making that assumption in QuartiCal for now. The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid. That said, I think that if the data has a change in nominal interval e.g. intervals were 8s during the first 30 minutes of a scan and 5s thereafter, it is probably not a good idea to average over the change anyway. A more sensible approach might be to split the scan i.e. we assume some approximately constant fundamental/underlying interval within a scan partition.

I wouldn't object to a subtable which preserves the original indexing columns though. May be generally useful.

sjperkins commented 2 years ago

The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid.

Yep, it just occurred to me that Quartical is now working in something of a decomposition/reconstruction regime (like wavelets?) where Quartical is reconstructing the original resolution data (I know this isn't strictly true as the averaged data is merely expanded to full resolution?). I wonder if there's literature or general frameworks that could be employed here?

sjperkins commented 2 years ago

The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid.

Yep, it just occurred to me that Quartical is now working in something of a decomposition/reconstruction regime (like wavelets?) where Quartical is reconstructing the original resolution data (I know this isn't strictly true as the averaged data is merely expanded to full resolution?). I wonder if there's literature or general frameworks that could be employed here?

I'm guessing you've looked at a lot of this already though and I am merely interested in whether you found something useful?

JSKenyon commented 2 years ago

The alternative becomes quite complicated as it will effectively require an interpolation of the data onto a grid.

Yep, it just occurred to me that Quartical is now working in something of a decomposition/reconstruction regime (like wavelets?) where Quartical is reconstructing the original resolution data (I know this isn't strictly true as the averaged data is merely expanded to full resolution?). I wonder if there's literature or general frameworks that could be employed here?

I'm guessing you've looked at a lot of this already though and I am merely interested in whether you found something useful?

I haven't looked into it extensively. My short term goal was always proof-of-concept - currently I just retile the averaged visibilities onto a regular grid by using the native interval assumption. This is done implicitly along the row axis i.e. has no memory footprint. Channel is simply retiled to the highest frequency resolution in the data. This is done explicitly as two ragged axes becomes problematic without moving into a flattened data regime which I didn't want to consider for PoC code.

While it may be possible to interpolate the BDA data onto an arbitrary grid, it may be a little fraught i.e. what is a sensible interpolation method? And if any bad data ended up in the averaged data, it is possible that interpolating will spread it into more samples. I think that when we last chatted about this I tried to defer the complexity into dask-ms, i.e. implementing an xds_from_bda_storage type function that could take a parameter like regular=True to return seemingly regular datasets. This is a big ask though - I was simply unconvinced that the behaviour belongs in QuartiCal.

JSKenyon commented 2 years ago

@Kincaidr @bennahugo Is the MS in question only BD-averaged in time? There appears to be only a single SPW i.e. uniform/no averaging in channel.

sjperkins commented 2 years ago

This is a big ask though - I was simply unconvinced that the behaviour belongs in QuartiCal.

I'd argue that it might live in codex as the forward and reverse transform will be maintained in the same place.

Kincaidr commented 2 years ago

@Kincaidr @bennahugo Is the MS in question only BD-averaged in time? There appears to be only a single SPW i.e. uniform/no averaging in channel.

Yes, time-only averaging is only supported by xova currently.

sjperkins commented 2 years ago

@Kincaidr @bennahugo Is the MS in question only BD-averaged in time? There appears to be only a single SPW i.e. uniform/no averaging in channel.

Yes, time-only averaging is only supported by xova currently.

I think it can do channel averaging, but this just hasn't been verified for correctness yet.

JSKenyon commented 2 years ago

@Kincaidr I have solved your first problem and QuartiCal runs through but produces horrible results. I am still debugging.

JSKenyon commented 2 years ago

Closing this for now - should be "fixed" on both the main and stimelation branches.