tigger lsm error in DDE

[Sinenhlanhla]

I am trying 3GC using a tigger skymodel with a few sources tagged. I am getting the error below.

2023-04-03 17:19:46 | ERROR | goquartical: | An error has been caught in function '', process 'MainProcess' (33259), thread 'MainThread' (140324414400320): Traceback (most recent call last):

File "/home/oem/venv/bin/goquartical", line 33, in sys.exit(load_entry_point('quartical', 'console_scripts', 'goquartical')()) │ │ └ <function importlib_load_entry_point at 0x7f9fd1bc9f70> │ └ └ <module 'sys' (built-in)>

File "/mnt/12tb/QuartiCal/quartical/executor.py", line 28, in execute _execute(stack) │ └ <contextlib.ExitStack object at 0x7f9fd1c1a700> └ <function _execute at 0x7f9f8ddd4f70>

File "/mnt/12tb/QuartiCal/quartical/executor.py", line 111, in _execute data_xds_list = add_model_graph( └ <function add_model_graph at 0x7f9f910958b0>

File "/mnt/12tb/QuartiCal/quartical/data_handling/model_handler.py", line 40, in add_model_graph predict_schemes = predict(data_xds_list, │ └ [ │ Dimensions: (row: 232960, chan: 1024, uvw: 3, corr: 2, ant: 64) │ Coordinates: │ ROWID (row) int3... └ <function predict at 0x7f9f90168f70>

File "/mnt/12tb/QuartiCal/quartical/data_handling/predict.py", line 378, in predict sky_model_dict = parse_sky_models(model_vis_recipe.ingredients.sky_models) │ │ │ └ {sky_model_nt(name='j0329.lsm.html', tags=('dE',))} │ │ └ Ingredients(model_columns=set(), sky_models={sky_model_nt(name='j0329.lsm.html', tags=('dE',))}) │ └ Recipe(ingredients=Ingredients(model_columns=set(), sky_models={sky_model_nt(name='j0329.lsm.html', tags=('dE',))}), instruct... └ <function parse_sky_models at 0x7f9f91095b80>

File "/mnt/12tb/QuartiCal/quartical/data_handling/predict.py", line 113, in parse_sky_models msg = "".join(

File "/mnt/12tb/QuartiCal/quartical/data_handling/predict.py", line 114, in "\n {:<8}: {} point source/s, {} Gaussian source/s".format(

TypeError: unsupported format string passed to NoneType.format

[JSKenyon]

Sorry to hear that you have run into a bug! Could you please upload/make the sky model in question available to me along with your QuaritCal config file/command/log?

[Sinenhlanhla]

I've attached both files. j0329_qcal.zip

[JSKenyon]

Ok - I can reproduce on an arbitrary dataset. Will sort out a fix asap.

[Sinenhlanhla]

Thanks

[JSKenyon]

OK, the reason this is happening is because the DD sources do not belong to specific component clusters. May take a little while for me to think of the best approach.

[JSKenyon]

Ok, I have made a PR with a fix. Need the tests to pass. Are you using the stimelation branch?

[Sinenhlanhla]

Uhmm, I'm currently using the development mode (which has polcal).

[JSKenyon]

If you are using the main branch installed with -e, then the fixes will be available shortly. Just need the tests to finish passing.

[Sinenhlanhla]

Yes, I'm using the main branch. Okay, great!

[JSKenyon]

OK, if you pull main the problem should be fixed. Let me know if you still encounter problems. :-)

[Sinenhlanhla]

Okay, will do. Thanks.

[Sinenhlanhla]

I've tried rerunning the config after git pull. I now get a different error.

2023-04-04 13:51:23 | INFO | preprocess:transcribe_recipe | The following model sources were obtained from --input-model-recipe: 
   Columns: None 
   Sky Models: {'j0329.lsm.html'}
2023-04-04 13:51:23 | INFO | preprocess:transcribe_recipe | Recipe contains sky models - enabling prediction step.
2023-04-04 13:51:23 | INFO | ms_handler:read_xds_list | Antenna table indicates 64 antennas were present for this observation.
2023-04-04 13:51:23 | INFO | ms_handler:read_xds_list | Polarization table indicates 2 correlations are present in the measurement set - ['XX', 'YY'].
2023-04-04 13:51:23 | INFO | ms_handler:read_xds_list | Field table indicates phase centre is at (0.913221714490383 -0.41019940114573505).
2023-04-04 13:51:30 | INFO | predict:parse_sky_models | Setting the default reference frequency for j0329.lsm.html to 1283900000.0.
2023-04-04 13:51:30 | INFO | predict:parse_sky_models | Source groups/clusters for j0329.lsm.html:
  DIE     : 854 point source/s, 2448 Gaussian source/s
  135     : 0 point source/s, 1 Gaussian source/s
  1431    : 0 point source/s, 1 Gaussian source/s
  2826    : 0 point source/s, 1 Gaussian source/s
  2970    : 0 point source/s, 1 Gaussian source/s
2023-04-04 13:51:31 | ERROR | goquartical:<module> | An error has been caught in function '<module>', process 'MainProcess' (40009), thread 'MainThread' (140593417078592):
Traceback (most recent call last):

> File "/home/oem/venv/bin/goquartical", line 33, in <module>
    sys.exit(load_entry_point('quartical', 'console_scripts', 'goquartical')())
    │   │    └ <function importlib_load_entry_point at 0x7fde738bbf70>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>

  File "/mnt/12tb/QuartiCal/quartical/executor.py", line 29, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7fde7390c700>
    └ <function _execute at 0x7fde2f82b4c0>

  File "/mnt/12tb/QuartiCal/quartical/executor.py", line 112, in _execute
    data_xds_list = add_model_graph(
                    └ <function add_model_graph at 0x7fde32da88b0>

  File "/mnt/12tb/QuartiCal/quartical/data_handling/model_handler.py", line 40, in add_model_graph
    predict_schemes = predict(data_xds_list,
                      │       └ [<xarray.Dataset>
                      │         Dimensions:     (row: 232960, uvw: 3, chan: 1024, corr: 2, ant: 64)
                      │         Coordinates:
                      │             ROWID       (row) int3...
                      └ <function predict at 0x7fde31e6cd30>

  File "/mnt/12tb/QuartiCal/quartical/data_handling/predict.py", line 458, in predict
    vis = DataArray(da.stack(source_vis).sum(axis=0),
          │         │  │     └ [dask.array<sum-aggregate, shape=(232960, 1024, 4), dtype=complex128, chunksize=(79040, 1024, 4), chunktype=numpy.ndarray>, d...
          │         │  └ <function stack at 0x7fde52a0a0d0>
          │         └ <module 'dask.array' from '/home/oem/venv/lib/python3.8/site-packages/dask/array/__init__.py'>
          └ <class 'xarray.core.dataarray.DataArray'>

  File "/home/oem/venv/lib/python3.8/site-packages/xarray/core/dataarray.py", line 419, in __init__
    coords, dims = _infer_coords_and_dims(data.shape, coords, dims)
    │              │                      │    │      │       └ ['row', 'chan', 'corr']
    │              │                      │    │      └ {'corr': ['XX', 'YY']}
    │              │                      │    └ <dask.utils.cached_property object at 0x7fde529f1ac0>
    │              │                      └ dask.array<sum-aggregate, shape=(232960, 1024, 4), dtype=complex128, chunksize=(79040, 1024, 4), chunktype=numpy.ndarray>
    │              └ <function _infer_coords_and_dims at 0x7fde39b605e0>
    └ {'corr': ['XX', 'YY']}
  File "/home/oem/venv/lib/python3.8/site-packages/xarray/core/dataarray.py", line 164, in _infer_coords_and_dims
    raise ValueError(

ValueError: conflicting sizes for dimension 'corr': length 4 on the data but length 2 on coordinate 'corr'

[JSKenyon]

Ah, annoying! Are you working on one of the Rhodes boxes?

[JSKenyon]

Actually, can you please update your codex-africanus? I think that this has already been fixed.

pip install -U codex-africanus

[Sinenhlanhla]

Okay, updating seems to have done the trick. Thanks.

[JSKenyon]

Awesome!