better model sub counting

[dane-kleiner]

Will reset the model subtraction counter when transfer_apply_gain or calibrate is run during selfcal.

[ratt-priv-ci]

Can one of the admins verify this patch?

[Athanaseus]

ok to test

[Athanaseus]

Hi @dane-kleiner, it seems the exception is missed somehow. The error is:

2023-11-24 14:07:18 CARACal INFO: selfcal: finished
2023-11-24 14:07:18 CARACal INFO: Rendering report(s)
# running run-radiopadre --non-interactive --auto-init --docker --nbconvert output/mypipelinerun-detailed-final-report.ipynb
# 1.2.0: Pulling from osmirnov/radiopadre
# Digest: sha256:70e5d6be4cc76a68634bafedd48c59fbd5b39e34dd319b6d29f3ee57ec8e9ee9
# Status: Image is up to date for quay.io/osmirnov/radiopadre:1.2.0
# quay.io/osmirnov/radiopadre:1.2.0
2023-11-24 14:07:56 CARACal INFO: Rendered report output/mypipelinerun-detailed-final-report.html
2023-11-24 14:07:56 CARACal INFO: line: initializing
2023-11-24 14:07:57 CARACal INFO: MS #0: 1532022061_subset-circinus_p3-corr.ms
2023-11-24 14:07:57 CARACal INFO:   1 spectral windows, with NCHAN=48
2023-11-24 14:07:57 CARACal INFO:   CHAN_FREQ from 1415033203.125 Hz to 1424855468.75 Hz with average channel width of 208984.375 Hz
2023-11-24 14:07:57 CARACal INFO:   Target RA, Dec for Doppler correction: 213.312 deg, -65.337 deg
2023-11-24 14:07:58 CARACal INFO: Calculated common Doppler-corrected channel grid for all input .MS: 45 channels starting at 1415321214.904Hz and with channel width 208996.045Hz.
2023-11-24 14:07:58 CARACal INFO: Check the MS name: 1532022061_subset-circinus_p3-corr.ms
2023-11-24 14:07:58 CARACal ERROR: NullTable::getColumn - Table object is empty [RuntimeError]

Traceback from log:

2023-11-24 14:07:56 CARACal.Stimela.line INFO: ---------------------------------
2023-11-24 14:07:56 CARACal.Stimela.line INFO: Stimela version 1.7.9
2023-11-24 14:07:56 CARACal.Stimela.line INFO: Running: line
2023-11-24 14:07:56 CARACal.Stimela.line INFO: ---------------------------------
2023-11-24 14:07:56 CARACal.Stimela.line INFO: Config directory cannot be found. Will create /var/lib/jenkins/jobs/PR_meerkathi/workspace/533/test-output/2d29f11/minitestConfig-docker/.stimela_workdir-1700834876098308/stimela_parameter_files
2023-11-24 14:07:56 CARACal INFO: line: initializing
2023-11-24 14:07:57 CARACal INFO: MS #0: 1532022061_subset-circinus_p3-corr.ms
2023-11-24 14:07:57 CARACal INFO:   1 spectral windows, with NCHAN=48
2023-11-24 14:07:57 CARACal INFO:   CHAN_FREQ from 1415033203.125 Hz to 1424855468.75 Hz with average channel width of 208984.375 Hz
2023-11-24 14:07:57 CARACal INFO:   Target RA, Dec for Doppler correction: 213.312 deg, -65.337 deg
2023-11-24 14:07:58 CARACal INFO: Calculated common Doppler-corrected channel grid for all input .MS: 45 channels starting at 1415321214.904Hz and with channel width 208996.045Hz.
2023-11-24 14:07:58 CARACal INFO: Check the MS name: 1532022061_subset-circinus_p3-corr.ms
2023-11-24 14:07:58 CARACal.Stimela.line INFO: Adding cab 'quay.io/stimela/msutils:1.4.6' (1.0.1) to recipe, container name 'modelsub-ms0'
2023-11-24 14:07:58 CARACal ERROR: NullTable::getColumn - Table object is empty [RuntimeError]
2023-11-24 14:07:58 CARACal INFO:   More information can be found in the logfile at output/logs-20231124-133759/log-caracal.txt
2023-11-24 14:07:58 CARACal INFO:   You are running version 1.1.3
2023-11-24 14:07:58 CARACal ERROR: Traceback (most recent call last):
2023-11-24 14:07:58 CARACal ERROR:   File "/var/lib/jenkins/jobs/PR_meerkathi/workspace/533/test-output/2d29f11/caracal_venv/lib/python3.8/site-packages/caracal/main.py", line 189, in __run
2023-11-24 14:07:58 CARACal ERROR:     pipeline.run_workers()
2023-11-24 14:07:58 CARACal ERROR:   File "/var/lib/jenkins/jobs/PR_meerkathi/workspace/533/test-output/2d29f11/caracal_venv/lib/python3.8/site-packages/caracal/workers/worker_administrator.py", line 436, in run_workers
2023-11-24 14:07:58 CARACal ERROR:     worker.worker(self, recipe, config)
2023-11-24 14:07:58 CARACal ERROR:   File "/var/lib/jenkins/jobs/PR_meerkathi/workspace/533/test-output/2d29f11/caracal_venv/lib/python3.8/site-packages/caracal/utils/requires.py", line 25, in inner_func
2023-11-24 14:07:58 CARACal ERROR:     return func(*args, **kw)
2023-11-24 14:07:58 CARACal ERROR:   File "/var/lib/jenkins/jobs/PR_meerkathi/workspace/533/test-output/2d29f11/caracal_venv/lib/python3.8/site-packages/caracal/workers/line_worker.py", line 475, in worker
2023-11-24 14:07:58 CARACal ERROR:     t.putcolkeyword('CORRECTED_DATA', 'modelSub', nModelSub - 1)
2023-11-24 14:07:58 CARACal ERROR:   File "/var/lib/jenkins/jobs/PR_meerkathi/workspace/533/test-output/2d29f11/caracal_venv/lib/python3.8/site-packages/casacore/tables/table.py", line 1418, in putcolkeyword
2023-11-24 14:07:58 CARACal ERROR:     return self._putkeyword(columnname, keyword, -1,
2023-11-24 14:07:58 CARACal ERROR: RuntimeError: NullTable::getColumn - Table object is empty

Happening here:

+            with table('{0:s}/{1:s}'.format(pipeline.msdir, msname), readonly=False) as t:
+                caracal.log.info(f"Check the MS name: {msname}")
+                try:
+                   nModelSub = t.getcolkeyword('CORRECTED_DATA', 'modelSub')
+                    caracal.log.info(f"NmodelSub here = {nModelSub}")
+                except RuntimeError:
+                    nModelSub = 0

Full-log: log-caracal.txt

[dane-kleiner]

@Athanaseus

Taking suggestions on how to catch a RuntimeError when the exception doesn't... Maybe it needs that pass after all? I'll look into it some time in the near future.

[dane-kleiner]

Just had a quick trawl through the log, and the RuntimeError was caught successfully two previous times. So, I wonder what's going on at this stage. It says that table object is empty.

[Athanaseus]

That's what puzzles me. And also the first error raised is ERROR: NullTable::getColumn which seems internal to casacore before the RunTime appears.

[dane-kleiner]

Can you please upload the config you used?

[Athanaseus]

Same as minitestConfig.yml and it runs with the pipeline defaults.

[dane-kleiner]

@Athanaseus The commit should have fixed the problem (which I needlessly introduced).

[Athanaseus]

retest this please