tjgalvin
commented
2 weeks ago Actually happened again in a RACS field. This was using the --rename-ms option on the CLI. So very likely it is related to that and the problem is somewhere in the rename_ms_and_columns_for_selfcal function and the steps in preparation for it.
Still unclear what. For this field this was the only error across all beams and rounds of self-calibration. It also happened on the last round of self-cal. Maybe related to the zip archive option? Though this was turned off for the EMU data so probably not.
Measurement set to be self-calibrated: ms=MS(path=PosixPath('/scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round3.ms'), column='CORRECTED_DATA', beam=22, spw=None, field=None, model_column='MODEL_DATA')
02:44:43 AM
task_gaincal_applycal_ms-123
flint
Updating gaincal options with: {'solint': '480s', 'calmode': 'ap', 'round': 0, 'minsnr': 0.0, 'uvrange': '>235m', 'selectdata': True, 'gaintype': 'G', 'nspw': 4}
02:44:43 AM
task_gaincal_applycal_ms-123
flint
Detected a previous round of self-calibration.
02:44:43 AM
task_gaincal_applycal_ms-123
flint
Output MS name will be /scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round4.ms.
02:44:43 AM
task_gaincal_applycal_ms-123
flint
Copying /scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round3.ms to /scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round4.ms.
02:44:43 AM
task_gaincal_applycal_ms-123
flint
/scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round4.ms already exists. Removing it.
02:44:43 AM
task_gaincal_applycal_ms-123
flint
Removing folder /scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round4.ms
02:44:43 AM
task_gaincal_applycal_ms-123
flint
Encountered exception during execution:
Traceback (most recent call last):
File "/scratch3/gal16b/miniforge3/envs/flint_main/lib/python3.8/site-packages/prefect/engine.py", line 2099, in orchestrate_task_run
result = await call.aresult()
File "/scratch3/gal16b/miniforge3/envs/flint_main/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
return await asyncio.wrap_future(self.future)
File "/scratch3/gal16b/miniforge3/envs/flint_main/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
result = self.fn(*self.args, **self.kwargs)
File "/scratch3/gal16b/packages/flint_main/flint/flint/prefect/common/imaging.py", line 248, in task_gaincal_applycal_ms
return gaincal_applycal_ms(
File "/scratch3/gal16b/packages/flint_main/flint/flint/selfcal/casa.py", line 256, in gaincal_applycal_ms
cal_ms = copy_and_clean_ms_casagain(ms=ms, round=round, rename_ms=rename_ms)
File "/scratch3/gal16b/packages/flint_main/flint/flint/selfcal/casa.py", line 88, in copy_and_clean_ms_casagain
ms = rename_ms_and_columns_for_selfcal(
File "/scratch3/gal16b/packages/flint_main/flint/flint/ms.py", line 795, in rename_ms_and_columns_for_selfcal
raise FileNotFoundError(
FileNotFoundError: /scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round3.ms does not exists or is not a directory (hence measurement set).
02:44:43 AM
task_gaincal_applycal_ms-123
prefect.task_runs
Finished in state Failed('Task run encountered an exception FileNotFoundError: /scratch3/gal16b/racs_low3_proc/proc/57475/SB57475.RACS_1336-09.beam22.round3.ms does not exists or is not a directory (hence measurement set).')
Noticed that an EMU measurement set for beam14 failed to go through all four rounds of self-calibration. It seems that somehow the checks in
copy_and_clean_ms_casagainthat aim to detect whether a output MS already exists (which then automatically removes the file) picked up the original MS.I can trace the code and do not see how this function in isolation could rename -> delete -> try to rename. I am wondering whether somehow the naming creation tool is incorrect. I did find an unsaved change to the MS name creation function that might be related.
Only other consideration is some weird race / concurrent task issue. At a loss on this. Raising this issue to try to keep my thoughts together and track it.
For some ~5000 flint runs as far as I remember I have never seen this. Confusion.