askap-craco / CELEBI

The CRAFT Effortless Localisation and Enhanced Burst Inspection Pipeline
MIT License
4 stars 2 forks source link

FITLD issue in RFI subtraction (multiple processes at once) #438

Closed marcinglowacki closed 1 year ago

marcinglowacki commented 1 year ago

Task 'FITLD' during rfi subtraction would error out, returning '8' at this step. The issue was isolated to Nextflow execution, as running .command.sh and .command.run would work within the work directory, but the process would fail when the pipeline is executed.

_Using DADEVS.SH PRDEVS: cannot proceed, no printers found in /fred/oz002/askap/craft/craco/aips/DA00/PRDEVS.LIST 4154 Traceback (most recent call last): File "/fred/oz002/askap/craft/craco/psrvlbireduce/datareduction/uvsubScaled.py", line 42, in vlbatasks.fitld_uvfits(uvfile1, uvdata1, []) File "/fred/oz002/askap/craft/craco/psrvlbireduce/datareduction/vlbatasks.py", line 2486, in fitld_uvfits fitld() File "/fred/oz002/askap/craft/craco/parsel3-adam/share/parseltongue/python/AIPSTask.py", line 463, in call return self.go() File "/fred/oz002/askap/craft/craco/parsel3-adam/share/parseltongue/python/AIPSTask.py", line 445, in go self.wait(proxy, tid) File "/fred/oz002/askap/craft/craco/parsel3-adam/share/parseltongue/python/AIPSTask.py", line 397, in wait outputdict = inst.wait(tid) File "/fred/oz002/askap/craft/craco/parsel3-adam/share/parseltongue/python/Proxy/AIPSTask.py", line 395, in wait raise RuntimeError(msg) RuntimeError: Task 'fitld' returns '8'

In the end, the solution was to run the rfi subtraction processes one at a time rather than in parallel. This was attributed to AIPS/FITLD seemingly being far more likely to break when there is more than one instance running at the same time.