part a) pipeline issues with latest DDF/kMS (early 2023 versions)

[twshimwell]

One issue im seeing with the new software is:

MS PROPERTIES:

• File Name: /net/lofar9/data2/shimwell/ongoing-runs/GalField5/L2011631_127MHz_uv_pre-cal.ms
• Column Name: DATA
• Selection: FIELD_ID==0 && DATA_DESC_ID==0
• Pointing center: (ra, dec)=(19:09:57.150, +08.57.02.900)
• Frequency = 127.05 MHz [ 126.12 -> 127.98 MHz]
• Wavelength = 2.36 meters
• Time bin = 8.0 seconds
• Total Integration time = 4.00 hours
• Number of antenna = 58
• Number of baseline = 1653
• Number of SPW = 1
• Number of channels = 20

Traceback (most recent call last): File "/usr/local/src/killMS/killMS/kMS.py", line 1558, in main(OP=OP,MSName=MSName) File "/usr/local/src/killMS/killMS/kMS.py", line 793, in main Solver=ClassWirtingerSolver(VS,SM,PolMode=options.PolMode, File "/usr/local/src/killMS/killMS/Wirtinger/ClassWirtingerSolverLoops.py", line 65, in init ClassWirtingerSolver.init(self,*args,**kwargs) File "/usr/local/src/killMS/killMS/Wirtinger/ClassWirtingerSolver.py", line 148, in init self.SolPredictMachine=ClassSolPredictMachine.ClassSolPredictMachine(GD) File "/usr/local/src/killMS/killMS/Wirtinger/ClassSolPredictMachine.py", line 38, in init stop NameError: name 'stop' is not defined

• 10:55:03 - ClearSHM | Clear shared memory
• 10:55:03 - Multiprocessing | reaping 1 shared memory objects associated with 1 dead DDFacet processes
• 10:55:03 - ClearSHM | Clear Semaphores
• 10:55:03 - ClearSHM | Clear shared dictionaries FAILED to run kMS.py --MSName L2011631_127MHz_uv_pre-cal.ms --SolverType KAFCA --PolMode Scalar --BaseImageName image_phase1 --dt 1.000000 --NIterKF 6 --CovQ 0.020000 --LambdaKF=0.500000 --NCPU 32 --OutSolsName DDS1 --InCol DATA --DebugPdb=0 --Weighting Natural --UVMinMax=0.100000,1000.000000 --SolsDir=SOLSDIR --NChanSols 1 --BeamMode LO FAR --PhasedArrayMode=A --DDFCacheDir=. --BeamAt=facet --NodesFile image_dirin_SSD_m.npy.ClusterCat.npy --DicoModel image_phase1_masked.DicoModel --EvolutionSolFile DDS0_smoothed: return value is 1 Traceback (most recent call last): File "/disks/paradata/shimwell/LoTSS-DR2/Software/ddf-pipeline-master/ddf-pipeline/scripts/pipeline.py", line 1951, in main(o) File "/disks/paradata/shimwell/LoTSS-DR2/Software/ddf-pipeline-master/ddf-pipeline/scripts/pipeline.py", line 1333, in main CurrentDDkMSSolName=killms_data('image_phase1',o['mslist'],'DDS1',colname=colname, File "/disks/paradata/shimwell/LoTSS-DR2/Software/ddf-pipeline-master/ddf-pipeline/scripts/pipeline.py", line 533, in killmsdata run(runcommand,dryrun=o['dryrun'],log=logfilename('KillMS-'+f+'_'+rootfilename+'.log'),quiet=o['quiet']) File "/disks/paradata/shimwell/LoTSS-DR2/Software/ddf-pipeline-master/ddf-pipeline/utils/auxcodes.py", line 68, in run die('FAILED to run '+s+': return value is '+str(retval),database=database) File "/disks/paradata/shimwell/LoTSS-DR2/Software/ddf-pipeline-master/ddf-pipeline/utils/auxcodes.py", line 51, in die raise RuntimeError(s) RuntimeError: FAILED to run kMS.py --MSName L2011631_127MHz_uv_pre-cal.ms --SolverType KAFCA --PolMode Scalar --BaseImageName image_phase1 --dt 1.000000 --NIterKF 6 --CovQ 0.020000 --LambdaKF=0.500000 --NCPU 32 --OutSolsName DDS1 --InCol DATA --DebugPdb=0 --Weighting Natural --UVMinMax=0.100000,1000.000000 --SolsDir=SOLSDIR --NChanSols 1 --BeamMode LOFAR --PhasedArrayMode=A --DDFCacheDir=. --BeamAt=facet --NodesFile image_dirin_SSD_m.npy.ClusterCat.npy --DicoModel image_phase1_masked.DicoModel --EvolutionSolFile DDS0_smoothed: return value is 1

[mhardcastle]

This is usually @cyriltasse 's way of marking some code that he's debugging or hasn't written yet. We shouldn't be hitting it in production.

[cyriltasse]

I did not hit this one did you fix it with your changes @mhardcastle ?

[mhardcastle]

No I didn't, I didn't see it either in my tests.

[twshimwell]

did you get past the calibration after image_phase?

[mhardcastle]

I didn't...

[cyriltasse]

Oh yeah perhaps I did not get there yet :)

[twshimwell]

sorry id skipped ahead by doing some of the failing steps with the old kms/ddf

[cyriltasse]

Ok I've hit and fixed this one - and I've recompiled the ddf_dev.sif image!