mhardcastle / ddf-pipeline

LOFAR pipeline using killms/ddfacet
GNU General Public License v2.0
23 stars 19 forks source link

New version issues continued - issue as DDS1 with evolutionsolfile #308

Closed twshimwell closed 1 year ago

twshimwell commented 1 year ago

For some reason on a field im running I get the following error when doing the DDS1 step (i.e. Running: kMS.py --MSName L796450_125MHz_uv_pre-cal.avg.ms --SolverType KAFCA --PolMode Scalar --BaseImageName image_phase1 --dt 1.000000 --NIterKF 6 --CovQ 0.020000 --LambdaKF=0.500000 --NCPU 32 --OutSolsName DDS1 --InCol DATA_DI_CORRECTED_SCALED --DebugPdb=0 --Weighting Natural --UVMinMax=0.100000,1000.000000 --SolsDir=SOLSDIR --NChanSols 1 --BeamMode LOFAR --PhasedArrayMode=A --DDFCacheDir=. --BeamAt=facet --NodesFile image_dirin_SSD_m.npy.ClusterCat.npy --DicoModel image_phase1_masked.DicoModel --EvolutionSolFile DDS0_smoothed)

Solving [Chunk 1/1] ............ 1/120 [ ] 00% - 1'08"

If I remove the --EvolutionSolFile then it works fine... Not sure what the issue is.

I'm not sure why im seeing this issue and others have not in other runs. One thing I did notice is that the still contain the international baselines (fully flagged) for some reason so ill try removing those for a start.

mhardcastle commented 1 year ago

I have this in the very slow amplitude smooth too, by the look of things (and no int baselines):

Running: kMS.py --MSName L841876_123MHz_uv_pre-cal.ms --SolverType KAFCA --PolMode Scalar --BaseImageName image_full_ampphase_di_m --dt 43.630000 --NIterKF 6 --CovQ 0.100000 --LambdaKF=0.500000 --NCPU 32 --OutSolsName DDS3_full_slow --InCol DATA_DI_CORRECTED --DebugPdb=0 --Weighting Natural --UVMinMax=0.500000,1000.000000 --SolsDir=SOLSDIR --PreApplySols=[DDS3_full_smoothed] --NChanSols 1 --BeamMode LOFAR --PhasedArrayMode=A --DDFCacheDir=. --BeamAt=facet --NodesFile image_dirin_SSD_m.npy.ClusterCat.npy --DicoModel image_full_ampphase_di_m_masked.DicoModel
[...]

- 17:34:57 - ClassVisServer               | Load vis chunk #0: done
 - 17:34:57 - ClassWirtingerSolver         | DT=28801.000000, dt=2617.800000, nt=11.000000
  Solving [Chunk 1/1] .............   0/11 [                                              ]  00%  - 0'00" 
 - 17:36:25 - AsyncProcessPool             | process comp20: exception raised processing job worker_solver:Ch=0:Iter=0:Ant=56: Traceback (most recent call last):
  File "/usr/local/src/DDFacet/DDFacet/Other/AsyncProcessPool.py", line 873, in _dispatch_job
    result = call(*args, **kwargs)
  File "/usr/local/src/killMS/killMS/Wirtinger/ClassWirtingerSolverLoops.py", line 847, in _worker_solver
    if evP[iChanSol,iAnt].max()>2: stop
NameError: name 'stop' is not defined
cyriltasse commented 1 year ago

I had put that for debugging at some point I guess, I've removed the stop case, and I'm recompiling the image... I close for now

mhardcastle commented 1 year ago

New image doesn't fix my issue I'm afraid.

cyriltasse commented 1 year ago

Are you using /home/tasse/DDFSingularity/ddf_dev.sif ?

mhardcastle commented 1 year ago

I am, but it looks like you updated again and so I've gotten past that stage -- now I have a new problem which I'll put in a new ticket!