Issue with new version in ddf-pipeline runs

[mhardcastle]

Running ms2dynspec.py --ms mslist-L826950.txt --data DATA_DI_CORRECTED --model DD_PREDICT --sols [DDS3_full_smoothed,DDS3_full_slow_merged] --rad 2. --imageI image_full_ampphase_di_m.NS.int.restored.fits --imageV image_full_high_stokesV.dirty.corr.fits --LogBoring 0 --SolsDir SOLSDIR --BeamModel LOFAR --BeamNBand 1 --srclist brightlist.csv as in ddf-pipeline I get:

 - 14:56:24 - DynSpecMS                    | Using the surveys database
 - 14:56:25 - DynSpecMS                    | Created an array with 48643 records
 - 14:56:25 - DynSpecMS                    | Selected 7 target [out of the 48643 in the original list]
 - 14:56:25 - DynSpecMS                    | Including 14 off targets
 - 14:56:25 - DynSpecMS                    | Making random off catalog with 14 directions
Traceback (most recent call last):
  File "/usr/local/src/DynSpecMS/ms2dynspec.py", line 195, in <module>
    main(args)
  File "/usr/local/src/DynSpecMS/ms2dynspec.py", line 113, in main
    D = ClassDynSpecMS(ListMSName=MSList, 
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 132, in __init__
    self.InitFromCatalog()
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 299, in InitFromCatalog
    self.PosArray=np.concatenate([self.PosArray,self.GiveOffPosArray(NOff)])
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 406, in GiveOffPosArray
    self.DFacet=DFacet=DDFacet.Other.MyPickle.Load(self.DicoFacet)
  File "/usr/local/lib/python3.9/dist-packages/DDFacet/Other/MyPickle.py", line 47, in Load
    G= cPickle.load( open( filein, "rb" ))
FileNotFoundError: [Errno 2] No such file or directory: ''

I could take a look at this but would need to know which branch is the current one.

[cyriltasse]

Oh yeah ok this is the former call to ms2dynspec.py, I need to change that as well I forgot... I'll do this asap and let you know...

[cyriltasse]

I'm a little bit confused with what DynSpecMS is supposed to do... We have a target db, and here we also input brightlist.csv... What's in the csv file?

[mhardcastle]

Bright LOFAR sources picked out of the image. The idea of this was to be able to do RRL studies, though I don't think anyone has ever touched the data.

[cyriltasse]

yes - but where I'm confused is that these sources are also in the database, right? DynSpecMS is supposed to concatenate the external db with the csv?

[mhardcastle]

No, they're not in the database, because they're found from the images and we don't know they're there till we have made the image. So the code should concatenate the database and the CSV file.

[cyriltasse]

Yes makes sense!

I've made changes to the ddf-pipeline and DynSpecMS branches and more specifically to the call to ms2dynspec.py in pipeline.py and in the way the csv is appended to the fits catalog. I can't really test this myself apart from running the pipeline from scratch so maybe something stupid doesn't work. Lemme know

[cyriltasse]

(I've recompiled the complete image on herts as well)

[cyriltasse]

The field you're running the pipeline on, does it also have the master references images? So many things have changed to the code, and I wouldn't be surprised to see differences... This is fine for the reprocessing (since we only do a DDFacet subtraction - and I've checked that it was ok) but for regular LoTSS processing I would be carefull.

[cyriltasse]

Sorry in the new image there were still packaging issues in the sngularity image (I don't know why but this seems endless), in the new image that's beeing recompiled right now this should be fixed. It should be ready in 15 minutes or so

[mhardcastle]

OK, unfortunately now the reprocessing crashes out:

 - 08:02:49 - ClassMS                      | 23 MS section(s) selected
 - 08:02:59 - DynSpecMS                    | DynSpecMS version v1.0-95-gff02057 starting up
 - 08:03:16 - DynSpecMS                    | Using the fits catalog: /CATALOGS/dyn_spec_catalogue_addedexo_addvlotss.fits
 - 08:03:40 - DynSpecMS                    | Adding data from file Transient_LOTTS.csv
Traceback (most recent call last):
  File "/usr/local/src/DynSpecMS/ms2dynspec.py", line 195, in <module>
    main(args)
  File "/usr/local/src/DynSpecMS/ms2dynspec.py", line 113, in main
    D = ClassDynSpecMS(ListMSName=MSList, 
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 132, in __init__
    self.InitFromCatalog()
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 242, in InitFromCatalog
    additional=np.genfromtxt(FileCoords,dtype=dtype0,delimiter=",")[()]
  File "/usr/local/lib/python3.9/dist-packages/numpy/lib/npyio.py", line 1749, in genfromtxt
    fid = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/usr/local/lib/python3.9/dist-packages/numpy/lib/_datasource.py", line 195, in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
  File "/usr/local/lib/python3.9/dist-packages/numpy/lib/_datasource.py", line 535, in open
    raise IOError("%s not found." % path)
OSError: Transient_LOTTS.csv not found.
Traceback (most recent call last):
  File "/home/mjh/git/ddf-pipeline/scripts/run_full_field_reprocessing_pipeline.py", line 325, in <module>
    do_run_dynspec(field)
  File "/home/mjh/git/ddf-pipeline/scripts/run_full_field_reprocessing_pipeline.py", line 180, in do_run_dynspec
    raise RuntimeError('ms2dynspec.py failed with error code %i' % result)
RuntimeError: ms2dynspec.py failed with error code 256

[cyriltasse]

Ok i've pushed the changes and recompiled the image on herts, let's hope that works better this time

[mhardcastle]

All sorted, closing this issue.