ValueError when running ddfacet

[amisk]

Hi,

I got the following error. Any ideas where to start looking for the cause?

Thx.

- 18:15:23 - ClassFacetMachine         | [1.2/1.2 40.3/40.3 39.8Gb]   Building Facet-normalisation image
Traceback (most recent call last):
  File "/opt/soft/ddfacet/DDFacet/DDFacet/DDF.py", line 387, in <module>
    main(OP, messages)
  File "/opt/soft/ddfacet/DDFacet/DDFacet/DDF.py", line 233, in main
    Imager.GivePredict()
  File "/opt/soft/ddfacet/DDFacet/DDFacet/Imager/ClassDeconvMachine.py", line 865, in GivePredict
    self.FacetMachine.BuildFacetNormImage()
  File "/opt/soft/ddfacet/DDFacet/DDFacet/Imager/ClassFacetMachine.py", line 1198, in BuildFacetNormImage
    FacetNorm[x0d:x1d, y0d:y1d] += np.real(SW)
ValueError: operands could not be broadcast together with shapes (6954,4796) (6875,4796) (6954,4796)
Exception AttributeError: "'NoneType' object has no attribute 'getpid'" in <bound method AsyncProcessPool.__del__ of <DDFacet.Other.AsyncProcessPool.AsyncProcessPool object at 0x7f0a38c39fd0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'SharedDict'" in <bound method ClassImageDeconvMachine.__del__ of <DDFacet.Imager.MSMF.ClassImageDeconvMachineMSMF.ClassImageDeconvMachine instance at 0x7f09b5ced8c0>> ignored
 - 18:15:36 - ClearSHM                  | Clear shared memory
 - 18:15:36 - ClearSHM                  | Clear Semaphores
 - 18:15:36 - ClearSHM                  | Clear shared dictionaries
FAILED to run DDF.py --Output-Name=Predict_DDS2 --Data-MS=/data/amisk/scratch/target/big-mslist.txt --Deconv-PeakFactor 0.010000 --Data-ColName DATA --Parallel-NCPU=48 --Beam-CenterNorm=1 --Deconv-CycleFactor=0 --Deconv-MaxMinorIter=1000000 --Deconv-MaxMajorIter=1 --Deconv-Mode SSD --Beam-Model=LOFAR --Beam-LOFARBeamMode=A --Weight-Robust -0.150000 --Image-NPix=20000 --CF-wmax 50000 --CF-Nw 100 --Output-Also onNeds --Image-Cell 1.500000 --Facets-NFacets=11 --SSDClean-NEnlargeData 0 --Freq-NDegridBand 1 --Beam-NBand 1 --Facets-DiamMax 1.5 --Facets-DiamMin 0.1 --Deconv-RMSFactor=3.000000 --SSDClean-ConvFFTSwitch 10000 --Data-Sort 1 --Cache-Dir=/data/amisk/scratch/killms/ --Log-Memory 1 --GAClean-RMSFactorInitHMP 1.000000 --GAClean-MaxMinorIterInitHMP 10000.000000 --DDESolutions-SolsDir=SOLSDIR --Cache-Weight=reset --Output-Mode=Predict --Predict-ColName DD_PREDICT --Output-RestoringBeam 12.000000 --Weight-ColName="None" --Freq-NBand=2 --RIME-DecorrMode=FT --SSDClean-SSDSolvePars [S,Alpha] --SSDClean-BICFactor 0 --Mask-Auto=1 --Mask-SigTh=5.00 --DDESolutions-GlobalNorm=None --DDESolutions-DDModeGrid=AP --DDESolutions-DDModeDeGrid=AP --DDESolutions-DDSols=DDS2_full_smoothed --Predict-InitDicoModel=image_ampphase1_di_masked.DicoModel --Selection-UVRangeKm=[0.100000,1000.000000] --GAClean-MinSizeInit=10: return value is 1
Traceback (most recent call last):
  File "/opt/soft/ddf/ddf-pipeline/scripts/pipeline.py", line 1767, in <module>
    main(o)
  File "/opt/soft/ddf/ddf-pipeline/scripts/pipeline.py", line 1379, in main
    ddsols=CurrentDDkMSSolName, PredictSettings=("Predict","DD_PREDICT"))
  File "/opt/soft/ddf/ddf-pipeline/scripts/pipeline.py", line 286, in ddf_image
    run(runcommand,dryrun=options['dryrun'],log=logfilename('DDF-'+imagename+'.log',options=options),quiet=options['quiet'])
  File "/opt/soft/ddf/ddf-pipeline/utils/auxcodes.py", line 54, in run
    die('FAILED to run '+s+': return value is '+str(retval))
  File "/opt/soft/ddf/ddf-pipeline/utils/auxcodes.py", line 36, in die
    raise Exception(s)
Exception: FAILED to run DDF.py --Output-Name=Predict_DDS2 --Data-MS=/data/amisk/scratch/target/big-mslist.txt --Deconv-PeakFactor 0.010000 --Data-ColName DATA --Parallel-NCPU=48 --Beam-CenterNorm=1 --Deconv-CycleFactor=0 --Deconv-MaxMinorIter=1000000 --Deconv-MaxMajorIter=1 --Deconv-Mode SSD --Beam-Model=LOFAR --Beam-LOFARBeamMode=A --Weight-Robust -0.150000 --Image-NPix=20000 --CF-wmax 50000 --CF-Nw 100 --Output-Also onNeds --Image-Cell 1.500000 --Facets-NFacets=11 --SSDClean-NEnlargeData 0 --Freq-NDegridBand 1 --Beam-NBand 1 --Facets-DiamMax 1.5 --Facets-DiamMin 0.1 --Deconv-RMSFactor=3.000000 --SSDClean-ConvFFTSwitch 10000 --Data-Sort 1 --Cache-Dir=/data/amisk/scratch/killms/ --Log-Memory 1 --GAClean-RMSFactorInitHMP 1.000000 --GAClean-MaxMinorIterInitHMP 10000.000000 --DDESolutions-SolsDir=SOLSDIR --Cache-Weight=reset --Output-Mode=Predict --Predict-ColName DD_PREDICT --Output-RestoringBeam 12.000000 --Weight-ColName="None" --Freq-NBand=2 --RIME-DecorrMode=FT --SSDClean-SSDSolvePars [S,Alpha] --SSDClean-BICFactor 0 --Mask-Auto=1 --Mask-SigTh=5.00 --DDESolutions-GlobalNorm=None --DDESolutions-DDModeGrid=AP --DDESolutions-DDModeDeGrid=AP --DDESolutions-DDSols=DDS2_full_smoothed --Predict-InitDicoModel=image_ampphase1_di_masked.DicoModel --Selection-UVRangeKm=[0.100000,1000.000000] --GAClean-MinSizeInit=10: return value is 1

[bennahugo]

Without more of the log it is hard to tell. Please send your parset options as well.

Looks like a proloid spheriodal filter normalization array issue - the new version may have changes in padding, etc. so please ensure to reset all caches before running.

[bennahugo]

(ie. delete the *.ddfcache directories or set all the cache switches to reset)

[amisk]

I think I wrote this into the wrong fork of DDfacet .... I installed the one from cyrilltasse.

I'll repost that there

[fvitello]

Hi, Did you fix this error? I am experiencing the same problem in my runs