search

ratt-ru / Stimela-classic

Containerized radio interferometry scripting framework -- NB: Classic version is no longer in active development, use stimela 2! See README for details.
GNU General Public License v2.0
28 stars 16 forks source link

DDFacet error: KeyError: 'wmax' #401

Closed KshitijT closed 5 years ago

KshitijT commented 5 years ago

I get the following error while running DDFacet:

- 10:45:13 - ClassFacetMachine         | [0.1/0.3 0.9/1.0 77.4Gb] max w=1000 as per --CF-wmax setting
 - 10:45:13 - CacheManager              | [0.1/0.3 0.9/1.0 77.4Gb] cache element /scratch/msdir/COMBINED.J1638.2-6420.1GC-corr-new.ms.F0.D0.ddfcache/CFPSF does not exist, will re-make
 - 10:45:13 - ClassImagerDeconv         | [0.1/0.3 0.9/1.0 77.5Gb] allowing PSF approximations for sparsification > 999999.0
 - 10:45:13 - CacheManager              | [0.1/0.3 0.9/1.0 77.5Gb] cache element /scratch/msdir/COMBINED.J1638.2-6420.1GC-corr-new.ms.F0.D0.ddfcache/Dirty does not exist, will re-make
 - 10:45:13 - CacheManager              | [0.1/0.3 0.9/1.0 77.5Gb] cache element /scratch/msdir/COMBINED.J1638.2-6420.1GC-corr-new.ms.F0.D0.ddfcache/PSF does not exist, will re-make
 - 10:45:13 - ClassImagerDeconv         | [0.1/0.3 0.9/1.0 77.5Gb] ============================== Making Dirty Image and/or PSF ====================
 - 10:45:14 - AsyncProcessPool          | [io00] [0.1/0.1 0.8/0.9 78.0Gb] process io00: exception raised processing job VisWeights: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/DDFacet/Other/AsyncProcessPool.py", line 849, in _dispatch_job
    result = call(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/DDFacet/Data/ClassVisServer.py", line 710, in _CalcWeights_handler
    wmax = max(wmax, msw["wmax"])
  File "/usr/local/lib/python2.7/dist-packages/DDFacet/Array/shared_dict.py", line 192, in __getitem__
    value = collections.OrderedDict.__getitem__(self, item)
KeyError: 'wmax'

 - 10:45:14 - DDFacet                   | [0.1/0.3 0.9/1.0 78.0Gb] Traceback (most recent call last):
  File "/usr/local/bin/DDF.py", line 372, in <module>
    main(OP, messages)
  File "/usr/local/bin/DDF.py", line 221, in main
    Imager.main()
  File "/usr/local/lib/python2.7/dist-packages/DDFacet/Imager/ClassDeconvMachine.py", line 1011, in main
    self.GiveDirty(psf=True, sparsify=sparsify, last_cycle=(NMajor==0))
  File "/usr/local/lib/python2.7/dist-packages/DDFacet/Imager/ClassDeconvMachine.py", line 599, in GiveDirty
    self.VS.startChunkLoadInBackground()
  File "/usr/local/lib/python2.7/dist-packages/DDFacet/Data/ClassVisServer.py", line 419, in starthChunkLoadInBackground
    if self.VisWeights[self.iCurrentMS][self.iCurrentChunk]["null"]:
  File "/usr/local/lib/python2.7/dist-packages/DDFacet/Array/shared_dict.py", line 192, in __getitem__
    value = collections.OrderedDict.__getitem__(self, item)
KeyError: 'null'

 - 10:45:14 - DDFacet                   | [0.1/0.3 0.9/1.0 78.0Gb]

Never seen this before. I am using Stimela with Singularity images.

o-smirnov commented 5 years ago

Both 'null' and 'wmax' are set up during weights computation. That happens at the beginning of the run. Can you post the full log?

KshitijT commented 5 years ago

Sure : 

- 10:58:26 - DDFacet                   | starting DDFacet (/usr/local/bin/DDF.py --Data-MS /scratch/msdir/COMBINED.J1638.2-6420.1GC-corr-new.ms --Data-ColName DATA --Data-ChunkHours 0.1 --Data-Sort True --Predict-ColName MODEL_DATA --Predict-MaskSquare None --Predict-FromImage None --Predict-InitDicoModel None --Predict-Overwrite True --Selection-Field 0 --Selection-DDID 0 --Selection-TaQL  --Selection-ChanStart 0 --Selection-ChanEnd -1 --Selection-FlagAnts  --Selection-UVRangeKm  --Selection-TimeRange  --Selection-DistMaxToCore  --Output-Mode Clean --Output-Clobber False --Output-Name /scratch/output/meerkathi-pipeline-DD-precal --Output-RestoringBeam None --Output-Also all --Output-Cubes all --Output-Images DdPAMRIikz --Image-NPix 3600 --Image-Cell 1.3 --Image-PhaseCenterRADEC  --Image-SidelobeSearchWindow 200 --Facets-NFacets 17 --Facets-CatNodes None --Facets-DiamMax 180.0 --Facets-DiamMin 0.0 --Facets-PSFOversize 1.5 --Facets-PSFFacets 0 --Facets-Padding 1.7 --Weight-ColName WEIGHT_SPECTRUM --Weight-Mode Briggs --Weight-MFS True --Weight-Robust -1.5 --Weight-SuperUniform 1.0 --RIME-PolMode I --RIME-FFTMachine FFTW --RIME-ForwardMode BDA-degrid --RIME-BackwardMode BDA-grid --RIME-DecorrMode FT --RIME-DecorrLocation Edge --CF-OverS 11 --CF-Support 7 --CF-Nw 100 --CF-wmax 1000.0 --Comp-GridDecorr 0.02 --Comp-GridFoV Facet --Comp-GridDecorr 0.02 --Comp-DegridFoV Facet --Comp-Sparsification 0 --Parallel-NCPU 32 --Parallel-Affinity 1 --Parallel-Affinity 0 --Cache-Reset False --Cache-SmoothBeam reset --Cache-PSF auto --Cache-Dirty auto --Cache-VisData auto --Cache-VisData auto --Cache-LastResidual True --Cache-Dir  --Cache-DirWisdomFFTW .fftw_wisdom --Cache-ResetWisdom True --Beam-Model None --Beam-LOFARBeamMode AE --Beam-NBand 0 --Beam-CenterNorm 0 --Beam-Smooth False --Beam-SmoothNPix 11 --Beam-FITSFile beam_$(corr)_$(reim).fits --Beam-FITSFeed None --Beam-FITSFeed None --Beam-DtBeamMin 5.0 --Beam-FITSParAngleIncDeg 5.0 --Beam-FITSLAxis -X --Beam-FITSMAxis Y --Beam-FITSVerbosity 0 --Freq-BandMHz 0.0 --Freq-DegridBandMHz 0.0 --Freq-NBand 5 --Freq-NDegridBand 12 --DDESolutions-DDSols  --DDESolutions-GlobalNorm None --DDESolutions-JonesNormList AP --DDESolutions-JonesMode Full --DDESolutions-DDModeGrid AP --DDESolutions-DDModeDeGrid AP --DDESolutions-ScaleAmpGrid False --DDESolutions-ScaleAmpDeGrid False --DDESolutions-CalibErr 10.0 --DDESolutions-Type Nearest --DDESolutions-Scale 1.0 --DDESolutions-gamma 4.0 --DDESolutions-RestoreSub False --DDESolutions-ReWeightSNR 0.0 --Deconv-Mode Hogbom --Deconv-MaxMajorIter 25 --Deconv-MaxMinorIter 100000 --Deconv-AllowNegative True --Deconv-Gain 0.1 --Deconv-FluxThreshold 0.0 --Deconv-CycleFactor 0 --Deconv-RMSFactor 5.0 --Deconv-PeakFactor 0.35 --Deconv-NumRMSSamples 10000 --Deconv-ApproximatePSF 0 --Deconv-PSFBox auto --Mask-External None --Mask-Auto False --Mask-SigTh 10 --Mask-FluxImageType ModelConv --HMP-Alpha [-1.0, 1.0, 11] --HMP-Scales [0] --HMP-Ratios [''] --HMP-NTheta 6 --HMP-SolverMode PI --HMP-AllowResidIncrease True --HMP-Taper 0 --HMP-Support 0 --HMP-Kappa 1.0 --Hogbom-PolyFitOrder 3 --Hogbom-MaxLengthScale 5 --Hogbom-FreqMode Poly --Hogbom-NumBasisFuncs 12 --Montblanc-TensorflowServerTarget  --SSDClean-Parallel True --SSDClean-IslandDeconvMode GA --SSDClean-SSDSolvePars ['S', 'Alpha'] --SSDClean-SSDCostFunc ['Chi2', 'MinFlux'] --SSDClean-BICFactor 0.0 --SSDClean-ArtifactRobust False --SSDClean-ConvFFTSwitch 1000 --SSDClean-NEnlargePars 0 --SSDClean-NEnlargeData False --SSDClean-RestoreMetroSwitch 0 --SSDClean-MinMaxGroupDistance [5, 50] --GAClean-NSourceKin 100 --GAClean-NMaxGen 100 --Log-Memory True --Log-Boring True --Log-Append False --Debug-PauseWorkers False --Debug-FacetPhaseShift [0.0, 0.0] --Debug-PrintMinorCycleRMS False --Debug-DumpCleanSolutions 0 --Debug-DumpCleanPostageStamps None --Debug-CleanStallThreshold 0.0 --Debug-MemoryGreedy False --Debug-APPVerbose 0 --Misc-RandomSeed None)
 - 10:58:26 - DDFacet                   |    version is 0.3.4.1
 - 10:58:26 - DDFacet                   |    working directory is /home/kshitij
 - 10:58:26 - DDFacet                   |  Selected Options:
 - 10:58:26 - DDFacet                   | [Data] Visibility data options
 - 10:58:26 - DDFacet                   |  - MS ................................................ = /scratch/msdir/COMBINED.J1638.2-6420.1GC-corr-new.ms
 - 10:58:26 - DDFacet                   |  - ColName ........................................... = DATA
 - 10:58:26 - DDFacet                   |  - ChunkHours ........................................ = 0.1
 - 10:58:26 - DDFacet                   |  - Sort .............................................. = True
 - 10:58:26 - DDFacet                   | [Predict] Predict
 - 10:58:26 - DDFacet                   |  - ColName ........................................... = MODEL_DATA
 - 10:58:26 - DDFacet                   |  - MaskSquare ........................................ = None
 - 10:58:26 - DDFacet                   |  - FromImage ......................................... = None
 - 10:58:26 - DDFacet                   |  - InitDicoModel ..................................... = None
 - 10:58:26 - DDFacet                   |  - Overwrite ......................................... = True
 - 10:58:26 - DDFacet                   | [Selection] Data selection options
 - 10:58:26 - DDFacet                   |  - Field ............................................. = 0
 - 10:58:26 - DDFacet                   |  - DDID .............................................. = 0
 - 10:58:26 - DDFacet                   |  - TaQL .............................................. = ''
 - 10:58:26 - DDFacet                   |  - ChanStart ......................................... = 0
 - 10:58:26 - DDFacet                   |  - ChanEnd ........................................... = -1
 - 10:58:26 - DDFacet                   |  - ChanStep .......................................... = 1
 - 10:58:26 - DDFacet                   |  - FlagAnts .......................................... = ''
 - 10:58:26 - DDFacet                   |  - UVRangeKm ......................................... = ''
 - 10:58:26 - DDFacet                   |  - TimeRange ......................................... = ''
 - 10:58:26 - DDFacet                   |  - DistMaxToCore ..................................... = ''
 - 10:58:26 - DDFacet                   | [Output] Options for input and output image names
 - 10:58:26 - DDFacet                   |  - Mode .............................................. = Clean
 - 10:58:26 - DDFacet                   |  - Name .............................................. = /scratch/output/meerkathi-pipeline-DD-precal
 - 10:58:26 - DDFacet                   |  - ShiftFacetsFile ................................... = None
 - 10:58:26 - DDFacet                   |  - RestoringBeam ..................................... = None
 - 10:58:26 - DDFacet                   |  - Also .............................................. = all
 - 10:58:26 - DDFacet                   |  - Cubes ............................................. = all
 - 10:58:26 - DDFacet                   |  - Images ............................................ = DdPAMRIikz
                                                                                                                                                                                          1,2           Top
 - 10:58:26 - DDFacet                   |  - alphathreshold .................................... = 7
 - 10:58:26 - DDFacet                   |  - alphamaskthreshold ................................ = 15
 - 10:58:26 - DDFacet                   |  - StokesResidues .................................... = I
 - 10:58:26 - DDFacet                   | [Image] General imager settings
 - 10:58:26 - DDFacet                   |  - NPix .............................................. = 3600
 - 10:58:26 - DDFacet                   |  - Cell .............................................. = 1.3
 - 10:58:26 - DDFacet                   |  - PhaseCenterRADEC .................................. = ''
 - 10:58:26 - DDFacet                   |  - SidelobeSearchWindow .............................. = 200
 - 10:58:26 - DDFacet                   | [Facets] Spacial tessellation settings
 - 10:58:26 - DDFacet                   |  - NFacets ........................................... = 17
 - 10:58:26 - DDFacet                   |  - CatNodes .......................................... = None
 - 10:58:26 - DDFacet                   |  - DiamMax ........................................... = 180.0
 - 10:58:26 - DDFacet                   |  - DiamMin ........................................... = 0.0
 - 10:58:26 - DDFacet                   |  - PSFOversize ....................................... = 1.5
 - 10:58:26 - DDFacet                   |  - PSFFacets ......................................... = 0
 - 10:58:26 - DDFacet                   |  - Padding ........................................... = 1.7
 - 10:58:26 - DDFacet                   |  - Circumcision ...................................... = 0
 - 10:58:26 - DDFacet                   | [Weight] Data and imaging weight settings
 - 10:58:26 - DDFacet                   |  - ColName ........................................... = WEIGHT_SPECTRUM
 - 10:58:26 - DDFacet                   |  - Mode .............................................. = Briggs
 - 10:58:26 - DDFacet                   |  - MFS ............................................... = True
 - 10:58:26 - DDFacet                   |  - Robust ............................................ = -1.5
 - 10:58:26 - DDFacet                   |  - SuperUniform ...................................... = 1.0
 - 10:58:26 - DDFacet                   | [RIME] Low level parameters related to the forward and backward image to visibility spaces transforms
 - 10:58:26 - DDFacet                   |  - Precision ......................................... = S
 - 10:58:26 - DDFacet                   |  - PolMode ........................................... = I
 - 10:58:26 - DDFacet                   |  - FFTMachine ........................................ = FFTW
 - 10:58:26 - DDFacet                   |  - ForwardMode ....................................... = BDA-degrid
 - 10:58:26 - DDFacet                   |  - BackwardMode ...................................... = BDA-grid
 - 10:58:26 - DDFacet                   |  - DecorrMode ........................................ = FT
 - 10:58:26 - DDFacet                   |  - DecorrLocation .................................... = Edge
 - 10:58:26 - DDFacet                   | [CF] Imager convolution function settings
 - 10:58:26 - DDFacet                   |  - OverS ............................................. = 11
 - 10:58:26 - DDFacet                   |  - Support ........................................... = 7
 - 10:58:26 - DDFacet                   |  - Nw ................................................ = 100
 - 10:58:26 - DDFacet                   |  - wmax .............................................. = 1000.0
 - 10:58:26 - DDFacet                   | [Comp] Compression settings (baseline-dependent averaging [BDA] and sparsification)
 - 10:58:26 - DDFacet                   |  - GridDecorr ........................................ = 0.02
 - 10:58:26 - DDFacet                   |  - GridFoV ........................................... = Facet
 - 10:58:26 - DDFacet                   |  - DegridDecorr ...................................... = 0.02
 - 10:58:26 - DDFacet                   |  - DegridFoV ......................................... = Facet
 - 10:58:26 - DDFacet                   |  - Sparsification .................................... = 0
 - 10:58:26 - DDFacet                   |  - BDAMode ........................................... = 1
 - 10:58:26 - DDFacet                   | [Parallel] Parallelization options
 - 10:58:26 - DDFacet                   |  - NCPU .............................................. = 32
 - 10:58:26 - DDFacet                   |  - Affinity .......................................... = 0
 - 10:58:26 - DDFacet                   |  - MainProcessAffinity ............................... = 0
 - 10:58:26 - DDFacet                   | [Cache] Cache management options
 - 10:58:26 - DDFacet                   |  - Reset ............................................. = False
 - 10:58:26 - DDFacet                   |  - SmoothBeam ........................................ = reset
                                                                                                                                                                                          79,2           5%
- 10:58:26 - DDFacet                   |  - PSF ............................................... = auto
 - 10:58:26 - DDFacet                   |  - Dirty ............................................. = auto
 - 10:58:26 - DDFacet                   |  - VisData ........................................... = auto
 - 10:58:26 - DDFacet                   |  - LastResidual ...................................... = True
 - 10:58:26 - DDFacet                   |  - Dir ............................................... = ''
 - 10:58:26 - DDFacet                   |  - DirWisdomFFTW ..................................... = .fftw_wisdom
 - 10:58:26 - DDFacet                   |  - ResetWisdom ....................................... = True
 - 10:58:26 - DDFacet                   |  - CacheCF ........................................... = True
 - 10:58:26 - DDFacet                   | [Beam] Apply E-Jones (beam) during imaging
 - 10:58:26 - DDFacet                   |  - Model ............................................. = None
 - 10:58:26 - DDFacet                   |  - At ................................................ = facet
 - 10:58:26 - DDFacet                   |  - LOFARBeamMode ..................................... = AE
 - 10:58:26 - DDFacet                   |  - NBand ............................................. = 0
 - 10:58:26 - DDFacet                   |  - CenterNorm ........................................ = False
 - 10:58:26 - DDFacet                   |  - Smooth ............................................ = False
 - 10:58:26 - DDFacet                   |  - SmoothNPix ........................................ = 11
 - 10:58:26 - DDFacet                   |  - FITSFile .......................................... = beam_$(corr)_$(reim).fits
 - 10:58:26 - DDFacet                   |  - FITSFeed .......................................... = None
 - 10:58:26 - DDFacet                   |  - DtBeamMin ......................................... = 5.0
 - 10:58:26 - DDFacet                   |  - FITSParAngleIncDeg ................................ = 5.0
 - 10:58:26 - DDFacet                   |  - FITSLAxis ......................................... = -X
 - 10:58:26 - DDFacet                   |  - FITSMAxis ......................................... = Y
 - 10:58:26 - DDFacet                   |  - FITSVerbosity ..................................... = 0
 - 10:58:26 - DDFacet                   | [Freq] Multifrequency imaging options
 - 10:58:26 - DDFacet                   |  - BandMHz ........................................... = 0.0
 - 10:58:26 - DDFacet                   |  - DegridBandMHz ..................................... = 0.0
 - 10:58:26 - DDFacet                   |  - NBand ............................................. = 5
 - 10:58:26 - DDFacet                   |  - NDegridBand ....................................... = 12
 - 10:58:26 - DDFacet                   | [DDESolutions] Apply DDE solutions during imaging (@cyriltasse please document this section)
 - 10:58:26 - DDFacet                   |  - DDSols ............................................ = ''
 - 10:58:26 - DDFacet                   |  - GlobalNorm ........................................ = None
 - 10:58:26 - DDFacet                   |  - JonesNormList ..................................... = AP
 - 10:58:26 - DDFacet                   |  - JonesMode ......................................... = Full
 - 10:58:26 - DDFacet                   |  - DDModeGrid ........................................ = AP
 - 10:58:26 - DDFacet                   |  - DDModeDeGrid ...................................... = AP
 - 10:58:26 - DDFacet                   |  - ScaleAmpGrid ...................................... = False
 - 10:58:26 - DDFacet                   |  - ScaleAmpDeGrid .................................... = False
 - 10:58:26 - DDFacet                   |  - CalibErr .......................................... = 10.0
 - 10:58:26 - DDFacet                   |  - Type .............................................. = Nearest
 - 10:58:26 - DDFacet                   |  - Scale ............................................. = 1.0
 - 10:58:26 - DDFacet                   |  - gamma ............................................. = 4.0
 - 10:58:26 - DDFacet                   |  - RestoreSub ........................................ = False
 - 10:58:26 - DDFacet                   |  - ReWeightSNR ....................................... = 0.0
 - 10:58:26 - DDFacet                   | [Deconv] Common deconvolution options. Not all of these apply to all deconvolution modes
 - 10:58:26 - DDFacet                   |  - Mode .............................................. = Hogbom
 - 10:58:26 - DDFacet                   |  - MaxMajorIter ...................................... = 25
 - 10:58:26 - DDFacet                   |  - MaxMinorIter ...................................... = 100000
 - 10:58:26 - DDFacet                   |  - AllowNegative ..................................... = True
 - 10:58:26 - DDFacet                   |  - Gain .............................................. = 0.1
 - 10:58:26 - DDFacet                   |  - FluxThreshold ..................................... = 0.0
 - 10:58:26 - DDFacet                   |  - CycleFactor ....................................... = 0.0
 - 10:58:26 - DDFacet                   |  - RMSFactor ......................................... = 5.0
 - 10:58:26 - DDFacet                   |  - PeakFactor ........................................ = 0.35
                                                                                                                                                                                          129,2         13%
- 10:58:26 - DDFacet                   |  - PrevPeakFactor .................................... = 0.0
 - 10:58:26 - DDFacet                   |  - NumRMSSamples ..................................... = 10000
 - 10:58:26 - DDFacet                   |  - ApproximatePSF .................................... = 0
 - 10:58:26 - DDFacet                   |  - PSFBox ............................................ = auto
 - 10:58:26 - DDFacet                   | [Mask] Masking options. The logic being Mask_{i+1} = ExternalMask | ResidualMask | Mask_{i}
 - 10:58:26 - DDFacet                   |  - External .......................................... = None
 - 10:58:26 - DDFacet                   |  - Auto .............................................. = False
 - 10:58:26 - DDFacet                   |  - SigTh ............................................. = 10
 - 10:58:26 - DDFacet                   |  - FluxImageType ..................................... = ModelConv
 - 10:58:26 - DDFacet                   | [Noise] When using a noise map to HMP or to mask
 - 10:58:26 - DDFacet                   |  - MinStats .......................................... = [60, 2]
 - 10:58:26 - DDFacet                   |  - BrutalHMP ......................................... = True
 - 10:58:26 - DDFacet                   | [HMP] Hybrid Matching Pursuit (aka multiscale/multifrequency) mode deconvolution options
 - 10:58:26 - DDFacet                   |  - Alpha ............................................. = [-1.0, 1.0, 11]
 - 10:58:26 - DDFacet                   |  - Scales ............................................ = [0]
 - 10:58:26 - DDFacet                   |  - Ratios ............................................ = ['']
 - 10:58:26 - DDFacet                   |  - NTheta ............................................ = 6
 - 10:58:26 - DDFacet                   |  - SolverMode ........................................ = PI
 - 10:58:26 - DDFacet                   |  - AllowResidIncrease ................................ = True
 - 10:58:26 - DDFacet                   |  - MajorStallThreshold ............................... = 0.8
 - 10:58:26 - DDFacet                   |  - Taper ............................................. = 0
 - 10:58:26 - DDFacet                   |  - Support ........................................... = 0
 - 10:58:26 - DDFacet                   |  - PeakWeightImage ................................... = None
 - 10:58:26 - DDFacet                   |  - Kappa ............................................. = 1.0
 - 10:58:26 - DDFacet                   |  - OuterSpaceTh ...................................... = 2.0
 - 10:58:26 - DDFacet                   |  - FractionRandomPeak ................................ = None
 - 10:58:26 - DDFacet                   | [Hogbom] Hogbom
 - 10:58:26 - DDFacet                   |  - PolyFitOrder ...................................... = 3
 - 10:58:26 - DDFacet                   |  - MaxLengthScale .................................... = 5
 - 10:58:26 - DDFacet                   |  - FreqMode .......................................... = Poly
 - 10:58:26 - DDFacet                   |  - NumBasisFuncs ..................................... = 12
 - 10:58:26 - DDFacet                   | [Montblanc] Montblanc settings (for --Image-PredictMode=Montblanc)
 - 10:58:26 - DDFacet                   |  - TensorflowServerTarget ............................ = ''
 - 10:58:26 - DDFacet                   | [SSDClean] SSD deconvolution mode settings
 - 10:58:26 - DDFacet                   |  - Parallel .......................................... = True
 - 10:58:26 - DDFacet                   |  - IslandDeconvMode .................................. = GA
 - 10:58:26 - DDFacet                   |  - SSDSolvePars ...................................... = ['S', 'Alpha']
 - 10:58:26 - DDFacet                   |  - SSDCostFunc ....................................... = ['Chi2', 'MinFlux']
 - 10:58:26 - DDFacet                   |  - BICFactor ......................................... = 0.0
 - 10:58:26 - DDFacet                   |  - ArtifactRobust .................................... = False
 - 10:58:26 - DDFacet                   |  - ConvFFTSwitch ..................................... = 1000
 - 10:58:26 - DDFacet                   |  - NEnlargePars ...................................... = 0
 - 10:58:26 - DDFacet                   |  - NEnlargeData ...................................... = False
 - 10:58:26 - DDFacet                   |  - RestoreMetroSwitch ................................ = 0
 - 10:58:26 - DDFacet                   |  - MinMaxGroupDistance ............................... = [5, 50]
 - 10:58:26 - DDFacet                   | [GAClean] GAClean
 - 10:58:26 - DDFacet                   |  - NSourceKin ........................................ = 100
 - 10:58:26 - DDFacet                   |  - NMaxGen ........................................... = 100
 - 10:58:26 - DDFacet                   |  - MinSizeInit ....................................... = 10
 - 10:58:26 - DDFacet                   |  - InitType .......................................... = HMP
 - 10:58:26 - DDFacet                   |  - ScalesInitHMP ..................................... = [0, 1, 2, 4, 8, 16, 24, 32]
 - 10:58:26 - DDFacet                   |  - GainInitHMP ....................................... = 0.1
                                                                                                                                                                                          142,2         22%
 - 10:58:26 - DDFacet                   |  - MaxMinorIterInitHMP ............................... = 10000
 - 10:58:26 - DDFacet                   |  - AllowNegativeInitHMP .............................. = False
 - 10:58:26 - DDFacet                   |  - RMSFactorInitHMP .................................. = 3.0
 - 10:58:26 - DDFacet                   | [MORESANE] PyMoresane internal options
 - 10:58:26 - DDFacet                   |  - NMajorIter ........................................ = 200
 - 10:58:26 - DDFacet                   |  - NMinorIter ........................................ = 200
 - 10:58:26 - DDFacet                   |  - Gain .............................................. = 0.1
 - 10:58:26 - DDFacet                   |  - ForcePositive ..................................... = True
 - 10:58:26 - DDFacet                   |  - SigmaCutLevel ..................................... = 1
 - 10:58:26 - DDFacet                   | [MUFFIN] MUFFIN internal options
 - 10:58:26 - DDFacet                   |  - mu_s .............................................. = 0.1
 - 10:58:26 - DDFacet                   |  - mu_l .............................................. = 0.2
 - 10:58:26 - DDFacet                   |  - nb ................................................ = ['(8', '0)']
 - 10:58:26 - DDFacet                   |  - NMinorIter ........................................ = 200
 - 10:58:26 - DDFacet                   | [Log] Options related to logging
 - 10:58:26 - DDFacet                   |  - Memory ............................................ = True
 - 10:58:26 - DDFacet                   |  - Boring ............................................ = True
 - 10:58:26 - DDFacet                   |  - Append ............................................ = False
 - 10:58:26 - DDFacet                   | [Debug] Debugging options for the discerning masochist
 - 10:58:26 - DDFacet                   |  - PauseWorkers ...................................... = False
 - 10:58:26 - DDFacet                   |  - FacetPhaseShift ................................... = [0.0, 0.0]
 - 10:58:26 - DDFacet                   |  - PrintMinorCycleRMS ................................ = False
 - 10:58:26 - DDFacet                   |  - DumpCleanSolutions ................................ = 0
 - 10:58:26 - DDFacet                   |  - DumpCleanPostageStamps ............................ = None
 - 10:58:26 - DDFacet                   |  - CleanStallThreshold ............................... = 0.0
 - 10:58:26 - DDFacet                   |  - MemoryGreedy ...................................... = False
 - 10:58:26 - DDFacet                   |  - APPVerbose ........................................ = 0
 - 10:58:26 - DDFacet                   |  - Pdb ............................................... = auto
 - 10:58:26 - DDFacet                   | [Misc] Miscellaneous options
 - 10:58:26 - DDFacet                   |  - RandomSeed ........................................ = None
 - 10:58:26 - DDFacet                   |  - ParsetVersion ..................................... = 0.2
 - 10:58:26 - DDFacet                   |  - ConserveMemory .................................... = False
 - 10:58:26 - DDFacet                   | [0.1/0.1 0.5/0.5 77.4Gb] random seed=1560941906 (automatic)
 - 10:58:26 - DDFacet                   | [0.1/0.1 0.7/0.7 77.4Gb] using up to 32 CPUs for parallelism
 - 10:58:26 - ClassMS                   | [0.1/0.1 0.7/0.7 77.4Gb] found 1 MSs matching

FULL log on Stills: /home/kshitij/meerkathi_3GC_tests/output/meerkathi-pipeline-DD-precal.log .

P.S. I don't get the same error when using DDFacet with the same option, but from commandline.

o-smirnov commented 5 years ago

I'll bet a beer it's Singularity setting a limit on the shared memory that a process inside the container can allocate. This needs to be raised. I'm also kind of suspicious that it reports "77.4Gb" of shared memory use straight up...

KshitijT commented 5 years ago

Yeah, that's what I suspected. I don't know if Data-ChunkHours helps, but that parameter was set pretty low as you can see.

ratt-priv-ci commented 5 years ago

Also note your version is 0.3.4.1. You are essentially running the version from 2017. That version had a chunking bug that has been fixed, so consider switching.

My branch should give you the latest version.

Cheers,

On Wed, Jun 19, 2019 at 1:45 PM Kshitij Thorat notifications@github.com wrote:

Yeah, that's what I suspected. I don't know if Data-ChunkHours helps, but that parameter was set pretty low as you can see.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SpheMakh/Stimela/issues/401?email_source=notifications&email_token=AEIVPJSXMTI4OUIZUDICSMDP3IL5PA5CNFSM4HZIKW4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYBTGDY#issuecomment-503526159, or mute the thread https://github.com/notifications/unsubscribe-auth/AEIVPJTWYJYLR7FAFYVZN23P3IL5PANCNFSM4HZIKW4A .

--

Benjamin Hugo

Junior Software Developer SARAO Black River Park, 2 Fir Street, Observatory, Cape Town, Western Cape, 7925 Contact: [+27] 0716293858 <+27%2071%20629%203858>

PhD. student, Radio Astronomy Techniques and Technologies, Department of Physics and Electronics, Rhodes University

Skype: benna.cn

KshitijT commented 5 years ago

@bennahugo , thanks! Though this is needed for MeerKATHI so maybe the solution should be to include the proper version of DDFacet to the master branch?

ratt-priv-ci commented 5 years ago

It is in my fork. There should be an images stimela/ddf0.4.1

On Wed, Jun 19, 2019 at 2:25 PM Kshitij Thorat notifications@github.com wrote:

@bennahugo https://github.com/bennahugo , thanks! Though this is needed for MeerKATHI so maybe the solution should be to include the proper version of DDFacet to the master branch?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SpheMakh/Stimela/issues/401?email_source=notifications&email_token=AEIVPJQHPX2DBH32BIYSHZTP3IQSLA5CNFSM4HZIKW4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYBWGRA#issuecomment-503538500, or mute the thread https://github.com/notifications/unsubscribe-auth/AEIVPJXBAJ4JLUVNHER7GZLP3IQSLANCNFSM4HZIKW4A .

--

Benjamin Hugo

Junior Software Developer SARAO Black River Park, 2 Fir Street, Observatory, Cape Town, Western Cape, 7925 Contact: [+27] 0716293858 <+27%2071%20629%203858>

PhD. student, Radio Astronomy Techniques and Technologies, Department of Physics and Electronics, Rhodes University

Skype: benna.cn

bennahugo commented 5 years ago

Can we close this one @KshitijT ?

bennahugo commented 5 years ago

Either way it is not a stimela issue

KshitijT commented 5 years ago

Yup.