ddcal help/error

[healytwin1]

I am attempting to run ddcal on my field using the following config:

ddcal:
  enable: true
  label_in: 'ddfacet'
  use_pb: false
  shared_mem: 80
  image_dd:
    enable: true
    npix: 5400
    use_mask: true
    mask_sigma: 15
    cell: 2
    weight_robust: -0.5
  calibrate_dd:
    enable: True
    min_dist_from_phcentre: 900
    dd_data_col: data
    dist_ncpu: 4
    de_sources_mode: manual
    de_target_manual: [ 'Abell 2626']
    de_sources_manual: [(352.9001666666667,21.022249999999996), (353.05266666666665,21.596833333333336), (353.5585,20.174305555555556), (354.0114999999999,20.54611111111112)]

but keep getting the following error: ERROR: The number of targets for de calibration does not match sources, please recheck.

What am I doing wrong? It is a little unclear from the documentation how one is supposed to specify de_target_manual and de_sources_manual.

[viralp]

The way you specify source coordinates in de_sources_manual is not correct. It is in "hms,dms" format along with source radius in asec. For multiple sources use "00h00m00s,00d00m00s,00;00h00m00s,00d00m00s,00;00h00m00s,00d00m00s,00"

On Tue, May 18, 2021 at 12:02 PM Julia H @.***> wrote:

I am attempting to run ddcal on my field using the following config:

ddcal: enable: true label_in: 'ddfacet' use_pb: false shared_mem: 80 image_dd: enable: true npix: 5400 use_mask: true mask_sigma: 15 cell: 2 weight_robust: -0.5 calibrate_dd: enable: True min_dist_from_phcentre: 900 dd_data_col: data dist_ncpu: 4 de_sources_mode: manual de_target_manual: [ 'Abell 2626'] de_sources_manual: [(352.9001666666667,21.022249999999996), (353.05266666666665,21.596833333333336), (353.5585,20.174305555555556), (354.0114999999999,20.54611111111112)]

but keep getting the following error: ERROR: The number of targets for de calibration does not match sources, please recheck.

What am I doing wrong? It is a little unclear from the documentation https://caracal.readthedocs.io/en/latest/manual/workers/ddcal/index.html#calibrate-dd how one is supposed to specify de_target_manual and de_sources_manual.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/caracal-pipeline/caracal/issues/1333, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBBM465WIDJNFYXQUEG2ULTOI3L5ANCNFSM45CGFJEA .

-- Postdoctoral Research Fellow South African Radio Astronomy Observatory (SARAO) and Rhodes University Square Kilometre Array, Cape Town South Africa

[KshitijT]

Apologies @healytwin1 .

Please try with:

de_sources_manual :  ['352h54m00.60s, 21d01m20.10s,size1; 353h03m09.60s, 21d35m48.60s, size2; .....']

Notice the format of coordinates as @viralp mentioned. Also note that you need to put in the size of the source in arcsecond (size1 could be 80, for example) and that different sources are separated by ";" .

I'll fix the documentation as soon as possible so that it's more transparent.

[KshitijT]

Also, @healytwin1 , please make sure you have a backup copy of your selfcal-ed dataset; ddcal can be very finicky at times.

[healytwin1]

Thanks both, that worked!

Also, @healytwin1 , please make sure you have a backup copy of your selfcal-ed dataset; ddcal can be very finicky at times.

Already done, but thanks for the reminder.

[healytwin1]

Any idea what causes this?:

#  - 12:05:20 - DDFacet            [19.6/31.5 22.7/40.4 18.4Gb] Traceback (most recent call last):
#   File "/usr/local/bin/DDF.py", line 453, in <module>
#     main(OP, messages)
#   File "/usr/local/bin/DDF.py", line 287, in main
#     Imager.main()
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/Imager/ClassDeconvMachine.py", line 1171, in main
#     FacetMachine=self.FacetMachine, BaseName=self.BaseName)
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/Imager/HOGBOM/ClassImageDeconvMachineHogbom.py", line 117, in Init
#     weights=kwargs["PSFVar"]["WeightChansImages"], PSFServer=self.PSFServer)
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/Imager/HOGBOM/ClassModelMachineHogbom.py", line 74, in setFreqMachine
#     self.FreqMachine.set_Method()
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/Imager/ClassFrequencyMachine.py", line 132, in set_Method
#     self.S[iChannel, ind] = 1.0/nchunk
# IndexError: index 29 is out of bounds for axis 1 with size 29
#  - 12:05:20 - DDFacet            [19.6/31.5 22.7/40.4 18.4Gb] There was a problem after 31m50.7s; if you think this is a bug please open an issue,
#  - 12:05:20 - DDFacet            [19.6/31.5 22.7/40.4 18.4Gb]   quote your version of DDFacet and attach your logfile.
#  - 12:05:20 - DDFacet            [19.6/31.5 22.7/40.4 18.4Gb] You are using DDFacet revision: 0.5.2.0
#  - 12:05:20 - DDFacet            [19.6/31.5 22.7/40.4 18.4Gb] Your logfile is available here: /stimela_mount/output/Abell2626_ddcal/LF_May21_dd_Abell2626-DD-masking.log
# Exception ignored in: <bound method ClassFacetMachine.__del__ of <DDFacet.Imager.ClassFacetMachineTessel.ClassFacetMachineTessel object at 0x7f9b23e3c4a8>>
# Traceback (most recent call last):
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/Imager/ClassFacetMachine.py", line 178, in __del__
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/Imager/ClassFacetMachine.py", line 190, in releaseCFs
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/Array/shared_dict.py", line 145, in delete
# FileNotFoundError: [Errno 2] No such file or directory: '/dev/shm/ddf.80/CF'
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 74, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['DDF.py', '--Data-MS', '/stimela_mount/msdir/LowFreq_1563407160_XXYY-Abell2626-ddfacet.ms,/stimela_mount/msdir/LowFreq_1563234582_XXYY-Abell2626-ddfacet.ms,/stimela_mount/msdir/LowFreq_1563320161_XXYY-Abell2626-ddfacet.ms', '--Data-ColName', 'CORRECTED_DATA', '--Data-ChunkHours', '0.05', '--Data-Sort', 'False', '--Predict-ColName', 'MODEL_DATA', '--Predict-MaskSquare', 'None', '--Predict-FromImage', 'None', '--Predict-InitDicoModel', 'None', '--Predict-Overwrite', 'True', '--Selection-Field', '0', '--Selection-DDID', '0', '--Selection-TaQL', '', '--Selection-ChanStart', '0', '--Selection-ChanEnd', '-1', '--Selection-FlagAnts', '', '--Selection-UVRangeKm', '', '--Selection-TimeRange', '', '--Selection-DistMaxToCore', '', '--Output-Mode', 'Clean', '--Output-Clobber', 'False', '--Output-Name', '/stimela_mount/output//Abell2626_ddcal//LF_May21_dd_Abell2626-DD-masking', '--Output-RestoringBeam', 'None', '--Output-Also', '', '--Output-Cubes', '', '--Output-Images', 'dmcrioekzp', '--Image-NPix', '5400', '--Image-Cell', '2.0', '--Image-PhaseCenterRADEC', '', '--Image-SidelobeSearchWindow', '200', '--Facets-NFacets', '24', '--Facets-CatNodes', 'None', '--Facets-DiamMax', '180.0', '--Facets-DiamMin', '0.0', '--Facets-PSFOversize', '1.0', '--Facets-PSFFacets', '0', '--Facets-Padding', '1.7', '--Weight-ColName', 'WEIGHT', '--Weight-Mode', 'Briggs', '--Weight-MFS', 'True', '--Weight-Robust', '-0.5', '--Weight-SuperUniform', '1.0', '--RIME-PolMode', 'I', '--RIME-FFTMachine', 'FFTW', '--RIME-ForwardMode', 'BDA-degrid', '--RIME-BackwardMode', 'BDA-grid', '--RIME-DecorrMode', '', '--RIME-DecorrLocation', 'Edge', '--CF-OverS', '11', '--CF-Support', '7', '--CF-Nw', '100', '--CF-wmax', '10000.0', '--Comp-GridDecorr', '0.02', '--Comp-DegridDecorr', '0.02', '--Comp-GridFoV', 'Facet', '--Comp-GridDecorr', '0.02', '--Comp-DegridFoV', 'Facet', '--Comp-Sparsification', '0', '--Parallel-NCPU', '0', '--Parallel-Affinity', '1', '--Parallel-Affinity', '0', '--Cache-Reset', 'True', '--Cache-SmoothBeam', 'reset', '--Cache-PSF', 'auto', '--Cache-Dirty', 'auto', '--Cache-VisData', 'auto', '--Cache-VisData', 'auto', '--Cache-LastResidual', 'True', '--Cache-Dir', '', '--Cache-DirWisdomFFTW', '.fftw_wisdom', '--Cache-ResetWisdom', 'True', '--Beam-Model', 'None', '--Beam-LOFARBeamMode', 'AE', '--Beam-NBand', '0', '--Beam-CenterNorm', '0', '--Beam-Smooth', 'False', '--Beam-SmoothNPix', '11', '--Beam-FITSFile', 'beam_$(corr)_$(reim).fits', '--Beam-FITSFeed', 'None', '--Beam-FITSFeed', 'None', '--Beam-DtBeamMin', '5.0', '--Beam-FITSParAngleIncDeg', '5.0', '--Beam-FITSLAxis', '-X', '--Beam-FITSMAxis', 'Y', '--Beam-FITSVerbosity', '0', '--Freq-BandMHz', '0.0', '--Freq-DegridBandMHz', '0.0', '--Freq-NBand', '10', '--Freq-NDegridBand', '15', '--DDESolutions-DDSols', '', '--DDESolutions-GlobalNorm', 'None', '--DDESolutions-JonesNormList', 'AP', '--DDESolutions-JonesMode', 'Full', '--DDESolutions-DDModeGrid', 'AP', '--DDESolutions-DDModeDeGrid', 'AP', '--DDESolutions-ScaleAmpGrid', 'False', '--DDESolutions-ScaleAmpDeGrid', 'False', '--DDESolutions-CalibErr', '10.0', '--DDESolutions-Type', 'Nearest', '--DDESolutions-Scale', '1.0', '--DDESolutions-gamma', '4.0', '--DDESolutions-RestoreSub', 'False', '--DDESolutions-ReWeightSNR', '0.0', '--Deconv-Mode', 'Hogbom', '--Deconv-MaxMajorIter', '20', '--Deconv-MaxMinorIter', '100000', '--Deconv-AllowNegative', 'True', '--Deconv-Gain', '0.1', '--Deconv-FluxThreshold', '1e-06', '--Deconv-CycleFactor', '0.0', '--Deconv-RMSFactor', '0.0', '--Deconv-PeakFactor', '0.25', '--Deconv-NumRMSSamples', '10000', '--Deconv-ApproximatePSF', '0', '--Deconv-PSFBox', 'auto', '--Mask-External', 'None', '--Mask-Auto', 'False', '--Mask-SigTh', '10', '--Mask-FluxImageType', 'ModelConv', '--HMP-Alpha', '[-1.0, 1.0, 11]', '--HMP-Scales', '[0]', '--HMP-Ratios', "['']", '--HMP-NTheta', '6', '--HMP-SolverMode', 'PI', '--HMP-AllowResidIncrease', 'True', '--HMP-Taper', '0', '--HMP-Support', '0', '--HMP-Kappa', '1.0', '--Hogbom-PolyFitOrder', '6', '--Montblanc-TensorflowServerTarget', '', '--SSDClean-Parallel', 'True', '--SSDClean-IslandDeconvMode', 'GA', '--SSDClean-SSDSolvePars', "['S', 'Alpha']", '--SSDClean-SSDCostFunc', "['Chi2', 'MinFlux']", '--SSDClean-BICFactor', '0.0', '--SSDClean-ArtifactRobust', 'False', '--SSDClean-ConvFFTSwitch', '1000', '--SSDClean-NEnlargePars', '0', '--SSDClean-NEnlargeData', '2', '--SSDClean-RestoreMetroSwitch', '0', '--SSDClean-MinMaxGroupDistance', '[5, 50]', '--GAClean-NSourceKin', '100', '--GAClean-NMaxGen', '100', '--Log-Memory', 'True', '--Log-Boring', 'True', '--Log-Append', 'False', '--Debug-PauseWorkers', 'False', '--Debug-FacetPhaseShift', '[0.0, 0.0]', '--Debug-PrintMinorCycleRMS', 'False', '--Debug-DumpCleanSolutions', '0', '--Debug-DumpCleanPostageStamps', 'None', '--Debug-CleanStallThreshold', '0.0', '--Debug-MemoryGreedy', 'False', '--Debug-APPVerbose', '0', '--Misc-RandomSeed', 'None']' returned non-zero exit status 1.
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR: cd /scratch3/users/juliahealy/LowFreq/.stimela_workdir-16213376024185755 && singularity returns error code 1
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR: job failed at 2021-05-18 14:05:23.617130 after 0:32:00.868980
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR: Traceback (most recent call last):
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/stimela/recipe.py", line 709, in run
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:     job.run_job()
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/stimela/recipe.py", line 421, in run_job
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:     self.job.run(output_wrangler=self.apply_output_wranglers)
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/stimela/singularity.py", line 124, in run
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:     utils.xrun(f"cd {self.execdir} && singularity", ["run", "--workdir", self.execdir, "--containall"] \
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR:     raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2021-05-18 14:05:23 CARACal.Stimela.ddf_image-for_mask-Abell2626 ERROR: stimela.utils.StimelaCabRuntimeError: cd /scratch3/users/juliahealy/LowFreq/.stimela_workdir-16213376024185755 && singularityreturns error code 1
2021-05-18 14:05:23 CARACal.Stimela.ddcal INFO: Completed jobs : []
2021-05-18 14:05:23 CARACal.Stimela.ddcal INFO: Remaining jobs : ['mask_ddf-precal-Abell2626']
2021-05-18 14:05:23 CARACal.Stimela.ddcal INFO: Logging remaining task: mask_ddf-precal-Abell2626:: Make a mask for the initial ddf image
2021-05-18 14:05:23 CARACal.Stimela.ddcal INFO: Saving pipeline information in .last_ddcal.json
2021-05-18 14:05:23 CARACal ERROR: Job 'ddf_image-for_mask-Abell2626:: DDFacet image for masking' failed: cd /scratch3/users/juliahealy/LowFreq/.stimela_workdir-16213376024185755 && singularity returns error code 1 [PipelineException]
2021-05-18 14:05:23 CARACal INFO:   More information can be found in the logfile at output/logs-20210518-133244/log-caracal.txt
2021-05-18 14:05:23 CARACal INFO:   You are running version 1.0.4
2021-05-18 14:05:24 CARACal ERROR: Traceback (most recent call last):
2021-05-18 14:05:24 CARACal ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/caracal/main.py", line 183, in __run
2021-05-18 14:05:24 CARACal ERROR:     pipeline.run_workers()
2021-05-18 14:05:24 CARACal ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/caracal/workers/worker_administrator.py", line 413, in run_workers
2021-05-18 14:05:24 CARACal ERROR:     worker.worker(self, recipe, config)
2021-05-18 14:05:24 CARACal ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/caracal/workers/ddcal_worker.py", line 437, in worker
2021-05-18 14:05:24 CARACal ERROR:     dd_precal_image(field,mslist)
2021-05-18 14:05:24 CARACal ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/caracal/workers/ddcal_worker.py", line 130, in dd_precal_image
2021-05-18 14:05:24 CARACal ERROR:     recipe.run()
2021-05-18 14:05:24 CARACal ERROR:   File "/scratch3/users/juliahealy/caracal_venv/lib/python3.8/site-packages/stimela/recipe.py", line 760, in run
2021-05-18 14:05:24 CARACal ERROR:     raise PipelineException(exc, self.completed, job, self.remaining) from None
2021-05-18 14:05:24 CARACal ERROR: stimela.exceptions.PipelineException: Job 'ddf_image-for_mask-Abell2626:: DDFacet image for masking' failed: cd /scratch3/users/juliahealy/LowFreq/.stimela_workdir-16213376024185755 && singularity returns error code 1
2021-05-18 14:05:24 CARACal INFO: exiting with error code 1

[KshitijT]

Same as #1321 . :(

[healytwin1]

Ah, so no solution yet?

[KshitijT]

Ah, so no solution yet?

I think @SpheMakh needs to upload a new ddf image.

[SpheMakh]

The image is ready, please try it @KshitijT stimela/ddfacet:1.6.9 on quay.io (https://quay.io/repository/stimela/ddfacet?tab=tags)

[SpheMakh]

@KshitijT use branch ddfacet-060 of stimela to test

[KshitijT]

Thanks @SpheMakh , let me test it out.

[healytwin1]

@KshitijT has the new image been uploaded to ilifu?

[KshitijT]

@KshitijT has the new image been uploaded to ilifu?

@healytwin1 , not yet; please see the discussion here: https://github.com/ratt-ru/Stimela/pull/721.

[bennahugo]

The latest SAOPICC releases bhugo/ddfacet:0.6.0 and bhugo/killms:3.0.0.1 images are now available from Dockerhub

[bennahugo]

You can use the stimela ddfacet-060 branch. The images work, I'm just finetuning an acceptance test for this. I'm going to go ahead and close the ticket.

[KshitijT]

Let's not close this issue till the branch is merged in stimela master.

[KshitijT]

Relevant branch merged 2 years ago, closing this issue. @healytwin1 , please reopen if you encounter this issue again.