caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

ddcal worker hanging on after initial Dico Model #1477

Closed AnnalisaB closed 4 months ago

AnnalisaB commented 1 year ago

Hi

I started ddcal with 2 directions and with automasking (use_mask: True) It proceeds up to the initial quick image with DDF, computes the model for the 2 directions and then it hangs on. I don't see any lock in the MS file in the msdir, so I don't know what it is waiting for

I attach the full log and copy below the last lines:

INFO 00:42:23 - DDFacetSim [x01] [25.4/160.8 62.1/543.9 22.4Gb] Computing visibilities in 1 facets for direction 'DDE_REG1' for model '/stimela_mount/output/DDcal_1_A2034-DD-precal.DicoModel'... INFO 00:43:52 - DDFacetSim [x01] [28.4/187.7 65.1/570.7 22.4Gb] Computing visibilities in 1 facets for direction 'DDE_REG2' for model '/stimela_mount/output/DDcal_1_A2034-DD-precal.DicoModel'... INFO 00:46:12 - main [x01] [4.2/163.4 31.9/537.5 28.4Gb] I/O job(s) complete

Thanks Annalisa

log-caracal.txt

KshitijT commented 1 year ago

@AnnalisaB , I haven't encountered this error before; could you please share the cubical log in the 3GC folder too - just in case it shows more information?

AnnalisaB commented 1 year ago

Sure, I have in the 3GC folder, under a directory called < target name >_ddcal 3 logs: DDcal_1_A2034-DD-masking.log DDcal_1_A2034-DD-precal.log DDcal_1dE_sub.log

I attach here the last one DDcal_1dE_sub.log, which is also teh last written by Caracal

Thanks!

DDcal_1dE_sub.log

KshitijT commented 1 year ago

@AnnalisaB , I am pretty sure this is due to a race condition described and fixed by @bennahugo here: https://github.com/ratt-ru/CubiCal/issues/459 . I think updating your stimela version should fix this issue.

AnnalisaB commented 1 year ago

Thanks, I will give it a try!

KshitijT commented 1 year ago

@AnnalisaB , could you please confirm if this issue is now solved?

AnnalisaB commented 1 year ago

Sorry, I didn't have the time to look into that yet

Athanaseus commented 4 months ago

Please open it if needed!