search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

Error merging spw=0's meta info #1456

Closed o-smirnov closed 6 months ago

o-smirnov commented 1 year ago

As reported by Savannah. I've never seen this error before, this happens at the G1 stage of the secondary. Why is append=True, can some cognoscenti remind me?

# CASA 5.8.0-109   -- Common Astronomy Software Applications
# 
# Creating a new telemetry file
# Telemetry initialized. Telemetry will send anonymized usage statistics to NRAO.
# You can disable telemetry by adding the following line to your ~/.casarc file:
# EnableTelemetry: False
# 2022-11-22 08:48:28   INFO    ::casa  CASA Version  5.8.0-109
# --> CrashReporter initialized.
# 2022-11-22 08:48:30   INFO    gaincal::::
# 2022-11-22 08:48:30   INFO    gaincal::::+    ##########################################
# 2022-11-22 08:48:30   INFO    gaincal::::+    ##### Begin Task: gaincal            #####
# 2022-11-22 08:48:30   INFO    gaincal:::: gaincal(vis="/stimela_mount/msdir/1617605022_sdp_l0-cal.ms",caltable="/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_primary_append-0.G1",field="J0025-2602",spw="",intent="",
# 2022-11-22 08:48:30   INFO    gaincal::::+            selectdata=True,timerange="",uvrange=">150",antenna="",scan="",
# 2022-11-22 08:48:30   INFO    gaincal::::+            observation="",msselect="",solint="60s",combine="",preavg=-1.0,
# 2022-11-22 08:48:30   INFO    gaincal::::+            refant="m022,m023,m024",refantmode="flex",minblperant=4,minsnr=3.0,solnorm=False,
# 2022-11-22 08:48:30   INFO    gaincal::::+            normtype="mean",gaintype="G",smodel=['1', '0', '0', '0'],calmode="ap",solmode="",
# 2022-11-22 08:48:30   INFO    gaincal::::+            rmsthresh=[],corrdepflags=False,append=True,splinetime=3600.0,npointaver=3,
# 2022-11-22 08:48:30   INFO    gaincal::::+            phasewrap=180.0,docallib=False,callib="",gaintable=['/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_secondary.K1', '/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_primary.B1'],gainfield=['J0025-2602', 'J0408-6545'],
# 2022-11-22 08:48:30   INFO    gaincal::::+            interp=['linear', 'linear'],spwmap=[],parang=False)
# 2022-11-22 08:48:30   INFO    gaincal::calibrater::open   ****Using NEW VI2-driven calibrater tool****
# 2022-11-22 08:48:30   INFO    gaincal::calibrater::open   Opening MS: /stimela_mount/msdir/1617605022_sdp_l0-cal.ms for calibration.
# 2022-11-22 08:48:30   INFO    gaincal::Calibrater::   Initializing nominal selection to the whole MS.
# 2022-11-22 08:48:30   INFO    gaincal:::: NB: gaincal automatically excludes auto-correlations.
# 2022-11-22 08:48:30   INFO    calibrater::setdata Beginning selectvis--(MSSelection version)-------
# 2022-11-22 08:48:30   INFO    calibrater::reset   Reseting solve/apply state
# 2022-11-22 08:48:30   INFO    Calibrater::selectvis   Performing selection on MeasurementSet
# 2022-11-22 08:48:30   INFO    Calibrater::selectvis+   Selecting on field: 'J0025-2602'
# 2022-11-22 08:48:30   INFO    Calibrater::selectvis+   Selecting on uvrange: '>150'
# 2022-11-22 08:48:30   INFO    Calibrater::selectvis+   Selecting with TaQL: 'ANTENNA1!=ANTENNA2'
# 2022-11-22 08:48:32   INFO    Calibrater::selectvis   By selection 1074528 rows are reduced to 566554
# 2022-11-22 08:48:32   INFO    Calibrater::selectvis   Frequency selection: Selecting all channels in all spws.
# 2022-11-22 08:48:32   INFO    calibrater::setdata chanmode=none nchan=1 start=0 step=1 mStart='0km/s' mStep='0km/s' msSelect='ANTENNA1!=ANTENNA2'
# 2022-11-22 08:48:32   INFO    calibrater::setapply    Beginning setapply--(MSSelection version)-------
# 2022-11-22 08:48:32   INFO    Calibrater::setapply(type, applypar)    Arranging to APPLY:
# 2022-11-22 08:48:32   INFO         (K Jones: Enforcing calWt()=false for phase/delay-like terms)
# 2022-11-22 08:48:32   INFO    Calibrater::setapply(type, applypar)    .   K Jones: table=/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_secondary.K1 select= interp=linear spwmap=[-1] calWt=false
# 2022-11-22 08:48:32   INFO    calibrater::setapply    Beginning setapply--(MSSelection version)-------
# 2022-11-22 08:48:32   INFO    Calibrater::setapply(type, applypar)    Arranging to APPLY:
# 2022-11-22 08:48:32   INFO    Calibrater::setapply(type, applypar)    .   B Jones: table=/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_primary.B1 select= interp=linear,linear spwmap=[-1] calWt=true
# 2022-11-22 08:48:32   INFO    calibrater::setsolve    Beginning setsolve--(MSSelection version)-------
# 2022-11-22 08:48:32   INFO    Calibrater::setsolve    Arranging to SOLVE:
# 2022-11-22 08:48:32   INFO    Calibrater::setsolve    .   G Jones: table=/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_primary_append-0.G1 append=true solint=60s refantmode='flex' refant='m022,m023,m024' minsnr=3 apmode=AP solnorm=false
# 2022-11-22 08:48:32   INFO    calibrater::solve   Beginning solve-----------------------------
# 2022-11-22 08:48:32   INFO    Calibrater::solve   The following calibration terms are arranged for apply:
# 2022-11-22 08:48:32   INFO    Calibrater::solve   .   B Jones: table=/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_primary.B1 select= interp=linear,linear spwmap=[-1] calWt=true
# 2022-11-22 08:48:32   INFO    Calibrater::solve   .   K Jones: table=/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_secondary.K1 select= interp=linear spwmap=[-1] calWt=false
# 2022-11-22 08:48:32   INFO    Calibrater::solve   The following calibration term is arranged for solve:
# 2022-11-22 08:48:32   INFO    Calibrater::solve   .   G Jones: table=/stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_primary_append-0.G1 append=true solint=60s refantmode='flex' refant='m022,m023,m024' minsnr=3 apmode=AP solnorm=false
# 2022-11-22 08:48:33   INFO    ChannelAverageTVI::parseConfiguration   Channel bin is [-1]
# 2022-11-22 08:48:33   INFO    Calibrater::solve   For solint = 60s, found 23 solution intervals.
# 2022-11-22 08:48:49   SEVERE  MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)  Leap second table TAI_UTC seems out-of-date.
# 2022-11-22 08:48:49   SEVERE  MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ Until the table is updated (see the CASA documentation or your system admin),
# 2022-11-22 08:48:49   SEVERE  MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ times and coordinates derived from UTC could be wrong by 1s or more.
# Found no unflagged data at:   (time=2021/04/05/07:23:29.6 field=1 spw=0 chan=0)
# Found no unflagged data at:   (time=2021/04/05/08:28:55.9 field=1 spw=0 chan=0)
# Found no unflagged data at:   (time=2021/04/05/09:56:09.7 field=1 spw=0 chan=0)
# 2022-11-22 08:53:46   INFO    Calibrater::solve     Found good G Jones solutions in 20 solution intervals.
# 2022-11-22 08:53:46   INFO        Applying refant: m022 (m023,m024) refantmode = flex (hold alternate refants' phase constant) when refant flagged
# 2022-11-22 08:53:46   INFO        Appending solutions to table: /stimela_mount/output/abell141-1617605022_sdp_l0-1gc1_primary_append-0.G1
# 2022-11-22 08:53:46   SEVERE  Calibrater::solve   Caught exception: Error attempting append=T
# 2022-11-22 08:53:46   INFO    Calibrater::solve   Reseting entire solve/apply state.
# 2022-11-22 08:53:46   SEVERE      Error merging spw=0's meta info
# 2022-11-22 08:53:46   SEVERE      Exception Reported: Error in Calibrater::solve.
# 2022-11-22 08:53:46   SEVERE  gaincal:::: Task gaincal raised an exception of class RuntimeError with the following message: Error in Calibrater::solve.
SavannahStanbury commented 1 year ago

here's the logfile: log-caracal.docx

paoloserra commented 1 year ago

Why is append=True, can some cognoscenti remind me?

I think that append=True because we're appending the secondary G's to the primary G's. This is useful to later run fluxscale on the table abell141-1617605022_sdp_l0-1gc1_primary_append-0.G1 in order to scale the secondary G's to the correct flux scale.

paoloserra commented 1 year ago

BTW, the table ...primary_append-0.G1 exists because

https://github.com/caracal-pipeline/caracal/blob/4f716f7c7f16ff5b7face76af053fb459a06bd77/caracal/workers/crosscal_worker.py#L193

Athanaseus commented 6 months ago

please re-open it if still experiencing the issue.