search

ratt-ru / Stimela-classic

Containerized radio interferometry scripting framework -- NB: Classic version is no longer in active development, use stimela 2! See README for details.
GNU General Public License v2.0
28 stars 16 forks source link

Errors while applying caiibration gain via stimela pipeline. #417

Closed kelahard closed 5 years ago

kelahard commented 5 years ago

am running a stimela script to apply saved gain to target column in the MS and got the output below. it send an error message ...not loading G solutions: /scratch/output/output_hi_emission/smoth2/hi_emission.gain.cp does not exist but still continue to the end. When I imaged the output data, there seems not to be any difference between the input and output data before and after applying the gains. I suspect the gains were not been applied.

Also posted here

I don't know if anyone has experience this and how he/she resolved it.

...
...
2019-06-28 15:32:20,494 - STIMELA - INFO - Starting container [YQG]. Timeout set to -1. The container ID is printed below.
Running: singularity run instance://YQG 
tee: /scratch/logfile: Permission denied
Running: /usr/bin/meqtree-pipeliner.py --mt 2 -c /scratch/code/tdlconf.profiles [stefcal] stefcal_gain.enabled=1 ms_sel.field_index=0 pybeams_fits.filename_pattern='/scratch/input/fits/JVLA-L_interp_by_weight_freq-$(xy)_$(reim).fits' stefcal_gain.flag_chisq=0 do_output=CORR_DATA stefcal_gain.flag_ampl=0 ms_sel.tile_size=512 pybeams_fits.l_axis=X stefcal_gain.reset=0 me.e_enable=1 stefcal_gain.flag_ampl_high=1.8 stefcal_gain.flag_chisq_threshold=10 stefcal_gain.flag_ampl_low=0.2 pybeams_fits.sky_rotation=1 stefcal_gain.mode=apply stefcal_gain.timeint=1 ms_sel.ms_read_flags=1 ms_sel.input_column=DIFFUSE_DATA ms_sel.msname=/scratch/msdir/hi_emission.ms stefcal_gain.table=/scratch/output/output_hi_emission/smoth2/hi_emission.gain.cp me.e_module=Siamese_OMS_pybeams_fits pybeams_fits.m_axis=Y tiggerlsm.lsm_subset=all stefcal_gain.freqint=1 ms_sel.ddid_index=0 me.p_enable=1 stefcal_gain.implementation=Gain2x2 ms_rfl.read_legacy_flags=1 me.e_all_stations=1 ms_sel.ms_corr_sel='2x2' tiggerlsm.filename=/scratch/input/skymodel01.txt ms_sel.output_column=CORR_DIFFUSE_DATA_smoth2 /usr/lib/python2.7/dist-packages/Cattery/Calico/calico-stefcal.py =stefcal 
### Starting meqserver
### Attaching to configuration file /scratch/code/tdlconf.profiles
### Loading config section stefcal
### Setting option stefcal_gain.enabled=1
### Setting option ms_sel.field_index=0
### Setting option pybeams_fits.filename_pattern=/scratch/input/fits/JVLA-L_interp_by_weight_freq-$(xy)_$(reim).fits
### Setting option stefcal_gain.flag_chisq=0
### Setting option do_output=CORR_DATA
### Setting option stefcal_gain.flag_ampl=0
### Setting option ms_sel.tile_size=512
### Setting option pybeams_fits.l_axis=X
### Setting option stefcal_gain.reset=0
### Setting option me.e_enable=1
### Setting option stefcal_gain.flag_ampl_high=1.8
### Setting option stefcal_gain.flag_chisq_threshold=10
### Setting option stefcal_gain.flag_ampl_low=0.2
### Setting option pybeams_fits.sky_rotation=1
### Setting option stefcal_gain.mode=apply
### Setting option stefcal_gain.timeint=1
### Setting option ms_sel.ms_read_flags=1
### Setting option ms_sel.input_column=DIFFUSE_DATA
### Setting option ms_sel.msname=/scratch/msdir/hi_emission.ms
### Setting option stefcal_gain.table=/scratch/output/output_hi_emission/smoth2/hi_emission.gain.cp
### Setting option me.e_module=Siamese_OMS_pybeams_fits
### Setting option pybeams_fits.m_axis=Y
### Setting option tiggerlsm.lsm_subset=all
### Setting option stefcal_gain.freqint=1
### Setting option ms_sel.ddid_index=0
### Setting option me.p_enable=1
### Setting option stefcal_gain.implementation=Gain2x2
### Setting option ms_rfl.read_legacy_flags=1
### Setting option me.e_all_stations=1
### Setting option ms_sel.ms_corr_sel=2x2
### Setting option tiggerlsm.filename=/scratch/input/skymodel01.txt
### Setting option ms_sel.output_column=CORR_DIFFUSE_DATA_smoth2
### Compiling /usr/lib/python2.7/dist-packages/Cattery/Calico/calico-stefcal.py
### (using previously set options)
  (Meow.MSUtils: found /usr/bin/lwimager, can use it for imaging.)
  (Meow.MSUtils: found image viewer /usr/bin/tigger)
Successful readonly open of autonoread-locked table /scratch/msdir/hi_emission.ms: 31 columns, 84240 rows
Successful readonly open of autonoread-locked table /scratch/msdir/hi_emission.ms/ANTENNA: 8 columns, 27 rows
Successful readonly open of default-locked table /scratch/msdir/hi_emission.ms/OBSERVATION: 9 columns, 1 rows
Successful readonly open of autonoread-locked table /scratch/msdir/hi_emission.ms/POLARIZATION: 4 columns, 1 rows
Successful readonly open of autonoread-locked table /scratch/msdir/hi_emission.ms/DATA_DESCRIPTION: 3 columns, 1 rows
Successful readonly open of autonoread-locked table /scratch/msdir/hi_emission.ms/SPECTRAL_WINDOW: 14 columns, 1 rows
Successful readonly open of autonoread-locked table /scratch/msdir/hi_emission.ms/FIELD: 9 columns, 1 rows
Using LSM module from Tigger (1.4.2) 1.4.2 at /usr/lib/python2.7/dist-packages/Tigger (in path)
Loading /scratch/input/skymodel01.txt: ASCII table
<Tigger.Models.ModelClasses.Polarization object at 0x7f2c925d4550>
<Tigger.Models.ModelClasses.Polarization object at 0x7f2c925d4610>
<Tigger.Models.ModelClasses.Polarization object at 0x7f2c925d4550>
<Tigger.Models.ModelClasses.Polarization object at 0x7f2c925d4610>
###  TDL script successfully compiled. 3607 node definitions
  (of which 2 are root nodes) sent to meqserver.
### Running TDL job "stefcal"
 1.69 1.3Gb gainopts(StefCal.py:169:update_state): initializing options for B
 1.70 1.3Gb gainopts(StefCal.py:188:update_state): using float di True dd True
2019-06-28 13:32:27 INFO    MeasIERS::fillMeas(MeasIERS::Files, Double)     Requested JD 58661.8 is outside the range of the IERS (Earth axis data) table.
2019-06-28 13:32:27 INFO    MeasIERS::fillMeas(MeasIERS::Files, Double) +   Calculations will proceed with less precision
2019-06-28 13:32:27 SEVERE  MeasTable::dUTC(Double) (file /build/casacore-ZbZUT7/casacore-3.0.0/measures/Measures/MeasTable.cc, line 4290)  Leap second table TAI_UTC seems out-of-date.
2019-06-28 13:32:27 SEVERE  MeasTable::dUTC(Double) (file /build/casacore-ZbZUT7/casacore-3.0.0/measures/Measures/MeasTable.cc, line 4290)+ Until the table is updated (see the CASA documentation or your system admin),
2019-06-28 13:32:27 SEVERE  MeasTable::dUTC(Double) (file /build/casacore-ZbZUT7/casacore-3.0.0/measures/Measures/MeasTable.cc, line 4290)+ times and coordinates derived from UTC could be wrong by 1s or more.
121.01 34.5Gb gainopts(StefCal.py:257:get_result): get_result entry
121.01 34.5Gb gainopts(GainOpts.py:218:load_initval): not loading G solutions: /scratch/output/output_hi_emission/smoth2/hi_emission.gain.cp does not exist 
121.01 34.5Gb gainopts(StefCal.py:279:get_result): new dataset id 1
121.03 34.5Gb gainopts(GainOpts.py:287:resolve_tilings): datashape (240, 101) expanded datashape is (240, 101)
121.03 34.5Gb gainopts(GainOpts.py:294:resolve_tilings): gain tiling is [1, 1] smoothing is ()
121.03 34.5Gb gainopts(GainOpts.py:295:resolve_tilings): based on an LCM tiling of [1, 1]
122.05 35.0Gb gainopts(StefCal.py:484:get_result): constructed internal arrays, trying to release array memory
122.09 35.0Gb gainopts(StefCal.py:487:get_result): released memory
122.09 35.0Gb gainopts(StefCal.py:494:get_result): Found 27 solvable antennas
122.09 35.0Gb gainopts(StefCal.py:495:get_result):   valid ifrs outside the solvable set: 
122.09 35.0Gb gainopts(StefCal.py:496:get_result):   ifrs with no data: 
/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py:1169: RuntimeWarning: divide by zero encountered in divide
  w = weight[pq] = 1/n;
122.52 35.0Gb gainopts(StefCal.py:629:get_result): 351 baselines with non-zero weights (0 with null weight, 0 have no valid data)
122.52 35.0Gb gainopts(GainOpts.py:305:init_solver): stefcal G solve=0 Gain2x2, using 351 solvable inteferometers
122.52 35.0Gb gainopts(GainOpts.py:306:init_solver):   solution intervals: [1, 1] smoothing kernel: ()
122.52 35.0Gb gainopts(GainOpts.py:313:init_solver):   default initial value is 1
122.52 35.0Gb gain2x2(Gain2x2.py:82:__init__): convergence target 23998 of 24240 real slots
122.52 35.0Gb gainopts(GainOpts.py:320:init_solver):   subshape (240, 101) tiled (240, 101)
122.52 35.0Gb gainopts(StefCal.py:683:get_result): ***DEBUG*** data ('00', '01') [ 0.10471139+0.12116221j  0.10508104+0.12133659j  0.10546564+0.12149719j
  0.10586514+0.12164378j  0.10627948+0.1217761j   0.10670862+0.12189392j
  0.10715244+0.12199696j  0.10761087+0.12208498j  0.10808381+0.12215773j
  0.10857116+0.12221494j]
122.52 35.0Gb gainopts(StefCal.py:684:get_result): ***DEBUG*** model0 ('00', '01') [ 2.45751405+2.06936002j  2.45549917+2.06951165j  2.45353532+2.06965041j
  2.45162272+2.06977677j  2.44976115+2.06989098j  2.44795084+2.06999278j
  2.44619179+2.0700829j   2.44448400+2.07016087j  2.44282770+2.07022738j
  2.44122267+2.07028222j]
122.52 35.0Gb gainopts(StefCal.py:707:get_result): ***DEBUG*** model ('00', '01') [ 2.45751405+2.06936002j  2.45549917+2.06951165j  2.45353532+2.06965041j
  2.45162272+2.06977677j  2.44976115+2.06989098j  2.44795084+2.06999278j
  2.44619179+2.0700829j   2.44448400+2.07016087j  2.44282770+2.07022738j
  2.44122267+2.07028222j]
122.52 35.0Gb gainopts(StefCal.py:708:get_result): ***DEBUG*** model0 ('00', '01') [ 2.45751405+2.06936002j  2.45549917+2.06951165j  2.45353532+2.06965041j
  2.45162272+2.06977677j  2.44976115+2.06989098j  2.44795084+2.06999278j
  2.44619179+2.0700829j   2.44448400+2.07016087j  2.44282770+2.07022738j
  2.44122267+2.07028222j]
123.29 35.3Gb gainopts(StefCal.py:905:get_result): checking flagging
123.29 35.3Gb gainopts(StefCal.py:918:get_result): 0.00% (0/8508240) data points were flagged in the stefcal process. Can take.
123.29 35.3Gb gainopts(StefCal.py:1009:get_result): computing result
123.86 35.8Gb gainopts(StefCal.py:1081:get_result): computing result: done
123.86 35.8Gb gainopts(StefCal.py:1117:get_result): ev.0.0.0.0.1 elapsed time 0m2.85s
input flagmask is
### Job result: None
### No more commands
### Stopping the meqserver
### All your batch are belong to us. Bye!
KshitijT commented 5 years ago

@kelahard , it would help if you posted your stimela code here.

SpheMakh commented 5 years ago

@kelahard could you post the stimela recipe section that calles meqtrees.

KshitijT commented 5 years ago

Sorted, put the gaintables in the output folder and not in subfolders. Better to name your gaintables appropriately than to make subfolders in this case.