search

ratt-ru / meqtrees-cattery

MeqTrees-based frameworks for simulation and calibration of radio interferometers
Other
5 stars 9 forks source link

GainDiagPhase calibration error #60

Open bennahugo opened 7 years ago

bennahugo commented 7 years ago

Things are going south when trying to do a phase only selfcal with GJonesMatrix type "GainDiagPhase". Perhaps everything is flagged for a the particular slot @SpheMakh and myself get the same error on different MSs, will investigate soon

812 running: /usr/bin/meqtree-pipeliner.py --mt 2 -c /code/tdlconf.profiles [stefcal] ms_sel.input_column=DATA ms_sel.field_index=0 ms_sel.msname=/home/sphe/msdir/meerkat_simulation_example.ms stefcal_gain.table=/home/sphe/msdir/meerkat_simulation_example.ms/meerkat_simulation_example.gain.cp stefcal_diffgain.freqsmooth=1 ms_wfl.write_bitflag=test tiggerlsm.lsm_subset=all do_ou    tput=CORRECTED_DATA stefcal_gain.flag_chisq=0 ms_sel.ms_fill_legacy_flags=1 stefcal_gain.flag_ampl=0 ms_sel.ddid_index=0 ms_sel.ms_write_flag_policy=add to set stefcal_diffgain.timesmooth=1 ms_rfl.read_flagsets=-stefcal stefcal_gain.freqint=1 stefcal_gain.implementation=Gain2x2 stefcal_gain.flag_chisq_threshold=10 ms_rfl.read_legacy_flags=1 stefcal_gain.flag_ampl_low=0.3 st    efcal_gain.flag_ampl_high=2 ms_sel.ms_write_flags=1 stefcal_gain.mode=solve-save tiggerlsm.filename=/input/nvss1deg.lsm.html stefcal_gain.timeint=1 ms_sel.output_column=CORRECTED_DATA /usr/lib/python2.7/dist-packages/Cattery/Calico/calico-stefcal.py =stefcal
813 MeqTrees Done!
814 Preparing to make gain plots
815 Successful readonly open of default-locked table /home/sphe/msdir/meerkat_simulation_example.ms/FEED: 12 columns, 64 rows
816 Extracting feed type from MS
817 Feed type is [YX]
818 Making Gain plots...
819 loading gain solutions from /home/sphe/msdir/meerkat_simulation_example.ms/meerkat_simulation_example.gain.cp
820 loaded solutions for 64 antennas, diagonal=False
821 generated plot /home/sphe/output/meerkat_simulation_example-gjones_plots-timeslot.png
822 generated plot /home/sphe/output/meerkat_simulation_example-gjones_plots-chan.png
823 Error: No such container: calibrator_example-140348667745592148786110502
824 Task was not started./usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
825   warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
826 ### Starting meqserver
827 ### Attaching to configuration file /code/tdlconf.profiles
828 ### Loading config section stefcal
829 ### Setting option ms_sel.input_column=DATA
830 ### Setting option ms_sel.field_index=0
831 ### Setting option ms_sel.msname=/home/sphe/msdir/meerkat_simulation_example.ms
832 ### Setting option stefcal_gain.table=/home/sphe/msdir/meerkat_simulation_example.ms/meerkat_simulation_example.gain.cp
833 ### Setting option stefcal_diffgain.freqsmooth=1
834 ### Setting option ms_wfl.write_bitflag=test
835 ### Setting option tiggerlsm.lsm_subset=all
836 ### Setting option do_output=CORRECTED_DATA
837 ### Setting option stefcal_gain.flag_chisq=0
838 ### Setting option ms_sel.ms_fill_legacy_flags=1
839 ### Setting option stefcal_gain.flag_ampl=0
840 ### Setting option ms_sel.ddid_index=0
841 ### Setting option ms_sel.ms_write_flag_policy=add
842 ### Setting option stefcal_diffgain.timesmooth=1
843 ### Setting option ms_rfl.read_flagsets=-stefcal
844 ### Setting option stefcal_gain.freqint=1
845 ### Setting option stefcal_gain.implementation=GainDiagPhase
846 ### Setting option stefcal_gain.flag_chisq_threshold=10
847 ### Setting option ms_rfl.read_legacy_flags=1
848 ### Setting option stefcal_gain.flag_ampl_low=0.3
849 ### Setting option stefcal_gain.flag_ampl_high=2
850 ### Setting option ms_sel.ms_write_flags=1
851 ### Setting option stefcal_gain.mode=solve-save
852 ### Setting option tiggerlsm.filename=/input/nvss1deg.lsm.html
853 ### Setting option stefcal_gain.timeint=1
854 ### Setting option ms_sel.output_column=CORRECTED_DATA
855 ### Compiling /usr/lib/python2.7/dist-packages/Cattery/Calico/calico-stefcal.py
856 ### (using previously set options)
857   (Meow.MSUtils: found /usr/bin/lwimager, can use it for imaging.)
858   (Meow.MSUtils: found image viewer /usr/local/bin/tigger)
859 Successful readonly open of autonoread-locked table /home/sphe/msdir/meerkat_simulation_example.ms: 27 columns, 120960 rows
860 Successful readonly open of autonoread-locked table /home/sphe/msdir/meerkat_simulation_example.ms/ANTENNA: 8 columns, 64 rows
861 Successful readonly open of default-locked table /home/sphe/msdir/meerkat_simulation_example.ms/OBSERVATION: 9 columns, 1 rows
862 Successful readonly open of autonoread-locked table /home/sphe/msdir/meerkat_simulation_example.ms/POLARIZATION: 4 columns, 1 rows
863 Successful readonly open of autonoread-locked table /home/sphe/msdir/meerkat_simulation_example.ms/DATA_DESCRIPTION: 3 columns, 1 rows
864 Successful readonly open of autonoread-locked table /home/sphe/msdir/meerkat_simulation_example.ms/SPECTRAL_WINDOW: 14 columns, 1 rows
865 Successful readonly open of autonoread-locked table /home/sphe/msdir/meerkat_simulation_example.ms/FIELD: 9 columns, 1 rows
866 Using LSM module from Tigger (devel) devel at /usr/local/lib/python2.7/dist-packages/Tigger (in path)
867 Loading /input/nvss1deg.lsm.html: Tigger sky model
868 ###  TDL script successfully compiled. 16206 node definitions
869   (of which 3 are root nodes) sent to meqserver.
870 ### Running TDL job "stefcal"
871  6.01 3.6Gb gainopts(StefCal.py:169:update_state): initializing options for B
872  6.01 3.6Gb gainopts(StefCal.py:188:update_state): using float di True dd True
873  8.07 4.2Gb gainopts(StefCal.py:257:get_result): get_result entry
874  8.07 4.2Gb gainopts(GainOpts.py:230:load_initval): G solutions in /home/sphe/msdir/meerkat_simulation_example.ms/meerkat_simulation_example.gain.cp are for class Gain2x2, expected GainDiagPhase
875  8.07 4.2Gb gainopts(GainOpts.py:233:load_initval): loaded 64 gain solutions from /home/sphe/msdir/meerkat_simulation_example.ms/meerkat_simulation_example.gain.cp
876  8.07 4.2Gb gainopts(StefCal.py:279:get_result): new dataset id 1
877  8.14 4.2Gb gainopts(GainOpts.py:279:resolve_tilings): datashape (60, 1) expanded datashape is (60, 1)
878  8.14 4.2Gb gainopts(GainOpts.py:286:resolve_tilings): gain tiling is [1, 1] smoothing is ()
879  8.14 4.2Gb gainopts(GainOpts.py:287:resolve_tilings): based on an LCM tiling of [1, 1]
880  8.57 4.2Gb gainopts(StefCal.py:484:get_result): constructed internal arrays, trying to release array memory
881  8.59 4.2Gb gainopts(StefCal.py:487:get_result): released memory
882  8.59 4.2Gb gainopts(StefCal.py:494:get_result): Found 64 solvable antennas
883  8.59 4.2Gb gainopts(StefCal.py:495:get_result):   valid ifrs outside the solvable set:
884  8.59 4.2Gb gainopts(StefCal.py:496:get_result):   ifrs with no data:
885  8.78 4.2Gb gainopts(StefCal.py:629:get_result): 2016 baselines with non-zero weights (0 with null weight, 0 have no valid data)
886  8.86 4.2Gb gainopts(StefCal.py:651:get_result): Found 64 solvable antennas
887  8.87 4.2Gb gainopts(StefCal.py:653:get_result): 65 valid visibility matrices per time/freq slot are required for a solution
888  8.87 4.2Gb gainopts(StefCal.py:654:get_result):    for this data chunk, Nvalid ranges from 2016 to 2016
889  8.87 4.2Gb gainopts(StefCal.py:676:get_result): Solvable: 2016 of 2016 inteferometers (2016 have valid data), with 64 solvable antennas
890  8.87 4.2Gb gainopts(GainOpts.py:297:init_solver): stefcal G solve=1 GainDiagPhase, using 2016 solvable inteferometers
891  8.87 4.2Gb gainopts(GainOpts.py:298:init_solver):   solution intervals: [1, 1] smoothing kernel: ()
892  8.87 4.2Gb gainopts(GainOpts.py:303:init_solver):   initial values loaded, first number is (-0.224546+0.151632j)
893  8.87 4.2Gb gaindiag(GainDiagPhase.py:92:__init__): convergence target 59 of 60 real slots
894  8.87 4.2Gb gainopts(GainOpts.py:312:init_solver):   subshape (60, 1) tiled (60, 1)
895  8.87 4.2Gb gainopts(StefCal.py:683:get_result): ***DEBUG*** data ('0', '1') [-0.08994675 +1.80725527j  0.12675297 -0.62525034j -6.88391972+21.50491905j
896  -0.77556306 +4.38443947j  3.55280352 +1.11491513j -0.66596287 +0.24490798j
897  -4.02992439 +0.52879071j -0.40441036 -6.15838528j -0.44240844 -0.37135202j
898  -0.53081179 +1.27594149j]
899  8.87 4.2Gb gainopts(StefCal.py:684:get_result): ***DEBUG*** model0 ('0', '1') [-0.14268520+0.33004239j -0.12173992+0.19450797j -0.12419887+0.10455056j
900  -0.18196888+0.0679823j  -0.28577715+0.03487613j -0.37953067-0.0534543j
901  -0.39933929-0.20536642j -0.32939655-0.36073738j -0.22604840-0.43268454j
902  -0.18194224-0.37738332j]
903  8.87 4.2Gb gainopts(StefCal.py:707:get_result): ***DEBUG*** model ('0', '1') [-0.14268520+0.33004239j -0.12173992+0.19450797j -0.12419887+0.10455056j
904  -0.18196888+0.0679823j  -0.28577715+0.03487613j -0.37953067-0.0534543j
905  -0.39933929-0.20536642j -0.32939655-0.36073738j -0.22604840-0.43268454j
906  -0.18194224-0.37738332j]
907  8.87 4.2Gb gainopts(StefCal.py:708:get_result): ***DEBUG*** model0 ('0', '1') [-0.14268520+0.33004239j -0.12173992+0.19450797j -0.12419887+0.10455056j
908  -0.18196888+0.0679823j  -0.28577715+0.03487613j -0.37953067-0.0534543j
909  -0.39933929-0.20536642j -0.32939655-0.36073738j -0.22604840-0.43268454j
910  -0.18194224-0.37738332j]
911  8.87 4.2Gb gainopts(StefCal.py:724:get_result): resetting data and model to initial values
912  8.88 4.2Gb gainopts(StefCal.py:751:get_result): ('0', '1') data type of data is complex64
913  8.88 4.2Gb gainopts(StefCal.py:752:get_result): ('0', '1') data type of model is complex64
914  8.88 4.2Gb gainopts(StefCal.py:777:get_result): G: solvable 1 from major loop 0 (current 0)
915  9.11 4.2Gb gainopts(StefCal.py:1276:run_gain_solution): solving for G, initial chisq is 0.258277427137
916 Traceback (most recent call last):
917   File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 783, in get_result
918     flagged |= self.run_gain_solution(opt,model,data,weight,bitflags,flag_null_gains=True,looptype=looptype);
919   File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 1285, in run_gain_solution
920     bounds=gopt.bounds if niter>2 else None);
921   File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/GainDiagPhase.py", line 204, in iterate
922     gaindiff2[p][pmask] = 0;
923 IndexError: in the future, 0-d boolean arrays will be interpreted as a valid boolean index
924 3.1Gb meqserver(meqserver.py:288:stop_default_mqs): meqserver not exited yet, waiting another 10 seconds
925 ### Job result: None
926 ### No more commands
927 ### meqserver reported 3 error(s) during the run:
928 ###   000: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.0.1)
929 ###   001: node 'VisDataMux': error processing footer 0.0.0
930 ###   002: node 'VisDataMux': execute() failed: in the future, 0-d boolean arrays will be interpreted as a valid boolean index (return code 0x810021)
931 ### Stopping the meqserver
932 ### All your batch are not belong to us, returning with error code
933 Traceback (most recent call last):
934   File "/code/run.py", line 202, in <module>
935     run_meqtrees(msname)
936   File "/code/run.py", line 170, in run_meqtrees
937     utils.xrun(cab['binary'], args)
938   File "/utils/utils/__init__.py", line 72, in xrun
939     raise SystemError('%s: returns errr code %d'%(command, process.returncode))
940 SystemError: /usr/bin/meqtree-pipeliner.py: returns errr code 1
941 running: /usr/bin/meqtree-pipeliner.py --mt 2 -c /code/tdlconf.profiles [stefcal] ms_sel.input_column=DATA ms_sel.field_index=0 ms_sel.msname=/home/sphe/msdir/meerkat_simulation_example.ms stefcal_gain.table=/home/sphe/msdir/meerkat_simulation_example.ms/meerkat_simulation_example.gain.cp stefcal_diffgain.freqsmooth=1 ms_wfl.write_bitflag=test tiggerlsm.lsm_subset=all do_ou    tput=CORRECTED_DATA stefcal_gain.flag_chisq=0 ms_sel.ms_fill_legacy_flags=1 stefcal_gain.flag_ampl=0 ms_sel.ddid_index=0 ms_sel.ms_write_flag_policy=add to set stefcal_diffgain.timesmooth=1 ms_rfl.read_flagsets=-stefcal stefcal_gain.freqint=1 stefcal_gain.implementation=GainDiagPhase stefcal_gain.flag_chisq_threshold=10 ms_rfl.read_legacy_flags=1 stefcal_gain.flag_ampl_low=    0.3 stefcal_gain.flag_ampl_high=2 ms_sel.ms_write_flags=1 stefcal_gain.mode=solve-save tiggerlsm.filename=/input/nvss1deg.lsm.html stefcal_gain.timeint=1 ms_sel.output_column=CORRECTED_DATA /usr/lib/python2.7/dist-packages/Cattery/Calico/calico-stefcal.py =stefcal
o-smirnov commented 7 years ago

921 File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/GainDiagPhase.py", line 204, in iterate 922 gaindiff2[p][pmask] = 0;

Not in office ATM, but, I think "pmask" needs to be changed to "mask".

On Thu, Feb 23, 2017 at 5:09 PM, Benjamin Hugo notifications@github.com wrote:

Assigned #60 https://github.com/ska-sa/meqtrees-cattery/issues/60 to @o-smirnov https://github.com/o-smirnov.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/60#event-974504076, or mute the thread https://github.com/notifications/unsubscribe-auth/AGK5v4W4WtO4iVvHGak_F6aIhWHcHDpiks5rfaEigaJpZM4MKFMS .

Sebokolodi commented 7 years ago

I'm finding the same error too when trying to use GainDiagPhase/GainDiagGainDiang2a/GainDiagCommon but a slightly different one when using Gain2x2. For the latter, it complains about the 'int' object not having an attribute 'dtype', see below: 

169.65 28.9Gb gainopts(StefCal.py:724:get_result): resetting data and model to initial values
169.65 28.9Gb gainopts(StefCal.py:751:get_result): ('0', '1') data type of data is complex64
169.65 28.9Gb gainopts(StefCal.py:752:get_result): ('0', '1') data type of model is complex64
169.65 28.9Gb gainopts(StefCal.py:777:get_result): G: solvable 1 from major loop 0 (current 0)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 783, in get_result
    flagged |= self.run_gain_solution(opt,model,data,weight,bitflags,flag_null_gains=True,looptype=looptype);
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 1271, in run_gain_solution
    self.compute_chisq(model,data,gopt.solver,weight=weight,bitflags=bitflags);
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 1201, in compute_chisq
    res = gain.residual(model,data,pq);
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/Gain2x2.py", line 352, in residual
    c = self.apply(lhs,pq,cache=cache,tiler=tiler);
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/Gain2x2.py", line 390, in apply
    lhs = self._get_matrix(p,q,lhs);
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/Gain2x2.py", line 106, in _get_matrix
    return map(self._tile_complex_data,data[p,q]);
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/Gain2x2.py", line 102, in _tile_complex_data
    return self.tile_data(x, dtype=self._dtype)
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/DataTiler.py", line 50, in <lambda>
    self.tile_data = lambda x, dtype=None: x.astype(dtype) if ( dtype and dtype != x.dtype ) else x
AttributeError: 'int' object has no attribute 'dtype'
2.5Gb meqserver(meqserver.py:288:stop_default_mqs): meqserver not exited yet, waiting another 10 seconds
### Job result: None
### No more commands
### meqserver reported 3 error(s) during the run:
###   000: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.0.1)
###   001: node 'VisDataMux': error processing footer 0.0.0
###   002: node 'VisDataMux': execute() failed: 'int' object has no attribute 'dtype' (return code 0x810021)
### Stopping the meqserver
### All your batch are not belong to us, returning with error code
running: /usr/bin/meqtree-pipeliner.py --mt 2 -c /code/tdlconf.profiles [stefcal] ms_sel.input_column=DATA ms_sel.field_index=0 ms_sel.msname=/home/sebokolodi/msdir/CYG-ABCD-6680-64CH-10S.ms stefcal_gain.table=/home/sebokolodi/msdir/CYG-ABCD-6680-64CH-10S.ms/CYG-ABCD-6680-64CH-10S-gain1-only.gain.cp stefcal_diffgain.freqsmooth=1 tiggerlsm.lsm_subset=all stefcal_gain.freqint=1 stefcal_gain.flag_chisq=0 do_output=CORR_RES stefcal_gain.flag_ampl=0 ms_sel.ddid_index=0 stefcal_diffgain.timesmooth=1 stefcal_gain.implementation=Gain2x2 stefcal_gain.flag_chisq_threshold=10 stefcal_gain.flag_ampl_low=0.3 stefcal_gain.flag_ampl_high=2 stefcal_gain.mode=solve-save tiggerlsm.filename=/home/sebokolodi/output/CYG-ABCD-6680-64CH-10S-1-apparent.lsm.html stefcal_gain.timeint=1 ms_sel.output_column=CORRECTED_DATA /usr/lib/python2.7/dist-packages/Cattery/Calico/calico-stefcal.py =stefcal
Traceback (most recent call last):
  File "/code/run.py", line 190, in <module>
    run_meqtrees(msname)
  File "/code/run.py", line 158, in run_meqtrees
    utils.xrun(cab['binary'], args)
  File "/utils/utils/__init__.py", line 72, in xrun
    raise SystemError('%s: returns errr code %d'%(command, process.returncode))
o-smirnov commented 7 years ago

Does this dataset have XY/YX columns? Are you perhaps using "2x2, diagonal terms only" in the Compile-time options? Trying to narrow down the failure mode here...

Sebokolodi commented 7 years ago

Yes, my MS has 4 polarisation products (RR, LL, RL and LR). @SpheMakh does stimela have an option to specify which correlations to use (2x2, 2x2 diagonal terms only, 2 or 1)?.

@o-smirnov, you can find the MS file I'm using on elwood /home/sebokolodi/Imaging_Cygnus_Challenge/C-Band/SelfCal/StimelaSelfcal. I am using Stimela but it doesn't seem to save the config file. @SpheMakh do I have to create the config file myself?.

SpheMakh commented 7 years ago

check calibrator documentation by running stimela cabs -i calibrator

On Mon, 6 Mar 2017 at 14:31 Lerato notifications@github.com wrote:

Yes, my MS has 4 polarisation products (RR, LL, RL and LR). @SpheMakh https://github.com/SpheMakh does stimela have an option to specify which correlations to use (2x2, 2x2 diagonal terms only, 2 or 1)?.

@o-smirnov https://github.com/o-smirnov, you can find the MS file I'm using on elwood /home/sebokolodi/Imaging_Cygnus_Challenge/C-Band/SelfCal/StimelaSelfcal. I am using Stimela but it doesn't seem to save the config file. @SpheMakh https://github.com/SpheMakh do I have to create the config file myself?.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meqtrees-cattery/issues/60#issuecomment-284383763, or mute the thread https://github.com/notifications/unsubscribe-auth/AGUUGKab7qTOoZbO6fzmDS2PBOwJ2PBAks5ri_ysgaJpZM4MKFMS .

SpheMakh commented 7 years ago

save-config option now available in stimela calibrator/simulator. https://github.com/SpheMakh/Stimela/pull/96

bennahugo commented 7 years ago

@o-smirnov Example to reproduce is here hugo@ray ~/DATASETS/cattery_bug Fails: make calibrate_diag Succeeds: make calibrate_2x2

KshitijT commented 7 years ago

I get similar error while using "GainDiag" : 

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 783, in get_result
    flagged |= self.run_gain_solution(opt,model,data,weight,bitflags,flag_null_gains=True,looptype=looptype);
  File "/usr/local/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 1285, in run_gain_solution
    bounds=gopt.bounds if niter>2 else None);
  File "/usr/local/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/GainDiag.py", line 200, in iterate
    gaindiff2[p][pmask] = 0;
IndexError: in the future, 0-d boolean arrays will be interpreted as a valid boolean index
2.2Gb meqserver(meqserver.py:288:stop_default_mqs): meqserver not exited yet, waiting another 10 seconds
### Job result: None
### No more commands
### meqserver reported 15 error(s) during the run:
###   000: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.0.1)
###   001: node 'VisDataMux': error processing tile 0.0.0.0.1.1
###   002: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.1.1)
###   003: node 'VisDataMux': error processing tile 0.0.0.0.1.2
###   004: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.2.1)
###   005: node 'VisDataMux': error processing tile 0.0.0.0.1.3
###   006: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.3.1)
###   007: node 'VisDataMux': error processing tile 0.0.0.0.1.4
###   008: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.4.1)
###   009: node 'VisDataMux': error processing tile 0.0.0.0.1.5
###   010: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.5.1)
###   011: node 'VisDataMux': error processing tile 0.0.0.0.1.6
###   012: node 'stefcal': <2 exceptions> (while getting result for request ev.0.0.0.6.1)
###   013: node 'VisDataMux': error processing footer 0.0.0
###   014: node 'VisDataMux': execute() failed: in the future, 0-d boolean arrays will be interpreted as a valid boolean index (return code 0x810021)
### Stopping the meqserver
### All your batch are not belong to us, returning with error code
Traceback (most recent call last):
  File "/code/run.py", line 210, in <module>
running: /usr/bin/meqtree-pipeliner.py --mt 16 -c /code/tdlconf.profiles [stefcal] ms_sel.input_column=DATA ms_sel.field_index=1 ms_sel.msname=/home/kshitij/msdir/12A-405.sb7601493.eb10633016.56086.127048738424.ms stefcal_gain.table=/home/kshitij/output/12A-405.sb7601493.eb10633016.56086.127048738424.gain.cp stefcal_diffgain.freqsmooth=1 tiggerlsm.lsm_subset=all stefcal_gain.freqint=10 stefcal_gain.flag_chisq=0 do_output=CORR_RES stefcal_gain.flag_ampl=1 ms_sel.ddid_index=0 ms_sel.tile_size=512 stefcal_diffgain.timesmooth=1 ms_rfl.read_flagsets=-stefcal stefcal_gain.implementation=GainDiag stefcal_gain.flag_chisq_threshold=5 ms_rfl.read_legacy_flags=1 stefcal_gain.flag_ampl_low=0.15 ms_sel.ms_corr_sel='2x2' stefcal_gain.flag_ampl_high=3.5 stefcal_gain.mode=solve-save tiggerlsm.filename=/home/kshitij/output/vla_NGC4157_LBand-LSM2.lsm.html stefcal_gain.timeint=10 ms_sel.output_column=CORRECTED_DATA /usr/local/lib/python2.7/dist-packages/Cattery/Calico/calico-stefcal.py =stefcal -s /home/kshitij/output/selfcal_2nd_round
    run_meqtrees(msname)
  File "/code/run.py", line 178, in run_meqtrees
    utils.xrun(cab['binary'], args + ['-s {}'.format(saveconf) if saveconf else ''])
  File "/utils/utils/__init__.py", line 72, in xrun
    raise SystemError('%s: returns errr code %d'%(command, process.returncode))
SystemError: /usr/bin/meqtree-pipeliner.py: returns errr code 1

Similar fix, I assume, @o-smirnov , @SpheMakh ?

mcoriat commented 6 years ago

Not sure that it is related (sorry if it isn't) but I get an error when using "GainDiagCommon" for the diffgain term. It complains about GainDiagCommon not having "_dtype" attribute. Works fine when using "GainDiag" though.
I am using "2x2 diagonal term only" at compile time and GainDiag for the main gain term.

Successful readonly open of autonoread-locked table cirx.9G.64ch.ms: 27 columns, 52500 rows
Successful readonly open of autonoread-locked table /mnt/data/Mickael/Cirx1/9G/cirx.9G.64ch.ms/ANTENNA: 8 columns, 6 rows
Successful readonly open of default-locked table /mnt/data/Mickael/Cirx1/9G/cirx.9G.64ch.ms/OBSERVATION: 9 columns, 1 rows
Successful readonly open of autonoread-locked table /mnt/data/Mickael/Cirx1/9G/cirx.9G.64ch.ms/POLARIZATION: 4 columns, 1 rows
Successful readonly open of autonoread-locked table /mnt/data/Mickael/Cirx1/9G/cirx.9G.64ch.ms/DATA_DESCRIPTION: 3 columns, 1 rows
Successful readonly open of autonoread-locked table /mnt/data/Mickael/Cirx1/9G/cirx.9G.64ch.ms/SPECTRAL_WINDOW: 15 columns, 1 rows
Successful readonly open of autonoread-locked table /mnt/data/Mickael/Cirx1/9G/cirx.9G.64ch.ms/FIELD: 9 columns, 1 rows
Using LSM module from Tigger (1.3.8) 1.3.8 at /usr/lib/python2.7/dist-packages/Tigger (in path)
Loading cirx9G.4.lsm.html: Tigger sky model
applied =dE (involving 1 sources), 1 sources now selected
Number of diffgain predict groups: 2
3070.22 4.9Gb gainopts(StefCal.py:169:update_state): initializing options for G
3070.23 4.9Gb gainopts(StefCal.py:169:update_state): initializing options for B
3070.23 4.9Gb gainopts(StefCal.py:178:update_state): initializing options for dE
3070.23 4.9Gb gainopts(StefCal.py:182:update_state): initializing specific options for dE:A180
3070.23 4.9Gb gainopts(StefCal.py:188:update_state): using float di False dd False
Removing gain.cp as requested
gain1.cp does not exist, so not trying to remove
Removing diffgain.cp as requested
ifrgains.ma does not exist, so not trying to remove
input flagmask is
3078.86 5.0Gb gainopts(StefCal.py:257:get_result): get_result entry
3078.86 5.0Gb gainopts(GainOpts.py:218:load_initval): not loading G solutions: gain.cp does not exist
3078.86 5.0Gb gainopts(GainOpts.py:218:load_initval): not loading dE:A180 solutions: diffgain.cp does not exist
3078.86 5.0Gb gainopts(StefCal.py:279:get_result): new dataset id 6
3078.86 5.0Gb gainopts(GainOpts.py:287:resolve_tilings): datashape (3500, 64) expanded datashape is (3500, 64)
3078.86 5.0Gb gainopts(GainOpts.py:294:resolve_tilings): gain tiling is [1, 1] smoothing is (10, 20)
3078.87 5.0Gb gainopts(GainOpts.py:294:resolve_tilings): diffgain tiling is [1, 64] smoothing is (300, 0)
3078.87 5.0Gb gainopts(GainOpts.py:295:resolve_tilings): based on an LCM tiling of [1, 1]
3079.78 5.0Gb gainopts(StefCal.py:484:get_result): constructed internal arrays, trying to release array memory
3079.82 5.0Gb gainopts(StefCal.py:487:get_result): released memory
3079.82 5.0Gb gainopts(StefCal.py:494:get_result): Found 6 solvable antennas
3079.82 5.0Gb gainopts(StefCal.py:495:get_result):   valid ifrs outside the solvable set: 
3079.82 5.0Gb gainopts(StefCal.py:496:get_result):   ifrs with no data: 
3080.00 5.0Gb gainopts(StefCal.py:629:get_result): 15 baselines with non-zero weights (0 with null weight, 0 have no valid data)
3080.24 5.0Gb gainopts(StefCal.py:651:get_result): Found 6 solvable antennas
3080.24 5.0Gb gainopts(StefCal.py:653:get_result): 7 valid visibility matrices per time/freq slot are required for a solution
3080.24 5.0Gb gainopts(StefCal.py:654:get_result):    for this data chunk, Nvalid ranges from 0 to 15
3080.24 5.0Gb gainopts(StefCal.py:676:get_result): Solvable: 15 of 15 inteferometers (15 have valid data), with 6 solvable antennas
3080.24 5.0Gb gainopts(GainOpts.py:305:init_solver): stefcal G solve=1 GainDiag, using 15 solvable inteferometers
3080.24 5.0Gb gainopts(GainOpts.py:306:init_solver):   solution intervals: [1, 1] smoothing kernel: (10, 20)
3080.24 5.0Gb gainopts(GainOpts.py:313:init_solver):   default initial value is 1
3080.24 5.0Gb gaindiag(GainDiag.py:94:__init__): convergence target 212800 of 224000 real slots
3080.24 5.0Gb gainopts(GainOpts.py:320:init_solver):   subshape (3500, 64) tiled (3500, 64)
3080.24 5.0Gb gainopts(GainOpts.py:305:init_solver): stefcal dE:A180 solve=1 GainDiagCommon, using 15 solvable inteferometers
3080.24 5.0Gb gainopts(GainOpts.py:306:init_solver):   solution intervals: [1, 64] smoothing kernel: (300, 0)
3080.24 5.0Gb gainopts(GainOpts.py:313:init_solver):   default initial value is 1
3080.24 5.0Gb gaindiag(GainDiagCommon.py:93:__init__): convergence target 3325 of 3500 real slots
3080.24 5.0Gb gainopts(GainOpts.py:320:init_solver):   subshape (3500, 1) tiled [3500, 1, 64]
3080.24 5.0Gb gainopts(StefCal.py:683:get_result): ***DEBUG*** data ('1', '2') [ 0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j
  0.+0.j]
3080.24 5.0Gb gainopts(StefCal.py:684:get_result): ***DEBUG*** model0 ('1', '2') [ 0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j
  0.+0.j]
3080.24 5.0Gb gainopts(StefCal.py:689:get_result): ***DEBUG*** dgms ('1', '2') [ 0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j
  0.+0.j]
3080.25 5.0Gb gainopts(StefCal.py:694:get_result): adding dE-enabled terms into model
3080.47 5.1Gb gainopts(StefCal.py:707:get_result): ***DEBUG*** model ('1', '2') [ 0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j
  0.+0.j]
3080.47 5.1Gb gainopts(StefCal.py:708:get_result): ***DEBUG*** model0 ('1', '2') [ 0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j  0.+0.j
  0.+0.j]
3080.47 5.1Gb gainopts(StefCal.py:724:get_result): resetting data and model to initial values
3080.47 5.1Gb gainopts(StefCal.py:751:get_result): ('1', '2') data type of data is complex128
3080.47 5.1Gb gainopts(StefCal.py:752:get_result): ('1', '2') data type of model is complex128
3080.47 5.1Gb gainopts(StefCal.py:777:get_result): G: solvable 1 from major loop 0 (current 0)
3081.54 5.2Gb gainopts(StefCal.py:1276:run_gain_solution): solving for G, initial chisq is 0.250731584419
3090.02 5.2Gb gainopts(StefCal.py:1287:run_gain_solution): iter 1: 0.03% (64/224000) conv, 0 gfs, max update 0.847868
3090.94 5.3Gb gainopts(StefCal.py:1304:run_gain_solution): iter 1: ||G||=3.80778 max update is 0.847868 chisq is 0.25006446 (0.00066290394) diff 0.00267, 64/224000 conv
3099.47 5.3Gb gainopts(StefCal.py:1287:run_gain_solution): iter 2: 0.03% (64/224000) conv, 0 gfs, max update 0.127563
3100.39 5.3Gb gainopts(StefCal.py:1304:run_gain_solution): iter 2: ||G||=3.82496 max update is 0.127563 chisq is 0.25006014 (0.00066288761) diff 1.73e-05, 64/224000 conv
3100.40 5.3Gb gainopts(StefCal.py:1317:run_gain_solution): iter 2: chisq converged 113 to 1e-06, 926 to 1e-05, 8962 to 0.0001, 68499 to 0.001, 100459 to 0.01; diverging 99041; significantly 5699
3108.76 5.3Gb gainopts(StefCal.py:1287:run_gain_solution): iter 3: 0.03% (64/224000) conv, 0 gfs, max update 0.0232101
3109.71 5.3Gb gainopts(StefCal.py:1304:run_gain_solution): iter 3: ||G||=3.8255 max update is 0.0232101 chisq is 0.25006031 (0.00066288715) diff -6.53e-07, 64/224000 conv
3109.72 5.3Gb gainopts(StefCal.py:1317:run_gain_solution): iter 3: chisq converged 843 to 1e-06, 8758 to 1e-05, 66876 to 0.0001, 99378 to 0.001, 99426 to 0.01; diverging 100074; significantly 0
3109.72 5.3Gb gainopts(StefCal.py:1348:run_gain_solution): G chisq diverging, stopping at 0.250060307551 after 3 iterations and 28.18s
3109.72 5.3Gb gainopts(StefCal.py:1361:run_gain_solution): Final chi-sq 0.25006 higher than minimum 0.25006 achieved at iteration 2, rolling back.
3109.73 5.3Gb gainopts(StefCal.py:1369:run_gain_solution):   delta-chisq were 0.002668 1.726e-05 -6.533e-07
3109.73 5.3Gb gainopts(StefCal.py:1370:run_gain_solution):   convergence criteria were 0.85 0.13 0.023
3109.80 5.3Gb gainopts(StefCal.py:1430:run_gain_solution): M (median chisq value): [ 0.24382947]
3109.81 5.3Gb gainopts(StefCal.py:1435:run_gain_solution): flagging 0.000000% of data (0/199500 slots) due to chi-sq > M*5.0
3109.81 5.3Gb gainopts(StefCal.py:1439:run_gain_solution):   for threshold M*2.0 this would have been 0.083208%
3109.81 5.3Gb gainopts(StefCal.py:1439:run_gain_solution):   for threshold M*3.0 this would have been 0.003509%
3109.81 5.3Gb gainopts(StefCal.py:1439:run_gain_solution):   for threshold M*5.0 this would have been 0.000000%
3109.81 5.3Gb gainopts(StefCal.py:1439:run_gain_solution):   for threshold M*10.0 this would have been 0.000000%
3109.82 5.3Gb gainopts(StefCal.py:1439:run_gain_solution):   for threshold M*20.0 this would have been 0.000000%
3109.82 5.3Gb gainopts(StefCal.py:786:get_result): applying G-inverse to data
3110.35 5.4Gb gainopts(StefCal.py:790:get_result): done
3110.68 5.4Gb gainopts(StefCal.py:829:get_result): dE:A180 ('3', '6') data type of dgmodel is complex128
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/StefCal.py", line 830, in get_result
    corr = dgopt.solver.apply(dgmodel[idg],pq);
  File "/usr/lib/python2.7/dist-packages/Cattery/Calico/OMS/StefCal/GainDiag.py", line 361, in apply
    for d,(i,j) in zip(lhs,IJ2x2) ];
AttributeError: 'GainDiagCommon' object has no attribute '_dtype'