'int' object is not subscriptable

[mhardcastle]

The continuing saga of upgrading ddf-pipeline to use the current saopicc releases of DDF and kMS.

DDF.py --Misc-ConserveMemory=1 --Output-Name=Predict_DI1 --Data-MS=mslist.txt --Deconv-PeakFactor 0.001000 --Data-ColName DATA_DI_CORRECTED_SCALED --Parallel-NCPU=96 --Beam-CenterNorm=1 --Deconv-CycleFactor=0 --Deconv-MaxMinorIter=1000000 --Deconv-MaxMajorIter=1 --Deconv-Mode SSD --Beam-Model=LOFAR --Weight-Robust -0.150000 --Image-NPix=20000 --CF-wmax 50000 --CF-Nw 100 --Output-Also onNeds --Image-Cell 1.500000 --Facets-NFacets=11 --SSDClean-NEnlargeData 0 --Freq-NDegridBand 1 --Beam-NBand 1 --Facets-DiamMax 1.5 --Facets-DiamMin 0.1 --Deconv-RMSFactor=3.000000 --SSDClean-ConvFFTSwitch 10000 --Data-Sort 1 --Cache-Dir=. --Cache-DirWisdomFFTW=. --Debug-Pdb=never --Log-Memory 1 --GAClean-RMSFactorInitHMP 1.000000 --GAClean-MaxMinorIterInitHMP 10000.000000 --DDESolutions-SolsDir=SOLSDIR --Cache-Weight=reset --Beam-LOFARBeamMode=A --Output-Mode=Predict --Predict-ColName DD_PREDICT --Output-RestoringBeam 12.000000 --Weight-ColName="IMAGING_WEIGHT" --Freq-NBand=2 --RIME-DecorrMode=FT --SSDClean-SSDSolvePars [S,Alpha] --SSDClean-BICFactor 0 --Mask-Auto=1 --Mask-SigTh=5.00 --Mask-External=image_ampphase1.app.restored.fits.mask.fits --DDESolutions-GlobalNorm=None --DDESolutions-DDModeGrid=AP --DDESolutions-DDModeDeGrid=AP --DDESolutions-DDSols=DDS1_smoothed --Predict-InitDicoModel=image_ampphase1m_masked.DicoModel --Selection-UVRangeKm=[0.100000,1000.000000] --GAClean-MinSizeInit=10

runs for a while and then gives

  Init CFs....................  26/96 [=============                                     ]  27%  - 0'26"
    Init CFs....................  27/96 [==============                                    ]  28%  - 0'26"
 - 03:15:48 - CacheManager                 [6.3/7.5 7.7/9.0 17.9Gb] writing cache hash ./L234026_125MHz_uv_pre-cal.ms.F0.D0.ddfcache/ch0/Flags.npy.hash
 - 03:15:49 - ClassVisServer               [6.0/7.5 7.4/9.0 18.7Gb]   channel Mapping Gridding  : [0, 0, 0, 0, 0, 0, 0, 0,  ...., 0, 0, 0, 0, 0, 0, 0, 0], (len = 20)
 - 03:15:49 - ClassVisServer               [6.0/7.5 7.4/9.0 18.7Gb]   channel Mapping DeGridding: [0, 0, 0, 0, 0, 0, 0, 0,  ...., 0, 0, 0, 0, 0, 0, 0, 0], (len = 20)
 - 03:15:49 - AsyncProcessPool             [6.0/7.5 7.4/9.0 18.7Gb] process io00: exception raised processing job DATA:0:0: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/DDFacet/Other/AsyncProcessPool.py", line 870, in _dispatch_job
    result = call(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/DDFacet/Data/ClassVisServer.py", line 597, in _handler_LoadVisChunk
    DATA["Weights"] = DATA["Weights"][DATA["sort_index"]]
TypeError: 'int' object is not subscriptable

DDF.py has run several times on these MSes in the course of the pipeline run, so it shouldn't be a problem with the data.

I'll investigate further but any ideas appreciated.

[o-smirnov]

Away from my laptop now, but I'm pretty sure it puts a single 0 in the weights field of the dict when there's no weight column.

[mhardcastle]

There is a weight column, so not entirely sure what's going on, but I think the breakage wrt previous version may be related to this change in ClassVisServer.py:

if DATA["sort_index"] is not None: # and DATA["Weights"] is not 1: # OMS 2023/12 they're not "1" ever and this seems a bug
            DATA["Weights"] = DATA["Weights"][DATA["sort_index"]]

[mhardcastle]

OK, what's happening is that DATA["Weights"]==1 is used as a flag for 'don't use the weights' in GetVisWeights:

    def GetVisWeights(self, iMS, iChunk):
        """
        Returns path to weights array for the given MS and chunk number.

        Waits for CalcWeights to complete (if running in background).
        """
        # wmax-only means weights not computed (i.e. predict-only mode)
        if self._ignore_vis_weights:
            return 1

This change around line 597 therefore breaks existing functionality and I think should be reverted. Can I submit a PR?

[o-smirnov]

Yes please! Sorry about the misguided change...