search

saopicc / killMS

Package for direction dependent calibration
GNU General Public License v2.0
9 stars 3 forks source link

KillMS error with parameter TChunk #13

Open astro-fgao opened 2 years ago

astro-fgao commented 2 years ago

Hi there,

I tried to run kMS on some Meerkat data (only 2 mins scan) and I found that if I set the TChunk parameter smaller than the targeted data itself, then there will be en error message complaining about not finding data column (as shown below)

kms-TChunk-error

I've tried with different TChunk length and it seems as long as this parameter is smaller than the data, then this error will occur. Currently I'm using a large TChunk and everything is fine, but I can see this becoming a problem e.g. when working on some science targets with long scans.

FYI, the command I used are:

kMS.py --MSName="../../msdir/1579141864_sdp_l0.POLCALIBRATED.ms" --FieldID=17 --SolverType=KAFCA --PolMode=Scalar --BaseImageName=image_DI_Clustered.DeeperDeconv --dt 5 --NCPU 96 --OutSolsName DD0 --NChanSols 20 --InCol CORRECTED_DATA --TChunk 5 --BeamModel FITS --FITSParAngleIncDeg 0.5 --FITSFile='/localwork/feng/data/widefield_pol/fgao/beam-model/meerkat_pb_jones_cube95channels$(xy)_$(reim).fits' --CenterNorm 1 --FITSFeed xy --FITSFeedSwap 1 --ApplyPJones 1 --FlipVisibilityHands 1 --NChanBeamPerMS 20

Thank you ~

cyriltasse commented 2 years ago

Well - it's weird - there seem to be a problem reading DATA_DESC_ID in the MS... Or maybe it doesn't see the DATA_DESC_ID in the current chunk?

Also - I don't get it - you seem to be using TChunk=5, but the loaded chunk is only one hour?

How large is the MS? Could you post it somewhere with everything needed so I could reproduce?

astro-fgao commented 2 years ago

Hi Cyril,

Update on this issue:

the .ms data I used above contains multiple fields, and I'm only using one field there. Today I found that if I split out this field, then run ddf/kms on this single field data, kms can finish successfully even if I used a TChunk smaller than the data itself.

Instead, if I don't split the data, then the error would occur.

Currently I can trim the MS down to few hundreds Megabites, and I'm still looking for a place to post everything for you to try.

Also - I don't get it - you seem to be using TChunk=5, but the loaded chunk is only one hour?

Sorry, I copy/paste the wrong command line, which got you confused. In this case above, I was actually using TChunk=1, the total data chunk is like 4 something, this would cause the error message. When I was using TChunk=5, everything is fine.