problems reading Parm tables with GMRT data

Original comment thread migrated from bugzilla

at 2009-01-03 21:47:50 Oleg Smirnov replied:

I've hit an interesting snag trying to reduce GMRT data. I'm doing a G solution first, which comes out fine (see first image). Then, I ask for a B solution, and keep a G inspector open, and now the tracks look like the second image. The G's for timeslots 300 and 420 have mysteriopusly dropped to 0. All the other appear to be fine. I haven't yet noticed anything else that's special about these two timeslots.

I'll try to upload the MS to astron tomorrow (currently on my laptop), and I can give you a live demo of the problem in Malta. In the meantime, I just wanted the bug open because this is rather critical.

at 2009-01-03 21:48:38 Oleg Smirnov replied:

Created an attachment (id=52) G inspector after running a G solution

at 2009-01-03 21:49:15 Oleg Smirnov replied:

Created an attachment (id=53) G inspector after subsequent B solution

at 2009-01-03 21:50:39 Oleg Smirnov replied:

This has all the smell of yet another subtiling problem. We really need to sit down and consider my "domain ID" idea of a few weeks back, because that would really simplify all this subtiling business considerably.

at 2009-01-05 13:37:51 Maaijke Mevius replied:

(In reply to comment #2)

Created an attachment (id=53) [edit] G inspector after subsequent B solution

Something wrong with this attachment. try again.

at 2009-02-12 17:34:58 Oleg Smirnov replied:

Maaijke submitted by e-mail:

Hi Oleg,

I do notice in the requests many weird cellsizes in time of 0. I assume something goes wrong there. I can see that fastparmtables doesnot return the right funklets for a given domain. Especially the first one is missing in case of timeslot 300. Also most of the times I get 61 instead of 60 funklets (which is no problem because ComposedFunklet takes care of that, but it is weird if the domains are the same in the first and second run).

I found using your test_parmtables tool:

domlist[300]: MeqDomain({'freq':(353367187.5, 359820312.5),'time':(4570719878.4816732, 4570719878.4816742)})

domlist[301] MeqDomain({'freq':(353367187.5, 359820312.5),'time':(4570719886.8748569, 4570719903.6448574)})

As you can see the first timedomain is more or less 0. In the retrieve step, it needs to return funklets in the domain:

4570719878.4816742 4570722420.2270508

Which is the upper bound of funklet number 300, and that is why it is not returned. It would make more sense to me if the start of the domain of 301 would correspond to the end of domain 300...Could this be due to problems in the measurementset?? I also think that the problem occurs much more often, but we only notice it happens at the start of a new tile...

I will now check if we get the same problems if you use mep instead of fmep, but I do expect it since it already goes wrong with the request domains.

at 2009-02-12 19:07:10 Oleg Smirnov replied:

GMRT still makes me cry... indeed we end up with 0-sized time cells occasionally, which is due to missing rows in the MS, which in turn is due to destructive AIPS flagging. I'm therefore cc'ing Hans on this, to motivate him into becoming less destructive ASAP.

I still need to make the MS reader more robust against this sort of crap. Sigh...

at 2009-02-13 00:56:38 Oleg Smirnov replied:

Fixed as of revision 6748.

ratt-ru / meqtrees