search

revoltek / losoto

LOFAR solutions tool
GNU General Public License v3.0
11 stars 21 forks source link

Strange merge of fulljones h5parms from killMS2H5parm with H5parm_collector #105

Open tikk3r opened 4 years ago

tikk3r commented 4 years ago

Initiall reported here: https://github.com/lmorabit/lofar-vlbi/issues/70

After converting killMS solutions to fulljones h5parms with killMS2H5parm.py, merging these with H5parm_collector.py behaves a bit strange. For example, in the plots below the merged h5parm shows nothing for 164 MHz at an arbitrary timeslot, yet there are valid solutions there in the specific 164 MHz h5parm.

It appears that it gets confused about the axes or something like that, giving adjacent timeslots different "chunks of frequency". Converting them as only XX and YY with --nofulljones does not show this problem.

The plots show the merged h5parm in the first row and two example h5parms from individual 2 MHz blocks in the second row.

Time slot 2 Time slot 3
CS301_HBA1_concatenated_ddf_solutions2 CS301_HBA1_concatenated_ddf_solutions
150 MHz block timeslot 2 164 MHz block timeslot 2
CS301_HBA1_164MHz_ddf_solutions CS301_HBA1_150MHz_ddf_solutions
revoltek commented 4 years ago

yes, most likely it's a problem of axes ordering. I have tested that code only with LBA and only in a specific format. Are all h5parms at least with the same axes order/polarisations? It's dangerous it doesn't show any error though...

tikk3r commented 4 years ago

They have the same structure. I just realize I already mention a possible answer in the other issue. The axes do have the same order, but they do not have the same length necessarily. The kMS solutions can have different time and frequency axes per block. I guess the collector does not like that?

revoltek commented 4 years ago

I use the collector only to merge equal tables, have a look at the H5parm_interpolator.py, which has also a number of limitations. Coding a very flexible tool to merge tables is a bit of a pain, currently I've only these 2 ready

tikk3r commented 4 years ago

I tried H5parm_interpolator out, but it didn't quite do what I expected. What is it intended to do? I tried running it and it produced a single h5parm with apparently all the solutions added/multiplied (it seems) into a single solset and single amplitude and phase soltabs. The frequency axis only has the range of the first one and the amplitudes seem to all have been multiplied or something as now they are suddenly of order 100 where they are of order 1 in the input H5parms.

H5parm_interpolator sounds like what I'm looking for though, so I'll poke around in it a bit more.

revoltek commented 4 years ago

I use it to take all fast phase and slow amp solutions from many directions and combine them into a single h5parm to be used with DDF - I tested it only in a few cases and there's a number of possible restrictions that I probably never though about... if you come up with a more general version that would be great

tikk3r commented 4 years ago

Ah I see, that makes sense.

I think because each h5parm covers a different frequency range it "breaks" the interpolation in this case as well as that makes it not just interpolating to an identical, but faster grid but needing to interpolate different ranges of values to a common interval. I'm definitely in a non-standard messy case I think.