Reformatting of output spectral cubes for polarization

[sposullivan]

In order to efficiently incorporate RM synthesis into the pipeline, we would like to have the Q and U image cubes in separate files, so two output cubes instead of one, with ra,dec,freq for Stokes Q and ra,dec,freq for Stokes U. Currently they are put in a 4D compressed format (ra,dec,stokes,freq) using do_polcubes.py. This requires some re-arrangement before input to RM synthesis that is time consuming. It would also be good to have the option to write out the I and V spectral cubes for selected cases.

[twshimwell]

I've got a version with Q and U separated. As we do --RIME-PolMode=QU I am assuming

h[0].data[:,0,:,:] is Q and h[0].data[:,1,:,:] is V.

If there are no concerns about doing it this way then I can push to master.

I havent looked at I and V ones yet as we dont make those images at present. @sposullivan is this something that you routinely want or is it more just something to have as an option?

[sposullivan]

The I and V would be fine as an option later. The correct way to do RM synthesis is with Q/I and U/I, but I guess several upstream issues need to improved first before the I channel imaging is reliable. Typo: h[0].data[:,1,:,:] is V. --> U

[twshimwell]

This is now in branch separate_QU (not the V and I cubes but just the separate QU ones).

Its a copy of what id tested before so hopefully works but will test again next time I make some cubes.

[twshimwell]

The separate_QU branch seems to work if we wanted to merge this into master.

[mhardcastle]

Hi Tim, what I guess I'd prefer is to put this in as an option so that we can still get the old behaviour if we need it.

Also, I don't know what you would intend to do about existing processing but if we do this now then once I do a git pull we will suddenly have two incompatible outputs in the archives (assuming that the archiving code picks up the separated cubes correctly) and that doesn't strike me as obviously the right thing to do unless we commit to going through and separating the existing QU data (with some small loss of quality since we will need to uncompress, separate and compresss again).

[twshimwell]

Good point. I'll make it optional. For 6" pol imaging I was finding that I just didnt have enough memory on my nodes to do the combined cubes so if in the future we are able to switch to that (the compressed cubes are coming out at about 200GB in total - so including Q U and dirty and dirty.corr) then we could switch the default behaviour. But yeh in general for the resolutions we are making at default at present we could just split when required.

Perhaps @sposullivan has a thoughts on this?

[sposullivan]

I think we need some tests first with Aritra's RM synthesis code on separate 6" Q and U cubes, and see what sort of increase we get in the number of polarized sources. So probably best to have it as an option now (not the default). Any 6" QU cubes would need to be listed under a different heading on the website anyway. For the existing 20" combined QU cubes, I don't think that we should uncompress, separate and compress again (probably easier to change Aritra's code if we want to work with them as they are).

If with the separate 6" Q U cubes we are finding, say, twice as many polarized sources and not losing any, then in the long term my preference would be to re-image the existing data to get separate 6" Q and U cubes and just present the dirty.corr ones on the website (if that is possible/not too expensive to do), along with the final RM synthesis outputs (ie. no RM cubes, just the final maps).

[twshimwell]

It takes about 1-1.5 days for the 6" Q U cubes from the archived data so its not crazy - this is for 12,000 x 12,000 1.5" pixels so we might be able to make it smaller and faster too. That reminds me that I should send you a 6" P337+36 image. I'll try to do that over the coming days.

[sposullivan]

Thanks! 10k x 1.5" pixels would match what I have done with the 20" cubes (I only analyse 4 x 4 sq deg). Let's see how the RM synthesis goes. We worked out some final bugs, so Aritra wanted to meet sometime soon to discuss the implementation (it's very easy, but the options should be optimised for the hardware).

[twshimwell]

OK this is all in there as an option now in this branch (defaulted to False). I shall attempt a pull request and see if I can get this into master without breaking everything...

Note that presently the QU imaging is still done only at low and vlow resolution but we can fix that later (again through options).

[twshimwell]

That pull request seemed to work so this ticket is now done.