Improve Panstarrs cross matching

[twshimwell]

In offsets.py we attempt to match with Panstarrs to derive offsets per facet after removing extended sources from the LOFAR catalogues.

Current filtering in master branch is very crude (filter=(lofar['E_RA']3600.0)<2.0, filter&=(lofar['E_DEC']3600.0)<2.0, filter&=(lofar['Maj']*3600.0)<20). At low declinations and low galactic lattitude this filtering removes far too many LOFAR sources and the cross matching with Panstarrs doesnt work very well as a result.

In offsets_compact branch ive tweaked the filtering so that it corrects for radial smearing before finding an envelope to separate compact from extended sources (see attached plots - saved as "radial-correction.png" and "fitted-skew-envelope.png" in offsets.py). This leaves far more sources for cross matching compared to the old approach.

[twshimwell]

I tested this new offsets criteria for about 1,400 fields. For each field I ran offsets.py 3 times. Once with the old criteria, again with the new one and finally with simply using all sources in the facet for the panstarrs cross match.

The histogram below shows the derived astrometric error from offsets.py for these 3 different types. Here its clear the old version does the worst (so fewer facets with low astrometry errors) and that the new criteria does the best (the most facets with small astrometric errors) and does very slightly better than simply using all sources which is nice.

So I think this new criteria is all ready to go.

[mhardcastle]

Hurray! Will leave this open till we merge the new branch with master.

[mhardcastle]

PS the code breaks Python 2 compatibility thanks to using a later version of scipy than Py2 supports, but I think we can live with that at this point.

[mhardcastle]

Adding here the possible issue with RA/Dec offsets around the north pole to be looked into before we can close this out.

[mhardcastle]

So I think the correct way to deal with this is:

• project the positions of the LOFAR and optical sources on to a tangent plane (can use astropy WCS without an actual image; use the same projection that the images use)
• calculate offsets in that space with units of pixels in x and y
• in mosaicing apply those offsets translated to the projection of the mosaic (so to get pixel offsets go to RA, Dec and back).

[twshimwell]

That sounds good. I'm guessing this would also need for offsets.py to be rerun after code changes. Not much of an issue because the expensive bit was the pybdsf step which I dont think will need repeating.

[mhardcastle]

Yes that's right, will require changes to offsets.py .

For the immediate work I'll need a copy of the pybdsf catalogues for each field where you have them.