Manual init catalogs not being interpreted properly?

[drphilmarshall]

From Adri:

It seems that the initialization does something strange when reading in from .cat files; if you do

python LensTractor.py -n 4 -z -v examples/ps1/H*sci.fits --survey PS1

and then

python LensTractor.py -n 4 -z -v examples/ps1/H*sci.fits --survey PS1 --manual lenstractor_Nebula4.cat

you'll see that the last snapshot of the first passage does not correspond to the zeroth snapshot of the second passage! When reading from the catalogue, the point-sources are somehow scrambled and I don't understand why. Do you know what's happening?!

[drphilmarshall]

More from Adri:

I've tried running LT on the 0951 example, with manual initalisation otherwise it does stupid stuff, but there are serious problems with properly initialising the sources from the input catalog. I'm sending it attached, I've done it simply by drag-and-click with ds9 but then LensTractor puts the guys in completely wrong places! Try running and see...

python LensTractor.py -n 2 -z -v examples/... --survey SDSS --manual examples/.../init.txt

I've emailed you the init.txt file.

In model.py I have commented a line out (which was tuned on PS1) and inserted the correct option for the SQLS examples. There's a double round of optimization in drive.py, which I comment out when running Lens. Can you figure out what's happening?

[aagnello]

By now I'm trying to circumvent this by doing: -1- manual initialization plus cunning_and_guile -2- use that output as a manual input for two round of optimization with Nebula(2) -3- use the same output from pt.1 as a manual input for one round of Nebula(2)+Lens It has worked fine for 0951, but now when it runs on 0814 it wanders too much and crashes at the second round of mcmc. These are our minimum pass tests, it shouldn't fail on those!

[drphilmarshall]

OK, I am investigating the init catalog ingest.

[aagnello]

It was sort-of solved in 0814, now on 1242 it's doing a horrible mess. Even if I on't use manual, when Turnover gives the 0.1''random displacements they look more like many arcseconds away. There has to be something wrong with the SDSS headers or how they are read, @drphilmarshall is that what you're looking into already?

[drphilmarshall]

I was not checking WCS headers and their interpretation but I will now! The ball is firmly in my court.

On Tue, Jun 3, 2014 at 4:31 AM, Adriano Agnello notifications@github.com wrote:

It was sort-of solved in 0814, now on 1242 it's doing a horrible mess. Even if I on't use manual, when Turnover gives the 0.1''random displacements they look more like many arcseconds away. There has to be something wrong with the SDSS headers or how they are read, @drphilmarshall https://github.com/drphilmarshall is that what you're looking into already?

— Reply to this email directly or view it on GitHub https://github.com/davidwhogg/LensTractor/issues/44#issuecomment-44952664 .

[drphilmarshall]

OK, there was no actual bug here. The manual init code works just fine - the issue is that the magnitudes of the sources are not taken from the catalog, but instead set from the flux in the images, so they can be significantly brighter than the fitted versions from the previous LT run. In H1413 it seems pretty common for the Nebula galaxy to be used to fit the quasar images, while one or two of the point sources are just hidden away at large-ish radius and low flux. Here's a fairly typical result from a simple (not very good) optimization, where I am overlaying point sources in cyan and galaxy (back to exp, to try and stop it being used to fit point sources!) in orange:

Now, here's the initial plot when I try and use this result to initialize a Nebula4 model:

The positions are read in just fine, but the brightnesses are reset, such that you can see the hidden point sources.

[aagnello]

This is strange though: when I fit the PS1 example with --manual and -z I get image positions very close to the init.ones, the one in these plots is very strange Phil! Also, have you seen what happens with SQLS on 1242? The position initialization itself seems borked there!

[drphilmarshall]

OK, I see the problem. I initialize 0814 manually, and give LensTractor an init file made from these ds9 regions - but the way LT handles the image WCS, it ends up fitting nebula components with positions offset by about (0.06,0.06) degrees (200 arcsec in each direction). You're right, this is a WCS interpretation problem. I merged the above edits (better plots) into my sdss branch, and remember where I am stuck now. Line 1052 of doc/notes.txt: problems building astrometry within the Tractor. I think I need dustin to help with this, I think its a tractor build problem.

[drphilmarshall]

OK, with SDSS images from skyserver, manual initialization works fine. Remaining init problems are real and our own!