Hip iad

[sblunt]

Not fully tested and ready for integration yet, but there were some major structural changes so I'd like to get everyone's ok even at this early stage.

In particular, check out:

• the reformulation of System.compute_model
• the astrometric perturbations code, which now computes stellar astrometric motion as well
• the addition of the Hipparcos module & the treatment of where to actually include that likelihood calculation in the code (in sampler._lnlike)

[rjderosa]

Don't forget some functionality to remove the rejected measurements from the Hipparcos file. Unfortunately the IAD on the DVD doesn't indicate which are rejected, and only has an integer representing the fraction rejected (so 0.4% rejected is rounded down). The IAD area also stored in a binary file in a package on the ESA website (https://www.cosmos.esa.int/web/hipparcos/interactive-data-access). This dataset has the benefit of indicating rejected scans by negating the uncertainty, so you know immediately which scans should be excluded. Unfortunately, these are in a single binary file rather than stored as single text files. I have a notebook to parse the binary file, which I could adapt into a module if the user has the binary file stored on their computer. If you wanted to keep using the DVD IAD (which I wouldn't recommend if they have been superseded), I have a rather basic script that iteratively removes scans until the reconstructed astrometry matches the catalogue values (not perfect, but works most times).

This does raise another point that the two IADs are different, much like how the astrometric catalogue on the CDS is different from the one on the DVD. I think (but haven't double-checked) that the IAD in this binary file is consistent with the CDS catalogue that everyone uses. The IAD on the DVD are outdated, suffering from the same bug that affects the catalogue on the DVD (e.g., Appendix B here https://www.aanda.org/articles/aa/pdf/2014/11/aa24606-14.pdf, and a note in the readme of the CDS tables https://cdsarc.unistra.fr/ftp/I/311/ReadMe). There is a catalogue table in the data file, but of course it is another binary file and I haven't tried to extract it. A spot check of a few HIP stars confirms that the astrometry listed in the GUI of the application from the ESA website matches those stored in the CDS (and is different from those in Main_Cat.d on the DVD). It would be easy to check they all match if the catalogue stored in the binary format was extracted.

Another thing to consider is that the current code will only works when the solution type Sn == 5. Other solution types will need additional treatment in order to reconstruct the abscissa. Solutions 7 and 9 will require adding on the acceleration and jerk signals, for example. It might be simpler at the moment to return an error if Sn != 5.

[sblunt]

Thanks so much all! I'll get to work addressing these great suggestions.

Also wanted to include the suggestions/Eric's suggestions:

Yeah, the six versions I know are 5 parameter solutions (most stars), 7 parameter solutions (proper motion fit by a linear function), 9 parameter solutions (proper motion fit by a quadratic function), orbit solutions (they physically fit an orbit, and luckily gave their parameters), stochastic solutions (They gave up, and inflated the error bars on each astrometric point to get a final reduced chi-square of 1), and component solutions (where the binary is wide enough they could fit each component separately) I’ve struggled a bit trying to get a good plot to show the Hipparcos measurements, which is hard since they’re one-dimensional. I’ve tried to reproduce these plots in the past:

But the changes are usually too subtle to really tell what’s going on. I’ve also tried plotting “astrometric residual” vs. scan number, but that plot doesn’t tell you much more than just quoting a chi-square.

In any case, my favorite test is to convert the Hipparcos IAD to 1D measurements, then try to fit the 5-parameter astrometric model again, and hopefully get back the same median and sigma as in the Hipparcos catalog itself. I guess you can probably test that in orbitize! by trying an orbit with a semi-major axis of 1e-30 or something.

[sblunt]

I'm going to merge this code into develop now. I referenced the discussion in this PR in issue #231 so we don't forget all the great points that were made.

[sblunt]

Ok, finally getting back to this. I'm thinking about @rjderosa's comments first:

1. Cool, sounds like a good thing for me to do is to extract the binary file data and cross-check it with the CDS. I'll work on that.
2. Definitely a good idea for now to just reject Sn != 5. We can worry about that if we find a test case.