demitri
commented
9 years ago Please see my comments in the other issue, but I think we should discuss what is meant by indexing the data. There is a limited amount of how an image can be indexed, so I want to make sure we're on the same page with that. Second (per the other issue), Trillian will only have one pixel representation of the sky, but it will be adaptive such that some (Healpix) pixels are very large, but others are much smaller in places where there is more data available in a particular part of the sky. We don't want multiple Healpix representations.
Healpix come in a variety of resolutions, from 12 pixels on the lowest end to 3,458,764,513,820,540,928 on the highest end. For some FITS files, the resolution is given in Healpix, but for most, it is in regular pixel format.
One naive implementation would index every single Healpix in a given file in every single resolution. However, this would quickly become impossible to scale given that the FITS files alone comprise several TB of data.
I have come up with the following proposal to deal with this:
In any given image file, the data will be indexed by the lowest possible resolution first, and then recursively broken down into smaller resolutions until reaching an agreed upon minimum resolution. This means that for full-sky surveys, an image is indexed 12 times, once for each Healpix.
An example illustration of a more complex case (left incomplete for the sake of time):
Then, on the query side, an object with given coordinates will be queried against all reasonable resolutions of Healpix.
@demitri @maxogden