CosmoQuest Mini-Delivery

[acviana]

From Cory:

Can you send three master images and the finders to go with them? Three images, at least one of them 1725x1300 plus the sub-images?

Where in this case the "finders" refer to a SQL dump and the images should be "perfect" in terms of the overlay to image agreement.

Requirements:

• [x] Get the FITS and PNG data
• [x] Check the FITS to master PNG agreement
• [x] Check the master 'PNG' to subimage agreement
• [x] Check the subimage numbering
• [x] Check the overlays
• [x] Create and dump a mini db

[acviana]

I think the best place to start here is to look at the Mars data the Dylan analyzed in May. The Neptune data is good for this because there are plenty of labeled moons. The first three datasets in his spreadsheet are good candidates.

Proposal	Target	Rootname	DRZ Mode	CR Mode	Filename	Delta X	Delta Y
11156	Neptune	u9zy0301m	wide	TRUE	u9zy0301m_cr_c0m_wide_single_sci_linear_ephem.png	-5	7
11156	Neptune	u9zy0302m	wide	TRUE	u9zy0302m_cr_c0m_wide_single_sci_linear_ephem.png	-3	8
11156	Neptune	u9zy0303m	wide	TRUE	u9zy0303m_cr_c0m_wide_single_sci_linear_ephem.png	-3	8

[acviana]

I coped the PNG files and the FITS files for comparison:

$ cp /astro/mtpipeline/mtpipeline_outputs/wfpc2/11156_neptune/u9zy0301m_cr_c0m_wide_single_sci.fits .
$ cp /astro/mtpipeline/mtpipeline_outputs/wfpc2/11156_neptune/u9zy0302m_cr_c0m_wide_single_sci.fits .
$ cp /astro/mtpipeline/mtpipeline_outputs/wfpc2/11156_neptune/u9zy0303m_cr_c0m_wide_single_sci.fits .
$ cp /astro/mtpipeline/mtpipeline_outputs/wfpc2/11156_neptune/png/u9zy0301m_cr_c0m_wide_single_sci_linear* png/
$ cp -v /astro/mtpipeline/mtpipeline_outputs/wfpc2/11156_neptune/png/u9zy0302m_cr_c0m_wide_single_sci_linear* png/
$ cp -v /astro/mtpipeline/mtpipeline_outputs/wfpc2/11156_neptune/png/u9zy0303m_cr_c0m_wide_single_sci_linear* png/

[acviana]

So everything looks good except for the _ephem.png file which is pseudo-inverted. The left axis is inverted, which is a weird way to display the data, but technically correct. Since this output is not part of the delivery it doesn't require reprocessing at this point, but this should be corrected in the future.

[acviana]

New DB

I created a new db on my local host:

mysql> CREATE DATABASE mtpipeline_mini_delivery;
Query OK, 1 row affected (0.00 sec)

master_images

CREATE TABLE mtpipeline_mini_delivery.master_images 
SELECT * FROM master_images 
WHERE (fits_file LIKE "u9zy0301m%"
        OR fits_file LIKE "u9zy0302m%"
        OR fits_file LIKE "u9zy0303m%")
    AND (fits_file LIKE "%c0m_center_single_sci.fits"
        OR fits_file LIKE "%c0m_wide_single_sci.fits");

mysql> source /Users/viana/Work/mtpipeline/deliveries/14-06-20-mini/db_dump/third-delivery.sql
Query OK, 12 rows affected (0.02 sec)
Records: 12  Duplicates: 0  Warnings: 0

For each of the 3 rootnames we expect 4 different output modes so we expect 12 records.

master_finders

CREATE TABLE mtpipeline_mini_delivery.master_finders
SELECT * FROM mtpipeline.master_finders
    WHERE mtpipeline.master_finders.master_images_id IN (
        SELECT id FROM mtpipeline_mini_delivery.master_images);

mysql> source /Users/viana/Work/mtpipeline/deliveries/14-06-20-mini/db_dump/third-delivery.sql
Query OK, 168 rows affected (0.01 sec)
Records: 168  Duplicates: 0  Warnings: 0

There are 14 moons we query for Neptune observations and 12 master_images results so we expect 12 x 14 = 168 results.

sub_images

CREATE TABLE mtpipeline_mini_delivery.sub_images
SELECT * FROM mtpipeline.sub_images
    WHERE mtpipeline.sub_images.master_images_id IN (
        SELECT id FROM mtpipeline_mini_delivery.master_images);

mysql> source /Users/viana/Work/mtpipeline/deliveries/14-06-20-mini/db_dump/third-delivery.sql
Query OK, 78 rows affected (0.03 sec)
Records: 78  Duplicates: 0  Warnings: 0

We expect 12 records for each of the 6 wide images giving us 12 * 6 = 72 records. We expect 1 record for each center image giving us 72 + 6 = 78 total records.

finders

There are 2 equivalent queries that can be used to build the table we need, one for each foreign key.

SELECT * FROM mtpipeline.finders
    WHERE mtpipeline.finders.master_finders_id IN (
        SELECT id FROM mtpipeline_mini_delivery.master_finders);

SELECT *FROM mtpipeline.finders
    WHERE mtpipeline.finders.sub_images_id IN (
        SELECT id FROM mtpipeline_mini_delivery.sub_images);

Both return 192 records. We can validated that this is not a coincidence by replacing * with sum(id) and we get 27056544 for both tables. The fact that there are more records in finders than in master_finders can likely be explained by the overlap in the subimages resulting in the same ephemerides being present in more than one subimage from the same master image. A quick partial check of this:

SELECT DISTINCT(master_finders_id) FROM mtpipeline.finders
    WHERE mtpipeline.finders.sub_images_id IN (
        SELECT id FROM mtpipeline_mini_delivery.sub_images);

Finally creating the table:

CREATE TABLE mtpipeline_mini_delivery.finders
SELECT * FROM mtpipeline.finders
    WHERE mtpipeline.finders.sub_images_id IN (
        SELECT id FROM mtpipeline_mini_delivery.sub_images);

mysql> source /Users/viana/Work/mtpipeline/deliveries/14-06-20-mini/db_dump/third-delivery.sql
Query OK, 192 rows affected (0.03 sec)
Records: 192  Duplicates: 0  Warnings: 0

[acviana]

Dumping the database:

$ ./bin/mysqldump -u root -p --databases mtpipeline_mini_delivery > ../../Work/mtpipeline/deliveries/14-06-20-mini/db_dump/2014-06-23-mini-dump.sql

ls and less confirm that the files contents make sense.

[acviana]

Tarballing up the data:

$ tar -cf 2014-06-23-mini-delivery.tar db_dump/ png/

[acviana]

At the last minute it occurred to me that I needed to apply a scaling factor to the offsets that Dylan provided because the PNG pixels are not 1-1 with the FITS images in the _ephem outputs. The 1330 y ranges comes to about 470 pixels or a factor of 2.8. Also, I'm measuring offsets a few pixels worse (in the ephem frame) that what Dylan recorded, about double. So things are about a factor of 5 worse than recorded in the 1-to-1 PNG and FITS space. So on the order of 15 to 40 pixels of offset.