STScI-Citizen-Science / MTPipeline

Pipeline to produce CR rejected, astrodrizzled, png's of HST WFPC2 solar system data.
6 stars 1 forks source link

Script to Categorize Pipeline Files #133

Closed walyssonBarbosa closed 10 years ago

walyssonBarbosa commented 10 years ago

Create a script that goes over all fits and flt files in the filesystem (ACS, WFC3 and WFPC2). Create a dictionary and for each file check its TARGNAME, categorize the file as pm or notpm (Planet or Moon, Not Planet nor Moon).

walyssonBarbosa commented 10 years ago

Used planets_and_moons.txt and 'jup-', 'gany-', 'sat-', 'copernicus', 'gan-'] as parameters for planets or moons observations. Found 494 distinct observations in the filesystem using the following script:

#! /usr/bin/env python

from mtpipeline.get_settings import SETTINGS
import glob, os
from astropy.io import fits

locations = [('acs_input_path', 'flt.fits'), ('wfpc2_output_path', 'c0m.fits'), ('wfc3_input_path', 'flt.fits')]

files_list = []

for l, e in locations:
    all_files_list = glob.glob(os.path.join(SETTINGS[l], '*_*/*{}'.format(e)))
    files_list += [filename for filename
                     in all_files_list
                     if len(filename.split('/')[-1]) == 18
                     and filename.split('/')[-1].split('_')[-1] == '{}'.format(e)]
files_dict = {}
for f in files_list:
    hdu = fits.open(f)
    head = hdu[0].header['TARGNAME']
    if head not in files_dict:
        files_dict[head] = {'list': [f], 'category': ''}
    else:
        if f not in files_dict[head]['list']:
            files_dict[head]['list'].append(f)
planets_moons_list = ['jup-', 'gany-', 'sat-', 'copernicus', 'gan-']
pm_file = open('mtpipeline/ephem/planets_and_moons.txt', 'r')
for pm in pm_file:
    obj = pm.split(' ')
    if len(obj) > 3:
        planets_moons_list.append(obj[1])
for pla in planets_moons_list:
    if pla == 'io':
        pla = 'io-'
found = {}
for pm in planets_moons_list:
    for obj in files_dict:
        if pm.upper() in obj:
            found[obj] = 0
print len(found)

I looked over the list of files that are neither planet nor moon observation, they all seem to be asteroids, comets and other things. Next step is to create the list of files that are in this observations.

walyssonBarbosa commented 10 years ago

To modify the category of the observation:

for obj in files_dict:
    if obj in found:
        files_dict[obj]['category'] = 'pm'  # planet/moon observation
    else:
        files_dict[obj]['category'] = 'notpm' # not planet/moon observation

These are two of the results from the dictionary created:

'IO': {'category': 'pm',
  'list': ['/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0306t_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0403t_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0102t_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0206t_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0305t_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0205t_c0m.fits']},
'GYPTIS': {'category': 'notpm',
  'list': ['/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h4603r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h9601r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h4601r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h4602r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h4604r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h4605r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h4606r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h9602r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h9603r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h9604r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h9605r_c0m.fits',
   '/astro/mtpipeline/mtpipeline_outputs/wfpc2/08583_asteroids/u62h9606r_c0m.fits']}
acviana commented 10 years ago

Good start. A couple of questions:

  1. How did you decide on 'jup-', 'gany-', 'sat-', 'copernicus', 'gan-'?
  2. How would this be implemented in the pipeline?
walyssonBarbosa commented 10 years ago

I looked over the target names and saw some with those names, so I decided to include them in the list.

I need to take a look at build_master_finders_table.py again to see if I can understand how it works.

walyssonBarbosa commented 10 years ago

Am I right by saying that we should implement this here?

def get_header_info(filename):
    '''
    Gets the header info from the FITS file. Checks to ensure that the 
    target name, after string parsing, matches a known planet name.
    '''
    assert os.path.splitext(filename)[1] == '.fits', \
        'Expected .fits got ' + filename
    header = pyfits.getheader(filename)
    output = {}
    output['targname'] = header['targname'].lower().split('-')[0]
    output['ra_targ']  = header['ra_targ']
    output['dec_targ'] = header['dec_targ']
    output['CRVAL1']   = header['CRVAL1']
    output['CRVAL2']   = header['CRVAL2']
    output['CRPIX1']   = header['CRPIX1']
    output['CRPIX2']   = header['CRPIX2']
    planet_list = ['mars', 'jupiter', 'saturn', 'uranus', 'neptune', 'pluto']
    assert output['targname'] in planet_list, \
        'Header TARGNAME not in planet_list'
    return output

I am still trying to figure out how she should change it.

walyssonBarbosa commented 10 years ago

This is my implementation:

def get_planets_moons(pl_mo_file):
    planets_moons_list = ['jup-', 'gany-', 'sat-', 'copernicus', 'gan-']
    pm_file = open(pl_mo_file, 'r')
    for pm in pm_file:
        obj = pm.split(' ')
        if len(obj) > 3:
            if obj[1] == 'IO':
                planets_moons_list.append('io-')
            else:
                planets_moons_list.append(obj[1])
    return planets_moons_list

def get_header_info(filename):
    '''
    Gets the header info from the FITS file. Checks to ensure that the 
    target name, after string parsing, matches a known planet name.
    '''
    assert os.path.splitext(filename)[1] == '.fits', \
        'Expected .fits got ' + filename
    header = pyfits.getheader(filename)
    output = {}
    output['targname'] = header['targname'].lower().split('-')[0]
    output['ra_targ']  = header['ra_targ']
    output['dec_targ'] = header['dec_targ']
    output['CRVAL1']   = header['CRVAL1']
    output['CRVAL2']   = header['CRVAL2']
    output['CRPIX1']   = header['CRPIX1']
    output['CRPIX2']   = header['CRPIX2']
    planet_list = get_planets_moons('mtpipeline/ephem/planets_and_moons.txt')
    for pm in planet_list:
        assert pm in output['targname'], \
        'Header TARGNAME not in planet_list'
    return output

I tried to run it in an iPython notebook but it raises this error: KeyError: "Keyword 'CRVAL1' not found."

The assert statement is not correct though.

acviana commented 10 years ago

Ok so we're on the right track but I have some comments:

  1. You pass the name of the planet and moons list even though it's the same every time. If this isn't going to change we don't need to pass it as a variable.
  2. You open the file with every call to get_header_info. But it's the same file every time. It would be faster to just open it once and keep it in memory.
  3. This is getting to be pretty extensive modification so instead of pasting this into a ticket let's move this to a branch and discuss the changes in there. Please point back to this ticket number in your commits.
walyssonBarbosa commented 10 years ago

It works. I was not testing the script with the right fits files before.

> print get_header_info('/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0306t_c0m_wide_single_sci.fits')
{'CRPIX1': 862.5,
 'CRPIX2': 650.0,
 'CRVAL1': 283.4439879371963,
 'CRVAL2': -22.93860252428935,
 'dec_targ': -22.93754595322,
 'ra_targ': 283.4514282665,
 'targname': 'io'}
>  print get_header_info('/astro/mtpipeline/mtpipeline_outputs/wfpc2/05489_vesta/u2j70506t_c0m_wide_single_sci.fits')
AssertionError: Header TARGNAME not in planet_list
walyssonBarbosa commented 10 years ago

Tried to run jpl2db.py over u3aw0306t_c0m.fits (Jupiter's Io observation):

$ python mtpipeline/ephem/jpl2db.py -filelist '/astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0306t_c0m.fits'

This was displayed:

Processing 1 files.
2014-07-29 15:09: Starting processing

And in the log file there's this error:

CRITICAL: <class 'sqlalchemy.orm.exc.NoResultFound'> No row was found for one() 423
acviana commented 10 years ago

Any idea what is causing that error or how you could figure it out?

walyssonBarbosa commented 10 years ago

This error happens in this part of the script:

master_images_query = session.query(MasterImages).filter(\
        MasterImages.fits_file == os.path.basename(filename)).one()

Maybe the query is returning nothing:

mysql> select * from master_images where fits_file = 'u3aw0306t_c0m.fits';
Empty set (0.00 sec)
walyssonBarbosa commented 10 years ago

I tried to use /astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0306t_c0m_wide_single_sci.fits but it says that the list has no files. And this is the part of the code:

# Create the filelist.
    if args.filelist != None:
        filelist = glob.glob(args.filelist)
        filelist = [x for x in filelist if len(os.path.basename(x)) == 18]
        assert isinstance(filelist, list), \
            'Expected list for filelist, got ' + str(type(filelist))
        assert filelist != [], 'No files found.'

Maybe the script is not selecting the right x in filelist.

walyssonBarbosa commented 10 years ago

So, I changed this line:

filelist = [x for x in filelist if len(os.path.basename(x)) == 18]

to this:

filelist = [x for x in filelist if ('c0m_center_single_sci.fits' in os.path.basename(x) or 'c0m_wide_single_sci.fits' in os.path.basename(x))]

And it seemed to work:

$ tail -1 '/astro/mtpipeline/logs/jpl2db/jpl2db_2014-07-29-16-11.log'
07/29/2014 16:11:36 PM MainProcess INFO: Completed for  : /astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/u3aw0306t_c0m_wide_single_sci.fits

The database had 212022 records in master_finders before and now it has 212098 records:

mysql> select count(*) from master_finders;
+----------+
| count(*) |
+----------+
|   212089 |
+----------+
1 row in set (0.04 sec)
acviana commented 10 years ago

Great, does that difference match with the number of moons you expect for a Jupiter image?

walyssonBarbosa commented 10 years ago

Yes, it does. There are 67 Jupiter moons in planets_and_moons.txt and this is the how much the records in the table have increased.

acviana commented 10 years ago

Perfect. Do you know what the Travis build is failing though?

walyssonBarbosa commented 10 years ago

The last one passed. I don't know why the previous one failed.

acviana commented 10 years ago

We've been having some issue with builds failing during the install process. Since the Travis link is grey instead of red (test failure) that's likely what happened here.

Can you please look over the code and see if you see any bottle necks that could be optimized?

walyssonBarbosa commented 10 years ago

Will do it.

I am currently running jpl2db.py on /astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/.

ktfhale commented 10 years ago

Build 111 indeed fail because one of those pesky Connection reset by peer packaged download errors. They were gone for a while, but they seem to have cropped up again today.

walyssonBarbosa commented 10 years ago

Completed running on /astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/. 15133 records were inserted in master_finders table.

acviana commented 10 years ago

Great, can you analyze the log file and give me some performance numbers?

walyssonBarbosa commented 10 years ago

Ok.

While I do that I will leave the script running over the remaining files.

acviana commented 10 years ago

Great, while that's running take some of your new images and run mtpipeline/ephem/ephem_viewer.py on them with -label True to check that your images are being correctly marked.

walyssonBarbosa commented 10 years ago

So, in average it took 20.22 seconds to run each of the 231 files from /astro/mtpipeline/mtpipeline_outputs/wfpc2/06752_jupiter_moons/. The standard deviation was 0.96 seconds.

acviana commented 10 years ago

That's 67 requests per file correct? So really it's 67 requests every 20.22 seconds?

walyssonBarbosa commented 10 years ago

It varied from 64 to 67 requests per file {61 files: 64 requests, 53 files: 65 requests,61 files: 66 requests, 61 files: 67 requests}.

walyssonBarbosa commented 10 years ago

I tried to run mtpipeline/ephem/ephem_viewer.py on some images and nothing happened. I used the following commands:

$ python mtpipeline/ephem/ephem_viewer.py -file '/astro/mtpipeline/mtpipeline_outputs/wfpc2/05836_saturn/png/u2tf0202t_cr_c0m_center_single_sci.png' -label True
$ python mtpipeline/ephem/ephem_viewer.py -file '/astro/mtpipeline/mtpipeline_outputs/wfpc2/05329_neptune/png/u2j30302t_cr_c0m_center_single_sci.png' -label True
$ python mtpipeline/ephem/ephem_viewer.py -file '/astro/mtpipeline/mtpipeline_outputs/wfpc2/05329_neptune/png/*single_sci.png' -label True
acviana commented 10 years ago

OK. So then you need to explain:

In the future I expect commentary and analysis with results.

walyssonBarbosa commented 10 years ago

ephem_viewer.py wasn't doing anything because I was using the a wrong file name. Instead of using a *_single_sci_linear.png file I used a *_single_sci.png.

Now I'm getting this error with moon.ephem_x and moon.ephem_y:

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

In this part of the code:

if args.label == "True":
                ax1.text(moon.ephem_x + 20, moon.ephem_y + 20, 
                         moon.object_name.title(), color="blue")

I still don't know why some files had less than 67 requests.

walyssonBarbosa commented 10 years ago

It seems that some images don't have ephemerides for some of the moons. Like this neptune observation:

$ python mtpipeline/ephem/ephem_viewer.py -file '/astro/mtpipeline/mtpipeline_outputs/wfpc2/05329_neptune/png/u2j30302t_cr_c0m_center_single_sci_linear.png' -label True
deimos 4 331
phobos 199 297
mars 259 243
triton None None
nereid None None
psamathe None None
sao None None
thalassa None None
laomedeia None None
naiad None None
halimede None None
neptune None None
despina None None
neso None None
proteus None None
galatea None None
larissa None None
walyssonBarbosa commented 10 years ago

Taking into consideration that the previous neptune observation had mars and its moons deimos and phobos ephemerides, the jupiter files that had 64 to 67 requests may have only requested ephemerides for the objects that were visible in each jupiter observation. I don't know if that makes sense and if it could be an explanation for that issue.

walyssonBarbosa commented 10 years ago

I didn't know that the png files were being saved in their respective folder. I was waiting them to open.

Here's one of the images created with ephem_viewer.py: u2j30201t_cr_c0m_center_single_sci_linear_ephem_lb I'm not sure how to interpret the labels because I can't see any object inside them.

walyssonBarbosa commented 10 years ago

But it seems wrong because Mars shouldn't be there. And deimos is too far away from its planet. Maybe the ephemerides is wrong.

acviana commented 10 years ago

So. The most obvious thing I see wrong is that Mars, Phobos, and Deimos data are being requested for a Neptune observation. That's definitely a bug, let's sort that out first and see what falls into place.

Also there are some instances were ephem_x and ephem_y are Null. That should never happen because there are computationally derived positions. Meaning if the system can calculate their position at all it can always do it. It's all or nothing. I think the error is on our end not on JPL's.

walyssonBarbosa commented 10 years ago

We are getting some errors while running jpl2db.py:

$ grep -c "CRITICAL: <type 'exceptions.TypeError'> 'NoneType' object is not iterable 419" /astro/mtpipeline/logs/jpl2db/jpl2db_2014-07-30-08-46.log
2036
walyssonBarbosa commented 10 years ago

When I looked over the log file for /astro/mtpipeline/mtpipeline_outputs/wfpc2/11156_neptune/u9zy5503m_cr_c0m_wide_single_sci.fits it didn't get ephem data from mars and its moons. But when I used the image referent to this fits file it created a png file with labels for mars and its moons. The image only clearly shows neptune, but it didn't have ephemerides for it.

acviana commented 10 years ago

Wait, which fits file did you use instead of the _single_sci.fits file that worked? Also, can you figure out when None is being passed? pdb should help with that.

walyssonBarbosa commented 10 years ago

It was not a fits file, it was a png file. I was using *_single_sci.png instead of *_single_sci_linear.png.

None is being passed in this part of the code:

class EphemPlot(object):
...
    def plot_image(self):
          ...
          if args.label == "True":
                    ax1.text(moon.ephem_x + 20, moon.ephem_y + 20,
                         moon.object_name.title(), color="blue")
          ...

I will use pdb to take a look at it.

walyssonBarbosa commented 10 years ago

I went to the database to look the neptune observation that has mars in it. It only has ephemerides for mars, deimos and phobos. Maybe the problem is in jpl2db.py and not in ephem_viewer.py. I will take look at jpl2db.py to see if I can find anything.

walyssonBarbosa commented 10 years ago

Is there the possibility that as we filled master_images with new data, the id of each observation may be different from the ids in the old records in master_finders that we reused?

That may be the reason why there's mars in neptune. Because I ran jpl2db in a neptune observation and it didn't request for mars, deimos nor phobos.

walyssonBarbosa commented 10 years ago

I deleted all files in master_finders from the database we are using for test and ran jpl2db in 11156_neptune/u9zy5503m_cr_c0m_wide_single_sci.fits. It inserted 14 values in master_finders, only related to neptune.

walyssonBarbosa commented 10 years ago

This are the values inserted:

mysql> select * from master_finders;
+--------+-------------+------------------+---------+---------+---------+-------------+-------------+-----------+----------+
| id     | object_name | master_images_id | ephem_x | ephem_y | version | jpl_ra      | jpl_dec     | magnitude | diameter |
+--------+-------------+------------------+---------+---------+---------+-------------+-------------+-----------+----------+
| 378485 | triton      |             3824 |    NULL |    NULL |       3 | 21:42:39.10 | -14:06:32.2 |     13.47 | 0.128302 |
| 378486 | nereid      |             3824 |    NULL |    NULL |       3 | 21:43:04.69 | -14:03:35.6 |      18.7 | 0.016129 |
| 378487 | psamathe    |             3824 |    NULL |    NULL |       3 | 21:41:19.56 | -13:53:45.5 |      -999 |     -999 |
| 378488 | sao         |             3824 |    NULL |    NULL |       3 | 21:41:53.49 | -13:58:13.4 |      -999 |     -999 |
| 378489 | thalassa    |             3824 |    NULL |    NULL |       3 | 21:42:39.58 | -14:06:18.6 |     23.81 | 0.003794 |
| 378490 | laomedeia   |             3824 |    NULL |    NULL |       3 | 21:43:43.90 | -14:15:01.0 |      -999 |     -999 |
| 378491 | naiad       |             3824 |    NULL |    NULL |       3 | 21:42:39.79 | -14:06:17.5 |     24.71 | 0.002751 |
| 378492 | halimede    |             3824 |    NULL |    NULL |       3 | 21:42:31.05 | -14:20:40.7 |      -999 |     -999 |
| 378493 | neptune     |             3824 |    NULL |    NULL |       3 | 21:42:39.71 | -14:06:18.8 |      7.84 |  2.34907 |
| 378494 | despina     |             3824 |    NULL |    NULL |       3 | 21:42:39.82 | -14:06:17.4 |     22.61 |  0.00702 |
| 378495 | neso        |             3824 |    NULL |    NULL |       3 | 21:44:30.35 | -14:03:44.1 |      -999 |     -999 |
| 378496 | proteus     |             3824 |    NULL |    NULL |       3 | 21:42:39.95 | -14:06:19.8 |     20.31 | 0.020679 |
| 378497 | galatea     |             3824 |    NULL |    NULL |       3 | 21:42:39.65 | -14:06:17.6 |     22.31 | 0.007494 |
| 378498 | larissa     |             3824 |    NULL |    NULL |       3 | 21:42:39.61 | -14:06:20.8 |     22.01 | 0.009106 |
+--------+-------------+------------------+---------+---------+---------+-------------+-------------+-----------+----------+

JPL didn't find any ephemerides (ephem_x, ephem_y) for the objects.

acviana commented 10 years ago

OK, will maybe to start we need add some type of assert at the beginning of the script to ensure that we are taking the right kind of input. I think that's what's going on here is that the script can take the wrong input, and then by the time it crashes it's hard to work backward to figure out the source.

acviana commented 10 years ago

Yes, so looking back a few of your comments. master_finders.id is not the foreign key for master_images. master_finders.master_images.id is the foreign key, but I'm not sure if that was your question.

acviana commented 10 years ago

Yes, so does JPL actually not return anything or does it just get messed up once we get it back?

walyssonBarbosa commented 10 years ago

I just said id as a general thing, but I should've said master_finders.master_images_id and master_images.id instead.

walyssonBarbosa commented 10 years ago

JPL returns this values:

{'jpl_dec': '-14:06:32.2', 'jpl_ang_diam': '0.128302', 'jpl_ra_apparent': '21:43:09.18', 'jpl_ra_delta': '-4.17791', 'jpl_APmag': '13.47', 'jpl_ra': '21:42:39.10', 'jpl_dec_delta': '-1.09981', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:04:03.7'}
{'jpl_dec': '-14:03:35.6', 'jpl_ang_diam': '0.016129', 'jpl_ra_apparent': '21:43:34.76', 'jpl_ra_delta': '-3.55983', 'jpl_APmag': '18.70', 'jpl_ra': '21:43:04.69', 'jpl_dec_delta': '-1.27694', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:01:06.9'}
{'jpl_dec': '-13:53:45.5', 'jpl_ang_diam': '-999', 'jpl_ra_apparent': '21:41:49.64', 'jpl_ra_delta': '-3.62608', 'jpl_APmag': '-999', 'jpl_ra': '21:41:19.56', 'jpl_dec_delta': '-1.25123', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-13:51:17.7'}
{'jpl_dec': '-13:58:13.4', 'jpl_ang_diam': '-999', 'jpl_ra_apparent': '21:42:23.56', 'jpl_ra_delta': '-3.70973', 'jpl_APmag': '-999', 'jpl_ra': '21:41:53.49', 'jpl_dec_delta': '-1.35283', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-13:55:45.3'}
{'jpl_dec': '-14:06:18.6', 'jpl_ang_diam': '0.003794', 'jpl_ra_apparent': '21:43:09.67', 'jpl_ra_delta': '-4.78173', 'jpl_APmag': '23.81', 'jpl_ra': '21:42:39.58', 'jpl_dec_delta': '-2.42427', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:03:50.1'}
{'jpl_dec': '-14:15:01.0', 'jpl_ang_diam': '-999', 'jpl_ra_apparent': '21:44:13.99', 'jpl_ra_delta': '-3.53661', 'jpl_APmag': '-999', 'jpl_ra': '21:43:43.90', 'jpl_dec_delta': '-1.26787', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:12:32.0'}
{'jpl_dec': '-14:06:17.5', 'jpl_ang_diam': '0.002751', 'jpl_ra_apparent': '21:43:09.88', 'jpl_ra_delta': '-5.24794', 'jpl_APmag': '24.71', 'jpl_ra': '21:42:39.79', 'jpl_dec_delta': '-1.05820', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:03:49.0'}
{'jpl_dec': '-14:20:40.7', 'jpl_ang_diam': '-999', 'jpl_ra_apparent': '21:43:01.17', 'jpl_ra_delta': '-3.70075', 'jpl_APmag': '-999', 'jpl_ra': '21:42:31.05', 'jpl_dec_delta': '-1.32007', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:18:12.3'}
{'jpl_dec': '-14:06:18.8', 'jpl_ang_diam': '2.349070', 'jpl_ra_apparent': '21:43:09.79', 'jpl_ra_delta': '-3.61498', 'jpl_APmag': '7.84', 'jpl_ra': '21:42:39.71', 'jpl_dec_delta': '-1.29898', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:03:50.3'}
{'jpl_dec': '-14:06:17.4', 'jpl_ang_diam': '0.007020', 'jpl_ra_apparent': '21:43:09.91', 'jpl_ra_delta': '-4.89597', 'jpl_APmag': '22.61', 'jpl_ra': '21:42:39.82', 'jpl_dec_delta': '-0.97855', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:03:48.9'}
{'jpl_dec': '-14:03:44.1', 'jpl_ang_diam': '-999', 'jpl_ra_apparent': '21:45:00.41', 'jpl_ra_delta': '-3.65105', 'jpl_APmag': '-999', 'jpl_ra': '21:44:30.35', 'jpl_dec_delta': '-1.26525', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:01:14.7'}
{'jpl_dec': '-14:06:19.8', 'jpl_ang_diam': '0.020679', 'jpl_ra_apparent': '21:43:10.03', 'jpl_ra_delta': '-2.70285', 'jpl_APmag': '20.31', 'jpl_ra': '21:42:39.95', 'jpl_dec_delta': '-0.58528', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:03:51.4'}
{'jpl_dec': '-14:06:17.6', 'jpl_ang_diam': '0.007494', 'jpl_ra_apparent': '21:43:09.73', 'jpl_ra_delta': '-5.23639', 'jpl_APmag': '22.31', 'jpl_ra': '21:42:39.65', 'jpl_dec_delta': '-2.04540', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:03:49.2'}
{'jpl_dec': '-14:06:20.8', 'jpl_ang_diam': '0.009106', 'jpl_ra_apparent': '21:43:09.69', 'jpl_ra_delta': '-2.20157', 'jpl_APmag': '22.01', 'jpl_ra': '21:42:39.61', 'jpl_dec_delta': '-1.22722', 'date': '2008-Jul-27 23:30', 'jpl_dec_apparent': '-14:03:52.3'}

Now I think that there's some problem when trying to use or convert the values to insert them in master_finders.

acviana commented 10 years ago

You should know that I'm in a meeting up the hall and we are literally talking about how important this work will be. :thumbsup:

walyssonBarbosa commented 10 years ago

I'm aware of that.

Is jpl2db.py responsible for inserting ephem_x and ephem_y in master_finders? Because I think that it's not doing that:

def insert_record(moon_dict,  master_images_id):
    '''
    Make a new record to be in the master_finders table.
    '''
    record = MasterFinders()
    record.object_name = moon_dict['object']
    record.jpl_ra = moon_dict['jpl_ra']
    record.jpl_dec = moon_dict['jpl_dec']
    try:
        record.diameter = float(moon_dict['jpl_ang_diam'])
    except Exception as err:
        logging.critical('{0} {1} {2}'.format(
            type(err), err.message, sys.exc_traceback.tb_lineno))
    try:
        record.magnitude = float(moon_dict['jpl_APmag'])
    except Exception as err:
        logging.critical('{0} {1} {2}'.format(
            type(err), err.message, sys.exc_traceback.tb_lineno))
    record.master_images_id = master_images_id
    record.version = __version__
    session.add(record)
    session.commit()