search

jrmullaney / obs_superbit

LSST obs package for superBIT
0 stars 2 forks source link

possibly generate new index files for astrometry_net correction #8

Open mcclearyj opened 5 years ago

mcclearyj commented 5 years ago

Default ones might have wrong scale, or "not be good enough" for what need. James might have instructions on how to do this... We need to investigate this.

jrmullaney commented 5 years ago

Instructions on how to build astrometry.net index files for the LSST stack based on UCAC4 data. (Modified from: https://github.com/dstndstn/astrometry.net/tree/master/doc/UCAC4_guide)

  1. Install astrometry.net and make sure you can run hsplit.

  2. Create a new directory called anetIndex and cd to that directory:

    mkdir anetIndex cd anetIndex

  3. Clone the astrometry.net repository from github into this new directory

    git clone https://github.com/dstndstn/astrometry.net

  4. To convert the raw UCAC files to fits files you need to 'make' the ucac4tofits script. When I did this, I ran into an error in astrometry.net/includ/astrometry/os-features.h, so I had to edit that file to comment out the following lines:

    ifndef DONT_INCLUDE_OS_FEATURES_CONFIG_H

    include "astrometry/os-features-config.h"

    endif

    After doing that, I was able to make ucac4tofits by:

    cd astrometry.net/catalogs make ucac4tofits

  5. In anetIndex/ , make a new directory to contain the raw UCAC4 data files and cd there:

    cd ../../ mkdir UCAC4 cd UCAC4

  6. Grab the download file (for me, the one that cloned was not the latest version, oddly):

    wget https://github.com/dstndstn/astrometry.net/blob/master/doc/UCAC4_guide/get_ucac4.py

  7. Run the download file:

    python get_ucac4.py

  8. Two of the files z???.bz2 files that are downloaded by get_ucac4.py are corrupted, and no matter how much re-downloading I did seemed to fix them (corrupted at source?). As such, you need to make sure that they are ignored by the fits builder:

    mv z431.bz2 z431_bad.bz2 mv z664.bz2 z664_bad.bz2

  9. Copy the fits file generator you created in step 4 to the UCAC4 directory:

    cp ../astrometry.net/catalogs/ucac4tofits

  10. Run ucac4tofits to convert the binary z files to fits files:

    ./ucac4tofits -N 1 z???.bz2

  11. For the LSST, don't require all the columns in the UCAC4 tables, but we do need: id, ra, dec, some mags and a star-not-galaxy flag. So, I wrote my own python script to strip out what the LSST stack needs and add the starnotgal flag. Run it by:

    python trimUCAC4.py

  12. Run this command to split the fits tables into 12 healpix tiles with 2 deg overlap (-m 2):

    hpsplit -o split-%02i.fits -n 1 -m 1 trim_???.fits

  13. For GOTO, I find that index scale 5 works well, but I also include some larger scales as well. It turnes out that the default scales in build_index.py are already ok, so you don't need to edit them for GOTO. If you found you needed some other scales, then edit: scale_range = [5, 7]

You may also need to edit the column that the index-builder sorts on, do that by editing: sortcol = 'v'

  1. After editing (if needed), then run:

    python build-index.py (which takes some time)

  2. Once build_index.py has done its thing, you'll have a lovely suite of UCAC4-based astrometry.net index files suitable for working with the LSST stack. You'll need to copy or move them over to the stack:

    mkdir /path/to/astrometry_net_data/UCAC4 cp index-ucac4* /path/to/astrometry_net_data/UCAC4/

  3. Write an andConfig.py file for the stack to refer to and put it in: /path/to/astrometry_net_data/UCAC4/ That file should contain the following: root.starGalaxyColumn = "starnotgal" filters = ('b','v','m') root.magColumnMap = dict([(f,f) for f in filters]) root.magErrorColumnMap = dict([(f, f + '_err') for f in filters]) root.indexFiles = [ 'index-ucac4-5-00.fits', 'index-ucac4-5-01.fits', 'index-ucac4-5-02.fits', ...all the other index files... (replace this with filenames, obviously!) ''index-ucac4-7-11.fits'']

  4. Create a ups directory in UCAC4:

    mkdir mkdir /path/to/astrometry_net_data/UCAC4/ups and in it create a file called: astrometry_net_data.table which contains a single line: setupRequired(sconsUtils)

  5. With the stack set up, add the new astrometry index files to eups: eups declare astrometry_net_data UCAC -r /path/to/astrometry_net_data/UCAC4/

  6. Then, finally, make sure you set up the new index files when setting up your obs_package by adding the following line to your obs_package's eups table: setupRequired(astrometry_net_data UCAC)

jrmullaney commented 5 years ago

Trim UCAC4: ''' from astropy.io import fits import glob import numpy as np

files = glob.glob('ucac4*.fits')

for file in files: hdu = fits.open(file) dat = hdu[1].data ind = np.where(dat['OBJTYPE']<=1) id = fits.Column(name='id', array=dat['RNM'][ind], format='K') ra = fits.Column(name='RA', array=dat['RA'][ind], format='D') de = fits.Column(name='DEC', array=dat['DEC'][ind], format='D') mm = fits.Column(name='m', array=dat['MAG'][ind], format='F') me = fits.Column(name='m_err', array=dat['SIGMAG'][ind], format='F') vm = fits.Column(name='v', array=dat['VMAG'][ind], format='F') ve = fits.Column(name='v_err', array=dat['VMAG_ERR'][ind], format='F') bm = fits.Column(name='b', array=dat['BMAG'][ind], format='F') be = fits.Column(name='b_err', array=dat['BMAG_ERR'][ind], format='F') sng = fits.Column(name='starnotgal', array=np.squeeze(np.ones_like(ind)), format='K')

t = fits.BinTableHDU.from_columns([id, ra, de, mm, me, vm, ve, bm, be, sng])

newname = file.replace('ucac4', 'trim')
t.writeto(newname, overwrite=True)

'''