cdips-pipeline
This is a time-series photometry pipeline that reduces images to light curves. It is a port of a pipeline originally developed for the HATPI project.
This pipeline has been used for the Cluster Difference Imaging Photometric Survey (CDIPS) image-subtraction reductions. The CDIPS project has made and is making light curves for stars in open clusters, moving groups, etc. It has a stand-alone repo.
In theory, if one wished to reproduce the CDIPS reductions, this pipeline would be the place to start. More practically, the code-base should provide readers and users with an entry-point into a particular set of programs that can be used for photometric and time-series analyses.
We're releasing it for general reproducibility and method-sharing and improvement reasons. (Without the expectation of converting you into a user).
The pipeline is a collection of python functions that can be called from a
single "driver script" to go through steps described in the
paper. For the
CDIPS-I reduction, the driver script is
drivers/TESS_reduction.py
.
The idea is that you run this one program (calling the correct options), and
you get light curves from images. This can be done from a shell
script.
State-awareness (i.e., whether previous reduction attempts succeeded or failed)
is minimal, and based on the pre-existence of files and if-else logic. A few
similar driver scripts are also in /drivers/, for example to reduce the ETE6
images. Most of the intermediate files in the pipe-trex reduction (e.g.,
fistar, fiphot, ficonv, etc files) are stored on-disk. A few pieces of metadata
(e.g., image quality diagnostics) are collected to a PostgreSQL database and
used for the reference frame selection steps.
The most important collections of sub-scripts are in aperturephot.py
and
imagesubphot.py
, which wrap the fitsh
tools used to do the aperture
photometry and image subtraction. autoimagesub.py
rolls up many of these
functions to enable pipeline-like functionality.
This installation is only tested on linux boxes (Debian/Ubuntu). Compiling the
fitsh
binaries (see below) might be challenging, but if you do it on MacOS,
please submit a PR describing your experience.
First, clone this repo into a working directory, that we will call $TREX.
git clone https://github.com/waqasbhatti/cdips-pipeline
python3
environmentHave a working version of conda on your path (the 64-bit linux version, presumably). Then
cd $PIPE_TREX
conda env create -f environment_trex_37.yml -n trex_37
source activate trex_37
This deals with most of the details listed below, and is the recommended way to use pipe-trex. (A variety of "new" features since the 2018 development effort depend on using a 3.X environment).
An extra step is that to make pipe-trex
accessible to the virtual
environment, you currently need to add a .pth
file to the approriate
site-packages directory. See "Making pipe-trex accessible within virtual
environment" below.
python2.7
environmentIf you opt instead for a 2.7 environment, make a virtual environment in some
local directory, for instance ~/local
. We will call the environment trex_27:
pip install virtualenv; virtualenv --python=/usr/bin/python2.7 trex_27
Make sure your $PATH and $PYTHON_PATH environmental variables do not have any python-related things that will break your venv.
Active the empty python 2.7 environment:
source trex_27/bin/activate
(trex_27) cd $PIPE_TREX
(trex_27) pip install -r requirements.txt
This latter step will take some time. Then, ensure pyeebls and bleeding-edge astrobase are installed (they are commented out by default because they require a number of dependencies):
(trex_27) pip install pyeebls
(trex_27) cd $ASTROBASE_DIR
(trex_27) python setup.py develop
where I have assumed you may want to make contributions to astrobase as you develop, which you should!
For transit-fitting, you will want batman
, corner
, and emcee
installed:
(trex_27) pip install batman-package
(trex_27) pip install corner
(trex_27) cd $SOME_DIR
(trex_27) git clone https://github.com/dfm/emcee
(trex_27) cd emcee
(trex_27) python setup.py install
In order to perform photometry, we project known-star catalogs onto images in order to know where the stars are. (This is more reliable than source extraction.)
The best catalog in town is Gaia DR2. To access it quickly, gaia2read
is a
useful program. Jason Kim wrote it for his junior thesis, and his source is
available here.
Sam Yee added an important piece of functionality: cutting on different
magnitudes, his fork is available
here.
To install gaia2read
to the command line, do the following:
(trex_27) cd $SOME_DIRECTORY
(trex_27) git clone https://github.com/samuelyeewl/gaia2read
(trex_27) cd gaia2read/gaialib2
(trex_27) make
(trex_27) mv gaia2read $BINARY_DIRECTORY_ON_YOUR_PATH
(trex_27) echo "/nfs/phn15/ar0/H/CAT/GaiaDR2" > ~/.gaia2readrc
where $BINARY_DIRECTORY_ON_YOUR_PATH$
is for example ~/bin/
, or some other
directory from which your path can read binaries.
The assumption of the last line is that you are doing this on the
phn
/phs
/phtess
NFS filesystem, where the "DataPreparation" steps have
already been performed to download and sort the Gaia DR2 catalog.
You must use either anet
or astrometry.net
. The latter is strongly
recommended, since it's free. To install, follow this
page. If you're doing wide-field
work, be sure to get both the 4100 and 4200 indexes.
Note that on debian this requires
sudo apt-get install libcairo2-dev libnetpbm10-dev netpbm \
libpng-dev libjpeg-dev zlib1g-dev \
libbz2-dev swig libcfitsio-dev
This code inherits from the fitsh
project, developed mostly by Andras Pal. Much of fitsh
was inherited by
HATpipe
circa 2010, when it forked. Again, because they are free, we opt for
the public fitsh
versions, rather than the closed HATpipe
fork versions.
The utilities we want working on our path include: ficalib
, fistar
,
fiphot
, grmatch
, grtrans
, ficonv
, fitrans
, and ficombine
,
astrometry.net
, and gaia2read
(deprecated: 2massread
).
Most of these are fitsh
tasks. The fitsh
installation instructions are
here, and they are simple:
cd ~/local
wget http://fitsh.net/download/fitsh/fitsh-0.9.2.tar.gz
tar xvzf fitsh-0.9.2.tar.gz
cd fitsh-0.9.2
./configure
make
make install
Check to make sure this gives you ficalib
, fistar
, fiphot
, grmatch
,
grtrans
, ficonv
, fitrans
, and ficombine
.
For bookkeeping, you will also need a PostgreSQL database.
To install for macs, see here. For linux boxes, see here.
(If you're installing on Mac OS X, it is a good idea to change your kernel state
by modifying your /etc/sysctl.conf
file to include things discussed in the
READMEs from the above links.)
Assuming you are the administrator, you will need to initialize the database
and server, by logging in as the postgres
user and running `initdb -D