This is the pipeline that ingests and reduces new data from the lcogt key project.
Please follow these guidelines for asking for help with installing and running the pipeline
This is likely the quickest way to get the pipeline up and running and requires the least amount of installation. In the following instructions, the database server and data directories will live locally on your computer, so they will persist outside the Docker. The pipeline itself will run inside the Docker container and forward graphics to your local computer.
These instructions only need to be run once, when you set up the pipeline.
Install Docker.
-s cosmic) to run. On Mac, press the Docker icon in the toolbar, then click Preferences, then Resources, and increase Memory to 8 GB.Install docker-compose
(macOS only) Install XQuartz.
(macOS only) Install socat. If you have Homebrew installed, you can just run brew install socat.
Allow X11 connections (may be only necessary on Linux): xhost +local:docker
(Linux Only) Modify your X11 config files to allow TCP connections:
/etc/gdm<3>/custom.conf
[security]
DisallowTCP=false/usr/share/lightdm/lightdm.conf.d/50-xserver-command.conf
[Seat:*]
xserver-command=X -core -listen tcpand the following to /etc/lightdm/lightdm.conf.
This file probably won't exist, you may create it if it is missing.
[Seat:*]
xserver-allow-tcp=true
xserver-command=X -listen tcpTo test your X11 setup run
docker run --rm -it centos:7 /bin/bash
yum install -y xorg-x11-apps
xeyesIf a window appears, your computer is configured correctly. You only have to do this once.
If stage 4 results in an error similar to Error: Can't open display:, then rerun step 4 with this modified command instead.
docker run --rm -it -e DISPLAY=${DISPLAY} -v /tmp/.X11-unix:/tmp/.X11-unix centos:7 /bin/bash
yum install -y xorg-x11-apps
xeyesNow an window should appear.
Clone this repository:
git clone https://github.com/LCOGT/lcogtsnpipeBuild the Docker image:
docker build -t lcogtsnpipe lcogtsnpipeSet your environment variables to point to where you want to store data and catalogs.
You may want to add these lines to your .bashrc (usually Linux) or .bash_profile (usually macOS) file
so that you don't have to set them in every new terminal session.
export LCOSNDIR=/your/data/directory
export LCOSNDBPATH=/your/data/directory/mysqlThese directories do not need to exist. In fact, it is easier if they do not. Docker will automatically create them
with the correct permissions. If you need to use a pre-existing directory or in case docker doesn't set up the permissions correctly, you may have to update the permissions using
chmod -R 777 /path/to/data/.
If you do not set these environment variables, they default to being in data and mysql in repo directory.
(MacOS only) Run XQuartz from the Finder.
(MacOS only) Run this hack in the background to get the X11 forwarding to work:
socat TCP-LISTEN:6000,reuseaddr,fork UNIX-CLIENT:\"$DISPLAY\" &Startup your "pipeline server" (this is really a couple of docker containers instead of a true virtual machine, but this mental picture is close enough).
docker-compose -f lcogtsnpipe/docker-compose.yml up (Linux Only) If you initially had trouble in getting display output from docker in your linux machine (step 6), you will need to slightly modify your `docker-compose.yml` file in the following manner.
```diff
LCOSNDBUSER: "${LCOSNDBUSER:-supernova}"
LCOSNDBPASS: "${LCOSNDBPASS:-supernova}"
LCOSNDIR: "${LCOSNDIR:-/supernova}"
DISPLAY: "${DISPLAY}"
ports:
- "4306:3306"
links:
- sn-db:supernovadb
depends_on:
- sn-db
volumes:
- ${LCOSNDIR:-./data}:${LCOSNDIR:-/supernova}
- /tmp/.X11-unix:/tmp/.X11-unix
```
Now, rerun the above command.This will take over your current terminal. Eventually, the terminal will print that the mysql host is ready to accept connections
In a new terminal (making sure the environment variables from step 9 are still set), log in to the pipeline container:
docker exec -it lcosnpipe /bin/bashIf you're configured correctly, you should be able to open a ds9 window now using ds9 command.
From inside the container, initialize the database: sh /lcogtsnpipe/init-db.sh. You only need to run this command
the first time you setup the db.
From inside the container, run
cd $LCOSNDIR
mkdir -p data/lsc data/fts data/0m4 data/floyds data/extdata standard/cat/apass standard/cat/sloan standard/cat/landolt standard/cat/gaiaThis only needs to be done the first time you populate data in this directory.
You are now ready to use the pipeline!
Follow these instructions each time you want to use the pipeline.
socat TCP-LISTEN:6000,reuseaddr,fork UNIX-CLIENT:\"$DISPLAY\" & export LCOSNDISPLAY=`ifconfig docker0 | grep 'inet ' | cut -d: -f2 | awk '{print $2}'`:0$LCOSNDIR and $LCOSNDBPATH environment variables are set correctly. lcogtsnpipe directory, run
docker-compose up docker exec -it lcosnpipe /bin/bashexit to leave the Docker container.control-c in the terminal you ran docker-compose up.docker-compose down in the lcogtsnpipe directory.Note that you can access the mysql database directly by using -h supernovadb inside the lcosnpipe container.
You can access the database from your host machine, e.g. with sequelpro by setting the host to be 127.0.0.1 and the port to be
Install msql
Set mysql to only throw a warning (rather than an error) when inserting or updating empty values into fields without default values:
mysql -uroot -p
mysql> set session sql_mode = 'NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
mysql> set global sql_mode = 'NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
mysql> exit Create database called Supernova
mysql -u root -pmysql> CREATE DATABASE supernova;Expected output: Query OK, 1 row affected (0.01 sec)
Create user supernova and grant privileges
mysql> GRANT ALL PRIVILEGES ON *.* TO 'supernova'@'localhost' IDENTIFIED BY 'supernova';
mysql> exitExpected output: Query OK, 0 rows affected, 1 warning (0.01 sec)
Create the supernova database and choose a password:
mysql -u root -p *your-password* < *database-including-path*
e.g. mysql -u root -p zenith < /Users/valenti/Desktop/supernova_2020-01-31.sqlCheck that the database structure has been loaded
mysql -u supernova -D supernova -pEnter password: *your-password*
Check that the list of tables matches the list of tables matches the list in Appendix A
mysql> show tables;
mysql> exitPrepare your pipeline environment using Anaconda
conda config --add channels http://ssb.stsci.edu/astrocondaconda create -n lcogtsnpipe python=2.7conda activate lcogtsnpipeconda install iraf-all pyraf-all stsciconda install stsciconda install astroqueryconda install MySQL-pythonInstall the pipeline
git clone https://github.com/svalenti/lcogtsnpipecd lcogtsnpipe
git checkout exportpipecd trunk
python setup.py installSet up directory for data and configuration file
mkdir <your directory name>
e.g. mkdir /Users/valenti/lcoSet LCOSNDIR environment variable either on the command line for a single session or in your .bashrc or .bash_profile file (this assumes you are using a bash shell) that points to where your data will live
export LCOSNDIR=<your directory name>
e.g. export LCOSNDIR='/Users/valenti/lco'Set LCOSNPIPE environment variable either on the command line for a single session or in your .bashrc or .bash_profile file (this assumes you are using a bash shell) that points to your lcogtsnpipe git repository
export LCOSNPIPE=<your directory name>
e.g. export LCOSNPIPE='/Users/valenti/lco/lcogtsnpipe'Create configuration file named configure in LCOSNDIR with the following lines:
hostname 127.0.0.1
database supernova
mysqluser supernova
mysqlpasswd *your-password*
proposal ['']
users ['']
triggerpass ''
extraobject ['']
skipobjects ['auto_focus']
proposalingestion ['']
ptfhost ''
ptfdatabase ''
ptfpasswd '''
ptfuser ''Don't forget to fill in your password on the third line
Install vizquery: https://vizier.u-strasbg.fr/vizier/doc/cdsclient.html
Note: I had to sudo make install on my mac
ingesttar.py -f snexdata_2020-01-30_07-37-02.455163_23_6139.tarNote: if you run this more than once, it may delete your entry. Check that the photlco table has rows. If not, run a third time.
comparecatalogs.pyNote: if you run this more than once, you may need to use the -F option to force the script to look again for a catalog
lscloop.py -e 20200101-20200129 -s cosmiclscloop.py -e 20200101-20200130 -f apass --catalog=$LCOSNDIR/standard/cat/apass/AT2020oi_apass.cat -s psfWhere \
ds9&
lscloop.py -e 20200101-20200129 -s psfmag --showlscloop.py -e 20200101-20200130 -f apass --catalog=$LCOSNDIR/standard/cat/apass/AT2020oi_apass.cat -s zcatWhere \
lscloop.py -e 20200101-20200129 -s mag -F -f apass --type fitNote: this step may give you a warning at the end: Error 1364: Field 'targetid' doesn't have a default value
lscloop.py -e 20200101-20200129 -s getmag --type mag --showmysql> show tables;
+---------------------+
| Tables_in_supernova |
+---------------------+
| aseatide |
| atels |
| classifications |
| contention |
| datarequests |
| dbsyncs |
| eseatide |
| favorites |
| glade |
| glade_2 |
| groups |
| headerdefaults |
| hitslogger |
| iaunames |
| instruments |
| interests |
| lvc_galaxies |
| lvc_triggers |
| notes |
| obslog |
| obsrequests |
| obsrequests_tags |
| papers |
| permissionlog |
| phot_deprecated |
| photlco |
| photlcoraw |
| photpairing |
| programs |
| psns |
| reference_status |
| scheduling_run |
| schedulinglog |
| spec |
| speclcoguider |
| speclcoraw |
| tags |
| targetnames |
| targets |
| telescopes |
| timecharged |
| useractionlog |
| userrequests |
| users |
| voevent_amon |
| voevent_lvc |
| ztf_alerts |
| ztf_targets |
+---------------------+
48 rows in set (0.00 sec) Liberally adopted from: https://iraf-community.github.io/install
xcode-select —installexport TERM='xterm'
export IRAFARCH='macintel'
export OS_VERS='catalina'command+R during restartcsrutil disablesudo mount -uw /sudo mkdir /irafsudo git clone https://github.com/iraf-community/iraf.gitcd iraf
sudo git checkout 567961f
sudo ./installset term to xterm
e.g. export PATH=‘/Users/bostroem/.iraf/bin':$PATHsudo make macintel
sudo make sysgen 2>&1 | tee build.log~/.iraf/setup.sh replacing xgterm -e with xtermIn the directions below,
./configure --prefix=/Users/<your username>
make
make installmake PREFIX=/Users/<your username> installsh autogen.sh
./configure --with-fftw-incdir=/Users/<your username>/include --with-fftw-libdir=/Users/<your username>/lib --with-openblas-incdir=/Users/<your username/include --with-openblas-libdir=/Users/<your username>/lib --with-atlas-incdir=/Users/<your username>/include --with-atlas-libdir=/Users/<your username>/lib --enable-openblas
make -j
sudo make PREFIX=/Users/<your username> installNote: with these direction you do NOT need ATLAS