Lightsheet Pipeline Flask App

This repo contains a python flask application to configure step parameters for pipeline jobs.

Development

Make sure, your python is version 3.5 or higher and your pip is using this correct python:

$ python --version
$ pip --version
$ which python
$ which pip

Create a virtualenv environment, here the environment is called 'env':

$ virtualenv env --no-site-packages

or better

$ virtualenv --no-site-packages -p [path to your python] env

If you are on a Mac, use venv instead of virtualenv, because mathplotlib needs a framework build of Python

$ python -m venv env

Activate the environment:

$ source env/bin/activate

Install the requirements:

$ pip install -r requirements.txt

In Ubuntu, I had trouble doing that, because of issues with permissions when accessing some python libs in my HOME directory. I resolved that by pointing the HOME environment variable to another folder:

$ export HOME=/opt/home
$ sudo chown [your user name]:[your group] /opt/home

With

$ python manage.py runserver [-b <binding ip>] [-p <port>]

you should be able to run the Flask application.

If you're done with coding, you can deactivate the environment with the command

$ deactivate

Deployment

If you haven't done so, install the node.js flightplan library and tool globally:

$ npm install -g flightplan

If there is no command fly available in your cmd, please troubleshoot issues with your $PATH.

In the folder where package.json is located, install the npm packages to use flightplan in your project:

$ npm install

Make changes to target and config in flightplan.js as necessary. Then use flightplan to create a new version entry in package.json and as a git commit with

$ fly version:local [ patch | minor | major ]

Finally, deploy the application with

$ fly deploy:production

If this does not work, make sure ssh-agent is running. If not, run ssh-agent and set the resulting environment variables. Then, run:

eval `ssh-agent -s`
ssh-add -k ~/.ssh/id_rsa

Locate lightsheet service file:

$ locate systemd | grep lightsheet
/etc/systemd/system/lightsheet.service

Restart lightsheet services:

$ sudo systemctl stop lightsheet
$ sudo systemctl start lightsheet

$ sudo systemctl restart nginx

Production

Supervisor config file is located in

/etc/supervisor/conf.d/pipeline.conf

After making changes to the supervisor config file, reload the configuration with

sudo supervisorctl reread; sudo supervisorctl update;

Restart the application with

sudo supervisorctl restart 'pipeline:'

Access the database

We use MongoDB to store the data. To open a Mongo cmd, type

mongo --host [your mongodb host]:[mongodb port]

Some other helpful commands:

> use lightsheet
> db.jobs.help()
> db.jobs.findOne()
> db.jobs.drop()
> db.jobs.remove({ state: { $eq: 'ERROR'} })
> db.step.remove({ "text1": { $exists: true } })
> db.parameter.find( { name: { $gt: 'test' } } )
> db.parameter.find(ObjectId("5b12047ea275276dec9a2eb9"))

Backup and restore

 mongodump --host 10.40.3.155 --port 27036 --db lightsheet --out /opt/tmp/dump/
 mongorestore --drop -d lightsheet /opt/tmp/dump/lightsheet/