search

mskcc / ridgeback

Toil API
1 stars 2 forks source link

Consistency between dev instance setup and production deployment methods #45

Open stevekm opened 4 years ago

stevekm commented 4 years ago

It would help development if we could use a consistent method for both production deployment and setting up development instances.

Currently, the Makefile has been used for setting up dev instances, which includes dependency installation and management of things like Postgres, RabbitMQ, and Celery.

In production and staging, Singularity containers are being used. However their configuration is independent of the configurations used in the Makefile. Also the containers are not currently being used to hold the Ridgeback source code (it gets passed through to reference source code on the host system), and uses a different dependency installation method, and does not appear to include management of Postgres and RabbitMQ.

One possible solution could be to update the Singularity containers to utilize the Makefile for self-installation of Ridgeback and dependencies. A PR has already been filed for using the Makefile to manage the Singularity instances and build containers. This could be extended by more usage of the Makefile within Singularity for configuration and setup, so that dev-only instances that utilize the Makefile will have the same resulting configuration as the production/staging instances running inside containers.

stevekm commented 4 years ago

looks like you can save environment variables to a .sh file and have conda load and unload those on activation:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#macos-and-linux

this works with the 'base' environment in conda as well, could be a useful way to unify the env variables we are currently using in the Singularity containers with the conda setup in the Makefile