Open esteinig opened 9 months ago
Some tricky development notes for the multi-stack subdomain deployment:
internal: true
for each separate stack defined in docker-compose
file stack network - use the usual external: false
only to distinguish from proxy: external: true
configtraefik
router names for the accessible services must be unique across all deployments. Now uses a UUID to designate the router names during deployment.domain=.cerebro.localhost
). In order to prevent cookie sharing on different stack subdomains, we must also set the cookie
domain attribute to the specific stack deployment subdomain. As cookies are required by both application and server, we cannot use a simple dev-app/dev-api + test-app/test-api
subdomain schema for multiple stacks. This means we must use deeper subdomains for multiple stack deployments on the same parent subdomain e.g. app.dev.cerebro.localhost
and api.dev.cerebro.localhost
where the domain=.dev.cerebro.localhost
allows sharing of cookies between app
and api
subdomains. This also requires more configuration in your AAA record provider if hosting the application online e.g. to ensure SSL certificate coverage for deeper subdomains on Cloudflare.
Summary
Semi-automated production runtime for accreditation; overview for aims and progress to complete the production feature on
feat/production
.Aims
Stack deployment
http-local
,http-local-secure
,https-web
,https-web-secure
)https-web
andhttps-web-secure
configsFile input and storage
Production pipeline
User improvements
libssl-dev
system dependency from stack deployment if possibleTesting modules
Documentation
Steps
Full-stack setup of the
Cerebro
production environment for continuous operations. For setting up parallel test or development environments, see details in the documentation.Requirements
Linux
systemMamba
installationCerebro
client installationCerebro
stack setup and operationStack setup and verification
Production setup and verification
Production directory and sub-directories are setup on the system - you can read more about the types of production environments that are currently supported in the documentation.
Here we setup the
RUNTIME
directory where all workflows are executed, and theINPUT
directory where wet-lab staff or laboratory data transfer can deposit the reads and sample sheet to trigger a worfklow execution.Multiple runtime and input folders can be setup for testing, development or validation configurations. Workflow execution and outputs are configured with specific production variables that ensure
Workflow setup and testing
Workflow is setup for production and integration tests are run for production.
Sample sheet for wet-lab
Current sample sheet is focused on dry-lab operation. We need a user-safe sample sheet template that registers the library identifiers, minimal sample meta-data, wet-lab comments, aneuploidy consent and links to the files in the same input directory
Initial template: https://github.com/esteinig/cerebro/blob/feat/production/templates/production/SampleSheet.xlsx
Automated watcher and input checks
Sample sheet and
fastq
files (demultiplexed, de-umified) are watched and validated in the input folder. Depending on the input configuration file the watcher will run production stream and upload to the specified team-database-collection at conclusion of run - different input configuration files (folders) can be watched by different production, test, validation... watchers and outputs deposited into the appropriate database section. Triggers run of the Nextflow pipeline and notifications to Slack.When the pipeline starts, sample identifiers are checked against the team-database-collection to ensure they are unique - the run is registered with the database and samples await confirmation of completion. If sample identifier exists in database the run fails.
Post-workflow sample checks
When the pipeline completes, sample identifiers are collected and validated against registered sample identifiers for this run. Each module (quality control, classification) is checked for completion in each sample. If a sample for some reason did not complete the module, it is marked in the database.
Post-workflow data compilation and upload
After completion, outputs are aggregated into the database models and uploaded into the specified collection via the API
Progress