As a developer I want to be able to automate runs of PIS because currently every run requires setting up all the infrastructure, doing the configuration and running manually, which takes time and is liable to introduce errors.
Background
Current process
Create a GCP VM with sufficient storage
Install dependencies
Setup user and dirs
Fetch GCP credentials
Run the specified PIS container with specified params and config
Check logs and tear down infrastructure
Proposal
Utilise CLI make interface and Terraform to automate the above steps.
Terminology
profile is a config where the following can be configured:
ensembl, chembl, efo versions etc. (small templatizable changes to PIS config.yaml)
PIS docker image/tag
GCS destination
session id automatically generated id for identifying a specific PIS run
Operations/Makefile targets
"credentials" -> copy credentials key from GCS to local.
"new profile" -> create a new profile from a template and set that profile.
"set profile" -> set a particular profile as the active profile
"instantiate config" -> create an instance of PIS config.yaml file from a template using information in the active profile
"launch PIS" (args: local/remote, PIS args) -> create "session id" -> create session id subfolder under "sessions" -> write the PIS config (from "instantiate config") and the running context (a file?)
if local ->
prepare dirs
prepare log
docker run $args
upload logs, config, context to our GCS operations bucket
if remote ->
terraform init
new tf workspace with "session id"
provision resources
vm + storage
startup script + dependencies, ops user env, creds, pis config, running context (a file?)
set ready flag
run PIS
upload the logs, config, context to our GCS operations bucket
shutdown VM
Tasks
[x] write the Makefile targets
[x] write helper scripts
[x] write the terraform infrastructure definition, provisioners and pipeline runner
[x] write the pipeline runner for the local option
As a developer I want to be able to automate runs of PIS because currently every run requires setting up all the infrastructure, doing the configuration and running manually, which takes time and is liable to introduce errors.
Background
Current process
Proposal
Utilise CLI
makeinterface and Terraform to automate the above steps.Terminology
Operations/Makefile targets
Tasks
Acceptance tests
How do we know the task is complete?