AWS Omics Store support

[SilinPavel]

In order to integrate with AWS Omics Store the next changes should be performed:

• Infrastructure - provisioning of new resources to facilitate AWS Omics
• api-srv - server side changes (new functionality, changes in the existing code base to support AWS Omics)
• UI - Changes related to the UI of the Cloud-Pipeline
• CLI - Implementation of a new functionality in CLI module

Also, this set of features can be divided by stages (S1, S2), First of all we will focus on S1 features to facilitate minimal set of features.

Infrastructure:

• [x] [S1] Add Omics Service role to the terrafrom infrastructure and expand permissions for S3viaSTS role: Expand infrastructure scripts to automatically create new role that will be used to start Omics Import Job
• [x] [S1] Add ECR repository to the infrastructure for further use with Omics Workflow

api-srv

• [x] [S1] Register files through the Omics Import Job
• [x] [S1] List AWS Omics Import Job
• [x] [S1] Add Omics Service role to AWS region
• [x] [S2] Support restore from archive: Add possibility to activate archived files in AWS Omics Store

UI

Common

• [x] [S1] Create Omics store in UI (ref/seq) A new form to register Omics Reference or Sequence store (form accept the next fields: alias, storage type, region id, description)
• [x] [S1] Fix path view for storages - In storage view, currently we see aws_omics_ref:// instead of omics://
• [x] [S1] Disable several UI elements for AWS Omics storage, such as: download, create, rename, share
• [x] [S1] Only folders should have delete button
• [x] [S1] Support AWS Omics storage path as a Input parameter in pipelines
• [x] [S1] Add new fields AWS Omics Service role (omicsServiceRole) and AWS Omics ECR Url (omicsEcrUrl) to AWS region
• [ ] #3491
• [x] #3474
• [x] [S2] Show Omics Import Job list for specific Omics Storage (corresponding server API: /omcisstore/{id}/import/list ) (optional)

SeqStore

• [ ] #3483 fields:

  • File type (BAM, CRAM, UBAM, FASTQ)
  • Reference from ref storage (required for BAM and CRAM),
  • Name (str)
  • Description (str)
  • SampleID (str),
  • SubjectID (str)
  • GeneratedFrom (str)

• [x] [S2] Support restores from archive (choose archive files and click restore) (corresponding server API: /omicsstore/{id}/restore)

CLI & scripts

For CLI the next command should be supported for Omics DataStorage

Common

• [x] [S1] pipe storage ls omics://
• [x] [S1] pipe storage cp s3:// omics://
• [x] [S1] pipe storage cp omics:// local

SeqStore

• [x] [S1] pipe storage cp local omics://

transfer_parameters.py

• [x] [S1] transfer_parameters.py should be able to download AWS Omics files based on input path parameter value
• [x] [S2] localize_inputs_from_files script should be able to work with AWS Omics files properly value