search

ebi-ait / dcp-ingest-central

Central point of access for the Ingestion Service of the HCA DCP
Apache License 2.0
0 stars 0 forks source link

Send fastq files directly from ingest prod. to SCEA #120

Open ami-day opened 3 years ago

ami-day commented 3 years ago

Once we have curated a project for HCA, we also convert the metadata to MAGE-TAB format for SCEA. As part of this process, we need to either provide paths to the fastq files (from NCBI/ENA) and if they aren't available, to send the fastq files to them directly. There is no relatively quick way to get fastq files for projects when only an SRA object or bam file is immediately available in NCBI/ENA. We don't want to have to do this twice. Ideally we would be able to transfer already uploaded fastq files from a completed project in HCA ingest prod. to SCEA.

ami-day commented 3 years ago

Need to set up a meeting with Jon and potentially others in the SCEA team to discuss sending fastq/bam/sra files.

lauraclarke commented 3 years ago

I think one of the constraints Jon is facing is that they don't have any long term storage so can't have fastq files sitting on disc for an indeterminate amount of time before they analyse them, instead working in a just in time manner

How many datasets are affected by this problem, how many more can we deliver to SCEA if we solve it?