Process data from object storage - Githubissues

DKFZ-ODCF / AlignmentAndQCWorkflows

The DKFZ alignment workflow plugin originally developed at the eilslabs

https://github.com/DKFZ-ODCF/AlignmentAndQCWorkflows/wiki

Other

7 stars 5 forks source link

Process data from object storage #41

Open vinjana opened 5 years ago

vinjana commented 5 years ago

Preferentially S3.

Stream directly into pipes e.g. s3cmd. Combine with decompression
How to determine the compression of input data? Add extra compression determination step?
Write all output data to target bucket

Problems:

Roddy cannot yet accept remote files (Are they present? Roddy checks them on FS in the moment but there is no support for remote file checking in S3.

We may first have to implement some basic support in Roddy. See https://github.com/TheRoddyWMS/Roddy/issues/331