In https://sagebionetworks.jira.com/browse/MC2DPQC-51 we found that files were being staged during the samplesheet_split step. As this occurs in the headjob rather than a process it proved impossible to slow/queue/limit this step and avoid hitting rate-limits on the GTex server.
Thanks to Brad for proposing a solution.
My feeling would be to add a new optional process called something like STAGE_FILES that occurs after SPLIT and before HISTOQC.
Passing through this process should be optional based on a param (say params.stage_in_process) which would be a bool with default false
In https://sagebionetworks.jira.com/browse/MC2DPQC-51 we found that files were being staged during the samplesheet_split step. As this occurs in the headjob rather than a process it proved impossible to slow/queue/limit this step and avoid hitting rate-limits on the GTex server.
Thanks to Brad for proposing a solution.
My feeling would be to add a new optional process called something like STAGE_FILES that occurs after SPLIT and before HISTOQC.
Passing through this process should be optional based on a param (say params.stage_in_process) which would be a bool with default false