HumanCellAtlas / secondary-analysis

Secondary Analysis Service of the Human Cell Atlas Data Coordination Platform
https://pipelines.data.humancellatlas.org/ui/
BSD 3-Clause "New" or "Revised" License
3 stars 2 forks source link

Optimus: fix aligner reference inefficiency #644

Open kbergin opened 5 years ago

kbergin commented 5 years ago

Currently the alignment step in optimus spends about 15 min to copy and untar the reference

The reference should be passed as individual files

The alignment instances are multicore machines and this leads to significant waste

Also it is the limiting factor for cutting down the testing time with any meaninful dataset

┆Issue is synchronized with this Jira User Story