The MTX to SCE conversion is memory intensive. On a recent run, I encountered 122 GB memory usage (requiring two fail-retry cycles) on this step (1.4T raw data). We should look into DelayedArray options. It may be feasible to convert each sample in series, writing out to a common HDF5 (or similar) file.
The MTX to SCE conversion is memory intensive. On a recent run, I encountered 122 GB memory usage (requiring two fail-retry cycles) on this step (1.4T raw data). We should look into
DelayedArrayoptions. It may be feasible to convert each sample in series, writing out to a common HDF5 (or similar) file.