Closed Hrovatin closed 3 years ago
Is there a reason not to gzip
these files? While they aren't big, you get ~4 fold storage savings and most text inspection tools (e.g. less) can read gzipped files.
We get non gzipped files from core facility and they also seemed to have worked with non gzipped files beforehand in the workflow. But maybe there was another reason for them to not use gzipped files (maybe older reading functions or something).
At some point scanpy switched to non-gzipped files by default as file I/O is faster that way. Reading files quickly was regarded as a more important that storage minimization. I guess it's always a memory vs storage question.
I'm not super into the idea of supporting much beyond exactly what cellranger
outputs for these functions. We expect very specific things from these files, and I think it's difficult to say what's a reasonable amount of modification once we start supporting any.
I'd be open to exposing some of the internally used functions so it's easier to write a custom reading function here, if that's a reasonable alternative to you? I think all that we'd really expose here is a faster scipy.io.mmread
, since the other files are read with pd.read_csv
.
@LuckyMD
At some point scanpy switched to non-gzipped files by default as file I/O is faster that way. Reading files quickly was regarded as a more important that storage minimization.
I believe this only applies to writing h5ad
. But really, lzf
is probably ideal here. It's much faster than gzip
, has similar compression, and is barely slower than no compression. But lzf
is vendored with h5py
not hdf5
(last I checked), so you might not be able to read a file compressed that way from R
or something else.
That is a fair point. Maybe we can just close this issue then?
That sounds good to me. Let me know if you'd like me to point out our mtx reading function. It's basically just calling pandas to read the coo array, since pandas' csv parser is much faster than scipy's
I believe this only applies to writing
h5ad
.
Hmm... i think i must have misunderstood at the time then... i thought it was faster for both.
sc.tools
?sc.pl
?sc.external.*
?In function read_10x_mtx there could be an option to search for non-gzipped files when reading v3 10x. Currently, I have files barcodes.tsv features.tsv matrix.mtx, but the function will not read them as they are not gzipped. ...
But I have 10x files there: