Closed LTLA closed 3 years ago
Latest commit solves the to-do above, but needs a lot of testing.
Briefly, we set up extensible HDF5 datasets and then iterate across row-wise chunks of the sparse DA. In each chunk, we extract the non-zero values, convert them to compressed sparse row format, and append them to the existing HDF5 dataset via h5set_extent
(to make the existing dataset larger) and h5writeDataset
to write the new values in the expanded space. This approach respects the memory constraints of block processing, by ensuring that we do not load all non-zero values into memory; while retaining a single-pass approach for rapid processing, by avoiding the need to know the total number of non-zeroes.
The code here makes a number of assumptions:
values
, indptrs
and index
for the values, column indices and pointers, respectively. Can't remember what H5AD actually calls them. Writing this now, I suspect I mixed up index
and indptrs
.Added a couple of tests and fixed how the loop for rewriting matrices handled paths.
Hit a roadblock in the form of grimbough/rhdf5#79; the AnnData reader doesn't like the fixed-width byte strings that rhdf5 emits.
For the time being, I suggest we just add a clause to the initial check for DAs where we do not skip a DA if it is_sparse()
is TRUE. This implies that it will be realized into memory in SCE2AnnData
- hopefully this is tolerated on the user machine. We can switch back to the current block processing behavior once the rhdf5 issue is resolved.
@lazappi were you going to work on this?
Sorry I've lost track of it a bit. Is it just the is_sparse()
check that we need to add?
Yes, I think line 60 in my PR should only skip the assay if it's a DA and !is_sparse()
. Sparse DAs should be coerced to in-memory dgCMatrix
objects for the time being.
Ok, I've added that check. The code for writing sparse DelayedArrays can't be reached by anything now which made the test coverage drop but that's fine and better to have it there for when the rhdf5 issue is fixed. Anything else to add?
Think that's it, just slap on some # nocov start
and end
s around it for the time being, I guess.
Closes #32.
Some points:
readH5AD
.Known to-dos:
is_sparse()
beingTRUE
. A little bit of work is required to achieve a one-pass writer that does not rely on knowing the total number of non-zero entries.