hpages
commented
4 years ago 2 things:
1. Taking a single row (or column) vs taking more than 1 row (or column) are TWO VERY DIFFERENT OPERATIONS
First let's see what happens with an ordinary matrix:
m <- matrix(1:30, nrow=5)
m[1, ] # a vector
# [1] 1 6 11 16 21 26
m[1:2, ] # a 2x6 matrix
# [,1] [,2] [,3] [,4] [,5] [,6]
# [1,] 1 6 11 16 21 26
# [2,] 2 7 12 17 22 27So by default, when taking a single row (or column) from a matrix, one gets a vector instead of a matrix. Note that this can be changed by specifying drop=FALSE:
m[1, , drop=FALSE] # a 1x6 matrix
# [,1] [,2] [,3] [,4] [,5] [,6]
# [1,] 1 6 11 16 21 26DelayedMatrix objects mimic this: when taking a single row (or column) from a DelayedMatrix object, by default an ordinary vector is returned. Otherwise, another DelayedMatrix object is returned. For example, with an HDF5Matrix object (HDF5Matrix and TENxMatrix objects are particular types of DelayedMatrix objects):
M <- as(m, "HDF5Matrix")
M
# <5 x 6> matrix of class HDF5Matrix and type "integer":
# [,1] [,2] [,3] [,4] [,5] [,6]
# [1,] 1 6 11 16 21 26
# [2,] 2 7 12 17 22 27
# [3,] 3 8 13 18 23 28
# [4,] 4 9 14 19 24 29
# [5,] 5 10 15 20 25 30
M[1, ]
# [1] 1 6 11 16 21 26
M[1, , drop=FALSE]
# <1 x 6> matrix of class DelayedMatrix and type "integer":
# [,1] [,2] [,3] [,4] [,5] [,6]
# [1,] 1 6 11 16 21 26
M[1:2, ]
# <2 x 6> matrix of class DelayedMatrix and type "integer":
# [,1] [,2] [,3] [,4] [,5] [,6]
# [1,] 1 6 11 16 21 26
# [2,] 2 7 12 17 22 27Now an even more important difference between M[1, ] and M[1:2, ] is this:
- when an ordinary vector is returned (the
M[1, ]case), all the data that ends up in the vector gets read from the file and loaded in memory; - when a DelayedMatrix object is returned (the
M[1:2, ]situation), the subsetting operation is delayed so nothing actually gets read from the file. Except for displaying purpose e.g. in this case only the data needed to displayM[1:2, ]gets read from the file. But if you were doingMM <- M[1:2, ], nothing would be read from the file (the file wouldn't be accessed at all).
This fundamental difference between delayed vs non-delayed subsetting is documented in ?DelayedArray (Subsetting section).
Conclusion: comparing the speed of M[1, ] vs M[1:2, ] (or M[c(1, 1), ]) is meaningless. The former has to access the file and read the entire row. The latter does not access the file at all: all it does is keep track of the delayed operation by recording it in the returned object.
A more meaningful comparison would be to compare the speed of M[1, ] vs as.matrix(M[1:2, ]). In that case, both operations need to read the extracted rows from the file. If you do this on your object H_logcounts you will see that they take about the same time. Both will be slow (unfortunately).
2. Data in the 10x format is not intended to be accessed by row
The 10x format uses a compressed sparse column representation of the data. With this representation, the full indices component (this is the HDF5 dataset where the row indices are stored) needs to be loaded in memory before a full row can be read. This explains why H_logcounts[100, ] is so slow. This is in fact the worst case scenario! In other words, data in the 10x format is intended to be accessed by column (or groups of columns), not by row. If you need to access the data by row, save it as a conventional (i.e. dense) HDF5 dataset first (with writeHDF5Array()). Pay a particular attention to the chunk geometry that you choose (see ?writeHDF5Array).
Hope this helps, H.
ccshao
LTLA
I met a quite strange problem using the
HDF5Array: taking a single row is extremely slow, comparing to take 2 or more rows/columns. However, Taking a single column is fast. Supprsingly, someting like c(100, 100), but not c(100), are also fast.It is about 3000 times slower than taking a single column, and 6000 times slower than taking at least 2 columns/rows.
Some tests on the TENxMatrix, with lastest HDF5Array from bioconductor(1.14.3)
sessionInfo()