For example, imagine you're parsing a .gctx with 720216 cols and 12328 rows, and you want to pull out 20000 columns.
The subset logic is going to try to subset on rows first because 12328 < 20000. But we're not even subsetting on rows here, that is all of them. This results in the entire array getting temporarily allocated into memory.
The better heuristic is to minimize the size of the intermediate array - you want to pick the minimum of: len(ridx) len(col_meta.index) vs len(cidx) len(row_meta.index)
There is some logic in parse_data_df (pasted below) that attempts to pick the best dimension to subset over first, but it isn't quite right.
For example, imagine you're parsing a .gctx with 720216 cols and 12328 rows, and you want to pull out 20000 columns.
The subset logic is going to try to subset on rows first because 12328 < 20000. But we're not even subsetting on rows here, that is all of them. This results in the entire array getting temporarily allocated into memory.
The better heuristic is to minimize the size of the intermediate array - you want to pick the minimum of: len(ridx) len(col_meta.index) vs len(cidx) len(row_meta.index)