Open joseph-siefert opened 1 year ago
Hi @joseph-siefert
Thanks for giving {zellkonverter} a go! Are you able to share the dataset at all? It's a bit hard to say if this is a dataset issue or something to do with your setup. I assume you don't have any issues reading the file in Python?
Unfortunately I can't share the dataset. I removed the layers and was able to get past the above error, however a new error arose:
'X' matrix does not support transposition and has been skipped
I think both errors are due to memory limitations, as I was able to circumvent this by subsampling the matrix. I have plenty of available memory, so it seems related to the available memory in R during the matrix conversion. Is that another more memory-efficient way to make the conversion and avoid R memory limits?
Sorry for the slow response. I thought I had replied to this but obviously not. The way the conversion works there are two copies of the data in memory, one in Python and one in R so for large datasets the memory requirement can be large. One approach to try is using the HDF5 backed mode which should help with this. The other thing is I made some fixes for this specific message recently so it might be worth trying the most recent version (see #96).
Thanks for the great tool. Unfortunately I am getting a segmentation error when converting a large dataset. A smaller subset works without issue. Here is the error from the large dataset: