maximilianh
commented
2 years ago Hi, thanks for opening this issue and the feedback. I must admit that I've never thought about that. We do use .mtx.gz for Seurat but not Scanpy. You're right it would make a lot of sense to use .mtx.gz here, too. Do you want to have a try ? I won't have the time before January I'm afraid...
On Fri, Nov 19, 2021 at 10:57 PM redst4r @.***> wrote:
Hi,
I've been visualizing quite large datasets (>100k cells) with cellbrowser (from scanpy .h5ad files) and building cellbrowser for these large data takes a significant amount of time. In particular writing the full gene expression matrix to disk in plain tsv (see anndataMatrixToTsv()) takes forever.
Going through the code that's called by 'cbImportScanpyandcbBuild` I actually don't see the need of saving the expression matrix to disk in "dense" format (i.e. plaintext with all the zeros in there).
We could just export it in "matrix-market" format (e.g. via scipy.io.mmwrite) which would save alot of time for sparsely populated matrices (like the raw counts). Even for dense (processed/normalized/filtered) matrices, it probably wouldn't be any worse then the plain dense format.
The code for the "import" step looks pretty easy to change, just writing the matrix via scipy.io.mmwrite, and the cell-ids and genenames in separate files. Not so sure about the "build" part, that looks convoluted.
Let me know what you thing!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/maximilianh/cellBrowser/issues/230, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TLFZCZDXJPEUD2FQITUM3B5ZANCNFSM5INDAYYA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
redst4r
Hi,
I've been visualizing quite large datasets (>100k cells) with cellbrowser (from scanpy .h5ad files) and building cellbrowser for these large data takes a significant amount of time. In particular writing the full gene expression matrix to disk in plain tsv (see
anndataMatrixToTsv()) takes forever.Going through the code that's called by
cbImportScanpyandcbBuildI actually don't see the need of saving the expression matrix to disk in "dense" format (i.e. plaintext with all the zeros in there).We could just export it in "matrix-market" format (e.g. via
scipy.io.mmwrite) which would save alot of time for sparsely populated matrices (like the raw counts). Even for dense (processed/normalized/filtered) matrices, it probably wouldn't be any worse then the plain dense format.The code for the "import" step looks pretty easy to change, just writing the matrix via
scipy.io.mmwrite, and the cell-ids and genenames in separate files. Not so sure about the "build" part, that looks convoluted.Let me know what you thing!