search

openpipelines-bio / openpipeline

https://openpipelines.bio
MIT License
25 stars 11 forks source link

scGPT: use sparse matrix calculations #790

Open dorien-er opened 3 weeks ago

dorien-er commented 3 weeks ago

see https://github.com/openpipelines-bio/openpipeline/pull/754#discussion_r1570273645

This will expand the sparse matrices into dense matrices again, which unfortunatly will cause out of memory issues. Especially when working at atlas scale.

Looking at the implementations for tokenize_batch https://github.com/bowang-lab/scGPT/blob/706526a76d547de4ed711fa028c99be5bdf6ad8a/scgpt/tokenizer/gene_tokenizer.py#L249

and pad_batch https://github.com/bowang-lab/scGPT/blob/706526a76d547de4ed711fa028c99be5bdf6ad8a/scgpt/tokenizer/gene_tokenizer.py#L312

I feel that an implementation that does not expand the matrices into memory is possible to implement. To be discussed.