Add scGPT Model

[jprivera44]

Model description

scGPT is a single celled foundation model, based off the GPT architecture. The model is shown to have captured meaningful biological insights into cells and genes. The authors state the model can be fine tuned to downstream tasks included, cell-type annotation, genetic perturbation etc.

I'd like to add scGPT to HuggingFace Transformers.

Open source status

• [X] The model implementation is available
• [X] The model weights are available

Provide useful links for the implementation

The paper scGPT: Towards Building a Foundation Model for Single-Cell 2 Multi-omics Using Generative AI by Haotian Cui, Chloe Wang , Hassaan Maan, Bo Wang

Github link: scGPT by subercui

Model Checkpoint: Google Drive

• From this checkpoint I can generate the model weights

[amyeroberts]

Hi @jprivera44, thanks for opening this issue!

The fastest and easiest way to add a model to be used in the transformers library, is to add the model code and its weights on the code. Here's a how-to guide: https://huggingface.co/docs/transformers/custom_models

cc @Rocketknight1

[jprivera44]

Hello @amyeroberts , thank you for the comment! After reviewing the content, I plan to stick with the process outlined in this link, which goes over how to add a model from scratch. Since the model is SOTA the involved process will make it easier for our community to leverage the model, and make the overall codebase more interpretable. If you have any questions please let me know!

To give you an update, I just ran the code to load the model weights and I'm now focusing on tracing the forward pass.

[amyeroberts]

Hi @jprivera44,

Glad to hear you've already got the weight loading logic working!

Anyone is welcome to open a model PR in the library. However, please be aware that it is no longer the preferred method. Any model code merged directly into the repo brings its own maintenance costs, and so the barrier to add is a lot higher.

Our experience is that model PRs always take a lot longer that one expects, and is a large amount of work for both parties, particularly if the contributor hasn't added models previously.

With regards to your points:

• SOTA or not, models are just as easily used if they're implemented on the hub. For example, the recent falcon model was first added there.

• I'm not sure how you're defining interpretability, however model code should be equivalently understandable in either place (I'd argue it's easier on the hub without unrelated code & commits - but it's all subjective)

• What will be a blocker is adding through a PR here. As mentioned above, it can be a long process and, as other community members haven't requested this model, it won't be a priority for us to review and merge in.

[jprivera44]

Hi @amyeroberts, that all makes sense on my end!

I'll go ahead and add the scGPT model via the custom models link you mentioned, as the initial version of this code.