[Performance] Tracking open Issues and PRs (pytorch transformers)

[stas00]

Let's use this Issue to track performance issues and enhancement requests, so it's easier to prioritize the work.

This is for pytorch transformers

Also I will label it as a Good Difficult Issue in case someone is ready for a challenging but rewarding experience of figuring things out. If you do want to take the challenge comment in the corresponding Issue/PR that resonates with you so others would know you're working on it.

If I missed any other relevant open performance-related Issues/PRs that need attention please comment below.

Regression:

• [ ] https://github.com/huggingface/transformers/pull/11218 Regression after Bart-like refactoring - need to compare the original Bart refactoring PR since most likely the regression happened there.
• [ ]

Odd slowness:

• [ ] https://github.com/huggingface/transformers/issues/10816 figuring out why eval with --fp16_full_eval is 25% slower
• [ ]

Fused kernels possibilities:

• [ ] https://github.com/huggingface/transformers/issues/11368 Megatron fused CUDA kernels to improve Hugging Face model classes' scalability
• [ ] research pytorch kernels?
• [ ] I know Deepspeed has various kernels that we might be able to use

Faster / leaner startup / module loading

• [ ] https://github.com/huggingface/transformers/issues/12274 - skip storage allocation which gets dropped for pretrained weights

Faster optimizers

• [ ] https://github.com/huggingface/transformers/issues/12084 - a proposal to port MemoryEfficientFP16Optimizer from fairseq
• [ ] https://github.com/huggingface/transformers/issues/9965 - torch.optim._multi_tensor faster optimizers - having some bottleneck in the test script - need to profile

Scalability

• [ ] https://github.com/huggingface/transformers/issues/10321 Tensor Parallelism

Deepspeed-specific features

• [ ] https://github.com/huggingface/transformers/issues/9606 a list of features that can be integrated
• [ ] https://github.com/huggingface/transformers/issues/12273 - make from_pretrained loading faster

Tests

• [ ] No issue yet, but we really need to add performance regression tests

[JuheonChu]

@stas00 If I want to work on this issue, should I pick one of those issues to keep track of its performance? Can you also tell me how I can keep track of the performances? Can you give me some guidance?

[stas00]

Hi @JuheonChu, this is not an Issue to work on. As the title says this is a collection of pointers to track other Issues. It's dated but many issues that it links to are still valid. So you can click on the issue that resonates with you and discuss the details there - not here.

I hope this addresses your question.