search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
9.77k stars 974 forks source link

Support for BLOOM #368

Open LamOne1 opened 1 year ago

LamOne1 commented 1 year ago

I'd like to request a support for BLOOM as it was pretrained on many languages

carmocca commented 1 year ago

This would require adding support for ALiBi. This is not a priority at the moment.

A related feature request is https://github.com/Lightning-AI/lit-gpt/issues/199

Jeronymous commented 10 months ago

+1

rasbt commented 6 months ago

Flash attention now supports Alibi btw, so in that case, Alibi support in Lit-GPT could be a bit easier.

carmocca commented 6 months ago

We don't support flash attention from flash-attn. Supporting alibi would warrant an entirely new model_alibi.py definition

rasbt commented 6 months ago

I think we should update this perhaps:

Screenshot 2024-02-23 at 8 40 28 AM
carmocca commented 6 months ago

We support flash attention via PyTorch's scaled_dot_product_attention, just not via Tri Dao's flash-attn. The former uses one of the latter's implementations internally.