search

rmihaylov / falcontune

Tune any FALCON in 4-bit
Apache License 2.0
468 stars 51 forks source link

FalconLite support #37

Open hvico opened 1 year ago

hvico commented 1 year ago

Hello.

Amazon has released a 4-bit quantized version of Falcon40B that supports long context (up to 11K tokens).

It would be great to be able to finetune that extended context version using falcontune on a custom dataset.

https://huggingface.co/amazon/FalconLite https://medium.com/@chenwuperth/extend-the-context-length-of-falcon40b-to-10k-85d81d32146f

Thanks for the great work!

fatymatariq commented 1 year ago

Hi

Any success finetuning FalconLite model?