search

princeton-nlp / AutoCompressors

[EMNLP 2023] Adapting Language Models to Compress Long Contexts
https://arxiv.org/abs/2305.14788
271 stars 20 forks source link

Does Auto-Compressors support newer base model like LLama-3 or Qwen-2.5? #28

Open RewindL opened 3 days ago

RewindL commented 3 days ago

I have tried to train with newer base model LLama-3-3B, and upgrade dependent libraries like transformers, accelerate and so on. But still fails to train.

CodeCreator commented 2 days ago

Hey! We built this library for an old version of transformers and we do not have the bandwidth to keep it up-to-date with every new huggingface version and model architecture. If you figure out which changes are necessary to support Llama-3, please open a pull request!

RewindL commented 2 days ago

Hey! We built this library for an old version of transformers and we do not have the bandwidth to keep it up-to-date with every new huggingface version and model architecture. If you figure out which changes are necessary to support Llama-3, please open a pull request!

Thanks for replying. Have a good day.