search

facebookresearch / stopes

A library for preparing data for machine translation research (monolingual preprocessing, bitext mining, etc.) built by the FAIR NLLB team.
https://facebookresearch.github.io/stopes/
MIT License
251 stars 37 forks source link

attribution of LLMs #65

Open Wafaa014 opened 10 months ago

Wafaa014 commented 10 months ago

Can you offer support for the ALTI attribution method for LLMs such as LLAMA?

avidale commented 10 months ago

Hi Wafaa! Currently, Stopes is focused only on translation models, and ALTI+ was implemented only for seq2seq transformers, such as NLLB. We are not currently planning to adapt ALTI+ to other architectures.

If I learn that our other colleagues who are working with LLMs implement ALTI+ or similar attribution methods for their models, I will update this thread accordingly.

Otherwise, I suggest that maybe you make such a contribution. LLaMA is a decoder-only transformer, so the part of ALTI+ code responsible for the decoder can in principle be adapted to it. But of course, this adaptation will depend on the chosen framework and other exact implementation details of the LLM code.