Request for a ChatBot Example

[Taha-Bahadori]

We intend to use Trak for attributing the outputs of a Q&A system to its data sources. We can see an example in qnli.py. Can you provide an example where the LLM is used for generating free-form text (generative setup)?

More specifically, consider tinyllama. How can we use Trak to attribute its response to questions to the data sources? How should we define the AbstractModelOutput in the generative case?

Even some guidelines would help too.

[kristian-georgiev]

Hi @Taha-Bahadori, check out our tutorial on implementing a custom model output for BERT: https://trak.readthedocs.io/en/latest/bert.html. I haven't used tiny llama before, but my guess is, to start, you'll need to replace transformers.AutoModelForSequenceClassification with transformers.AutoModelForCausalLM. Hope this helps.