search

Agentic-Learning-AI-Lab / anticipatory-recovery-public

Public code release for the paper "Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training"
MIT License
9 stars 0 forks source link

Documents used for fine-tuning the LLMs cyclicly? #1

Closed haizhou-shi closed 6 months ago

haizhou-shi commented 6 months ago

Hi, thanks for sharing the code of this intriguing paper!

I am wondering do you have the document list described in Section 2 "Training Setup"?

Thanks in advance!

yanlai00 commented 6 months ago

The dataset we used is the CNN/Dailymail news dataset on huggingface: https://huggingface.co/datasets/cnn_dailymail/viewer/3.0.0

The 25 documents used in the experiment is randomly sampled for each run.

We plan to include experiments on other datasets in a future update to the paper.

haizhou-shi commented 6 months ago

Thank you so much for your timely response!

For now, I am a bit curious about the 25 randomly sampled documents, and by taking a look at them, try to interpret this intriguing behavior of anticipatory recovery by taking. Do you have any suggestions on this issue?

Best,

yanlai00 commented 6 months ago

In the code there is a line to decode and visualize the sampled documents - you can apply it to every sample: https://github.com/Agentic-Learning-AI-Lab/anticipatory-recovery-public/blob/main/llm-memory/training/train_interleave.py#L275