About the version of FastChat

[Hannibal046]

Hi authors,

thanks for the great work!

In order to replicate SAIL-7B from scratch, may I ask which version or commit id of FastChat do you use?

[luohongyin]

Thanks for asking! I hope our latest research is helpful: https://arxiv.org/pdf/2310.00160.pdf

I'm now using FastChat 0.2.3, commit 7f18c273a28f31671033c508bb42c31b4da4e8ab.

The trained SAIL-7b model can be downloaded from link

[Hannibal046]

Thanks for bring the new paper. Could you please further explain the relationship between this new paper and SAIL? I am a little confused here. Or should I turn to this new paper to replicate a stronger SAIL model?

[luohongyin]

I brought the new paper because the SAIL fine-tuning needs a training corpus. If that available, the new paper provides a retrieval-augmented self instruction method I hope that would be helpful.

My understanding was that you were training to replicate SAIL on another domain (sorry if that was my hallucination). If you are using an existing instruction following corpus to fine-tune the model, this repo and previous information I provided might be helpful enough. Please feel free to let me know if there are any follow-up questions!

[Hannibal046]

Thanks for the response! Do you use this script to generate answer for Vicuna-80?

git clone https://github.com/lm-sys/vicuna-blog-eval.git
python get_model_answer.py --model-id sail-7b --model-path luohy/SAIL-7b --question-file table/question.jsonl --answer-file table/answer/answer_sail_7b.jsonl

These are some results I got from the above command:

{"question_id": 1, "text": "Time management is a valuable skill that can help you become more efficient and productive. Here are some tips to improve your time management skills:\n1. Set clear goals: Identify what you want to achieve and prioritize your goals in order of importance.\n2. Create a schedule: Plan out your day, week or month in advance and allocate time for each task.\n3. Stay focused: Avoid distractions and concentrate on one task at a time.\n4. Break up larger tasks: Divide large and daunting tasks into smaller, manageable parts.\n5. Take regular breaks: Give your brain a rest and take short breaks throughout the day.\n6. Delegate: Don\u2019t be afraid to delegate tasks to others if you can.\n7. Use tools and apps: Use time management tools and apps to help you stay on track, such as calendars, timers, and to-do lists.\n8. Learn from your mistakes: Evaluate your time management skills regularly and identify areas for improvement.\nBy following these tips, you can improve your time management skills and become more efficient and productive.", "answer_id": "bTVzbFqVCUNuAc7d98ofkx", "model_id": "sail-7b", "metadata": {}}
{"question_id": 2, "text": "There are many effective ways to deal with stress, some of which include:\n1. Exercise: Regular physical activity can help reduce stress and improve overall mental health.\n2. Relaxation techniques: Techniques such as deep breathing, meditation, and yoga can help calm the mind and reduce stress.\n3. Social support: Talking to friends, family, or a therapist about your feelings can provide emotional support and help reduce stress.\n4. Healthy lifestyle: A balanced diet, regular sleep, and avoiding drugs and alcohol can all contribute to better mental health and reduce stress.\n5. Time management: Managing your time effectively can help reduce stress by allowing you to better organize your responsibilities and avoid feeling overwhelmed.\n6. Problem-solving: Identifying the sources of your stress and developing a plan to address them can help reduce the stress and provide a sense of control.\n7. Laughter: Laughing and engaging in light-hearted activities can help reduce stress and improve your mood.\nRemember that everyone experiences stress differently, so it\u2019s important to find what works best for you. If stress becomes overwhelming or interferes with your daily life, consider seeking professional help.", "answer_id": "XmBx4hZCKtLCWnumBZR2dR", "model_id": "sail-7b", "metadata": {}}

It looks pretty good. And I have a question about the In-context Retrieval Selection part, since the LLM was trained to output something like Search result (1) is informative and search result (2) is distracting, so I will use the information from the search result (1) Why there is not a single answer starting with this?

[Hannibal046]

Hi, @luohongyin , Sorry to bother, could you please take some time to check this? Thanks so much!

[luohongyin]

Sry for being offline during holiday! I used vicuna-80's official prompt but not their script when I was running the experiments. To enable self-denoising, I added top-3 search result from duckduckgo for each input questions and constructed a search augmented prompt as shown in the paper.

[Hannibal046]

Thanks for reply! So my question is why downloaded checkpoint wouldn't output denoising part Search result (1) is informative and search result (2) is distracting, so I will use the information from the search result (1) which is treated as target during fine-tuning?

[luohongyin]

could you attach your prompt?

[Hannibal046]

Hi, I used the standard prompt here

And this is the results from a model trained within this repo and there are many denoisng part as model output:

However, using the provided checkpoint, such phenomenon disappeared.

[luohongyin]

Got it, did you pair vicuna questions with additional search results?在 2023年12月28日，10:19，Hannibal046 @.***> 写道： Hi, I used the standard prompt here And this is the results from a model trained within this repo and there are many denoisng part as model output: image.png (view on web) However, using the provided checkpoint, such phenomenon disappeared. image.png (view on web)

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[Hannibal046]

No, I didn't pair vicuna questions with additional search research.

[luohongyin]

As I mentioned in a previous response, I paired the questions with search results generated by duckduckgo and constructed search-augmented prompts as shown in the paper. Without additional search results, the model does not need "search denoising".

Hannibal046 @.***> 于2023年12月28日周四 12:07写道：

No, I didn't pair vicuna questions with additional search research.

— Reply to this email directly, view it on GitHub https://github.com/luohongyin/SAIL/issues/7#issuecomment-1871350729, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABL3SWHC64FK2XQBJKBE7ULYLWRM5AVCNFSM6AAAAABAFS5IRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZRGM2TANZSHE . You are receiving this because you were mentioned.Message ID: @.***>

-- Hongyin Luo 罗鸿胤 Ph.D. student, research assistant Computer Science & Artificial Intelligence Laboratory Massachusetts Institute of Technology

[Hannibal046]

Sorry, I don't know if I misunderstand it. Just to clarify, the search denosing here is given by an NLI model in the training set, for the Vicuna-80, since we don't know the ground truth before generation, so we couldn't get search denosing by an NLI model. Instead, the SAIL model should generate it. Do I understand this correctly?

[luohongyin]

Yes, what you said is correct, but SAIL can only do the denoising when search results are provided in a RAG pipeline. Please try our demo: https://huggingface.co/spaces/luohy/SAIL-7B

Click on "search augmentation" and the model will generate responses by concatenating search results AND the input instructions.

Hannibal046 @.***> 于2023年12月29日周五 01:15写道：

Sorry, I don't know if I misunderstand it. Just to clarify, the search denosing here is given by an NLI model in the training set, for the Vicuna-80, since we don't know the ground truth before generation, so we couldn't get search denosing by an NLI model. Instead, the SAIL model should generate it. Do I understand this correctly?

— Reply to this email directly, view it on GitHub https://github.com/luohongyin/SAIL/issues/7#issuecomment-1871759971, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABL3SWGNRJONYD3WWVSKKOTYLZNZFAVCNFSM6AAAAABAFS5IRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZRG42TSOJXGE . You are receiving this because you were mentioned.Message ID: @.***>

-- Hongyin Luo 罗鸿胤 Ph.D. student, research assistant Computer Science & Artificial Intelligence Laboratory Massachusetts Institute of Technology