Open obuzek opened 6 months ago
We need better documentation on ensuring that the skills you contribute will actually improve the model.
People have reported their
generate_.jsonl
andtrain_.jsonl
files (as a result of runninglab generate
) contain relatively inaccurate alternate instruction suggestions. We'll likely get a better skill tree if people learn to look at these outputs, and refine the verbiage of their skills until they're getting strong answers from the teacher model (the one you load up withlab serve
before you runlab generate
).@xukai92's commentary:
- how easy/hard to get good synthetic data depends on the q/a themselves. perhaps rewriting or rephrasing could help
- there are some improvements on the method itself we will deploy in the next few days
- potentially switch the model used by lab generate could be helpful, but it requires a better machine. all it needs is an endpoint but instructions to be added (instructions to use a stronger teacher model cli#398)
- if you can load
mixtral-8x7b-v0.1.Q4_K_M.gguf
from https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF in lab serve and document how to do this it would be very helpful for othersTasks
- [ ] Add Kai's commentary to the skill info docs
- [ ] Suggest in both skill development and triage info docs that changing your wording and rerunning the skill until your
.jsonl
output has improved will be more beneficial to the model
Hi @obuzek
It would be helpful for the first time contributers if you could please provide references to the docs you've mentioned above (skill info, skill development & triage info docs)
I've taken this opportunity to provide the references, kindly update if they are incorrect
skill info doc : SKILLS_GUIDE.md skill development doc : SKILLS_GUIDE.md triage info doc : skills-triage.md
The Community Learning Guide covers some relevant topics, but not yet this one.
Automatic generation in response to PRs is getting added soon, which may help people with this.
Leaving this open since this remains relevant information.
This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 31 days.
We need better documentation on ensuring that the skills you contribute will actually improve the model.
People have reported their
generate_.jsonl
andtrain_.jsonl
files (as a result of runninglab generate
) contain relatively inaccurate alternate instruction suggestions. We'll likely get a better skill tree if people learn to look at these outputs, and refine the verbiage of their skills until they're getting strong answers from the teacher model (the one you load up withlab serve
before you runlab generate
).@xukai92's commentary:
Tasks
.jsonl
output has improved will be more beneficial to the model