search

mlabonne / llm-course

Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.
https://mlabonne.github.io/blog/
Apache License 2.0
34.29k stars 3.58k forks source link

Train Custom data set PDF #12

Open diouck opened 8 months ago

diouck commented 8 months ago

Hello, Super interesting. I would like to train the model on my own data in pdf format. How to adapt the code. instead of using # The instruction dataset to use dataset_name = "mlabonne/guanaco-llama2-1k" I would like to replace it with dataset= doc.pdf but it sucks. Do you have an idea? THANKS

mlabonne commented 8 months ago

Hi, in your case you want to create a dataset based on your PDFs. The code you're talking about is designed for instruction tuning, which means it requires pairs of instructions and answers. You could generate those using ChatGPT for example (see this: https://github.com/teknium1/GPTeacher).

Another option is to change the training objective from instruction tuning to next-token prediction (the pre-training task), but you can't use this code. I'd recommend checking the Axolotl library in this case: https://github.com/OpenAccess-AI-Collective/axolotl