reproducible environment for finetuning

[despiegk]

aim is to make everyones life easier to experiment with AI

requirements & deliverables

• create python script to setup a VM in VAST.AI (all automated)
• easy install (integrated with the setup script on VAST.AI)
  • bash script for installing all required components (use Ubuntu 22.04)
  • requirements script for pip install (required python components)
  • a test script to check the cuda is working in right version & the GPU is found with minimal requirements
• a python script which parses pdf (use pdf2text commandline, not the python version), uses wiki txt files, download a website e.g. manual.grid.tf and question/answer files for finetuning and populates the model
• experiment with multiple models and demonstrate results (timing, quality, ...): see below
• bring openai compatible API life on top of Avast GPU machine (ok to do SSH portforwarding)
• all is opensource
• an end2end test
  • create a docker build (or use vbuilder even better to make build
  • build the docker upload to docker hub (even the test data: pdf, doing a web crawl to manual ... as part of it)
  • deploy using vast AI and use the above created docker
  • there should now a machine online which has API compatible with openaid
  • now do a script which calls the API and queries the content and proves that info from manual, pdf is there
  • there should be link back to where the info comes from in the result (can this be done?)
• document all in an mdbook: results of different models, performance in relation to GPU mem, quality, how to start, ...
  • idea is that with nothing more than the mdbook a scripter person with some linux expertise can re-do all the tests

gpu usage

• use instances on https://cloud.vast.ai/instances/ (A6000 max)
• test A6000, 4900 24 and 40 GB mem config's
• test single and double A6000

models

experiment with following models

• falcon 7B
• falcon 70B (load on 40 GB GPU, there are tricks)
• falcon 70b on DUAL GPU A6000 instance
• llama2 (see if better)

some info which can be used

Fine-tuning Large Language Model (LLM) on a Custom Dataset with QLoRA _ MLExpert - Crush Your Machine Learning interview.pdf

implementation details

• all scripts use 'set -ex' to make sure they stop when error
• can also use vscript (see vlang) as alternative to bash, we have quite some primitives working see crystallib (ask codescalers team for help if needed)
• put some example pdf, and text files in this repo' so its easy for people to experiment
• use info from threefold and see how the results are for questions and answer: https://manual.grid.tf/

[despiegk]

Chat with Multiple PDFs using Llama 2 and LangChain _ MLExpert - Crush Your Machine Learning interview.pdf Private GPT4All Chat with PDF Files Using Free LLM MLExpert - Crush Your Machine Learning interview.pdf Private Chatbot with Local LLM (Falcon 7B) and LangChain _ MLExpert - Crush Your Machine Learning interview.pdf

[despiegk]

Support Chatbot using Custom Knowledge Base with LangChain and Open LLM _ MLExpert - Crush Your Machine Learning interview.pdf