intelligentnode / IntelliNode

Access the latest AI models like ChatGPT, LLaMA, Diffusion, Gemini Hugging face, and beyond through a unified prompt layer and performance evaluation
https://show.intellinode.ai
Apache License 2.0
225 stars 15 forks source link

Update OpenAIWrapper to add fine tuning functionality with unit test cases. #15

Closed Barqawiz closed 9 months ago

Barqawiz commented 1 year ago

Openai Reference:

Work scope: The Openai API supports the ability to upload a json file with the training data and run fine-tuning functions on the data. Supporting this in intelliNode requires adding the following minimum functions to OpenAIWrapper.js:

Step1: Prepare the testing data in a file, for example:

{
  "messages": [
    { "role": "system", "content": "You are an assistant that occasionally misspells words" },
    { "role": "user", "content": "Tell me a story." },
    { "role": "assistant", "content": "One day a student went to schoool." }
  ]
}

Step2: Add a function inside OpenAIWrapper that match the following call to upload the file:

curl https://api.openai.com/v1/files \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F "purpose=fine-tune" \
  -F "file=@path_to_your_file" 

Step3: Add a function inside OpenAIWrapper that match the following call to create fine tuning hob:

curl https://api.openai.com/v1/fine_tuning/jobs \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
  "training_file": "TRAINING_FILE_ID",
  "model": "gpt-3.5-turbo-0613"
}'

Testing: Add the testing scripts in OpenAIWrapper.test.js:

Thedevelop3r commented 1 year ago

added fine tuning functionality, with unit tests

https://github.com/intelligentnode/IntelliNode/pull/48