Sweep: Implement uploading of files for model fine-tuning to OpenAI API

Here's the PR! https://github.com/lehcode/soakp/pull/162.

⚡ Sweep Free Trial: I used GPT-3.5 to create this ticket. You have 4 GPT-4 tickets left for the month and 0 for the day. For more GPT-4 tickets, visit our payment portal. To retrigger Sweep edit the issue.

Step 1: 🔍 Code Search

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/lehcode/soakp/blob/2ff492cdbb2b18dc2ef89f46ef4506e9a40af1f4/src/openai/OpenaiFinetunesApi.ts#L1-L213 https://github.com/lehcode/soakp/blob/2ff492cdbb2b18dc2ef89f46ef4506e9a40af1f4/src/SoakpProxy.ts#L1-L397 https://github.com/lehcode/soakp/blob/2ff492cdbb2b18dc2ef89f46ef4506e9a40af1f4/src/openai/OpenaiFilesApi.ts#L1-L187 https://github.com/lehcode/soakp/blob/2ff492cdbb2b18dc2ef89f46ef4506e9a40af1f4/src/SoakpServer.ts#L10-L133 https://github.com/lehcode/soakp/blob/2ff492cdbb2b18dc2ef89f46ef4506e9a40af1f4/src/openai/OpenaiFilesApi.ts#L154-L298

I also found some related docs:

Summary of related docs from https://platform.openai.com/docs/:

To implement uploading of files for model fine-tuning to the OpenAI API, you can follow these steps:

In the SoakpProxy.ts file, add a new method called uploadFineTuneFile that takes a file and a purpose as parameters. This method should use the OpenAI API's createFile method to upload the file. The purpose should be set to 'fine-tune'.

In the OpenaiFinetunesApi.ts file, add a new endpoint called POST /openai/fine-tunes/upload that handles the HTTP request for file upload. This endpoint should call the uploadFineTuneFile method in the SoakpProxy class to upload the file to the OpenAI API.

Convert the uploaded files into JSONL format using Python means.

By following these steps, you will be able to implement the uploading of files for model fine-tuning to the OpenAI API.

Sources:

https://platform.openai.com/docs/api-reference/files

https://platform.openai.com/docs/api-reference/embeddings

https://platform.openai.com/docs/api-reference/chat/object

https://platform.openai.com/docs/api-reference/completions/create

https://platform.openai.com/docs/api-reference/models/object

Step 2: 🧐 Snippet Analysis

From looking through the relevant snippets, I decided to make the following modifications:

File Path	Proposed Changes
`src/SoakpProxy.ts`	Modify src/SoakpProxy.ts with contents: • Add a new method `uploadFineTuneFile` that takes a file and a purpose as parameters. This method should use the OpenAI API's `createFile` method to upload the file. The purpose should be 'fine-tune'.
`src/openai/OpenaiFinetunesApi.ts`	Modify src/openai/OpenaiFinetunesApi.ts with contents: • Add a new endpoint `POST /openai/fine-tunes/upload` that handles the HTTP request for file upload. This endpoint should call the `uploadFineTuneFile` method in the `SoakpProxy` class to upload the file to the OpenAI API.
`src/openai/OpenaiFilesApi.ts`	Modify src/openai/OpenaiFilesApi.ts with contents: • Add a new method `convertToJSONL` that takes a file as a parameter and converts it into JSONL format using Python means. This method should be called before the file is uploaded to the OpenAI API.

Step 3: 📝 Planning

I have created a plan for writing the pull request. I am now working my plan and coding the required changes to address this issue. Here is the planned pull request:

Implement uploading of files for model fine-tuning to OpenAI API sweep/upload-files-for-fine-tuning

Description

This PR implements the functionality to upload files for model fine-tuning to the OpenAI API. It adds a new method uploadFineTuneFile to the SoakpProxy class, which uses the OpenAI API's createFile method to upload the file. The purpose is set to 'fine-tune'. Additionally, a new endpoint POST /openai/fine-tunes/upload is added to the OpenaiFinetunesApi class, which handles the HTTP request for file upload and calls the uploadFineTuneFile method in the SoakpProxy class.

To ensure compatibility with the OpenAI API, the uploaded files are converted into JSONL format using Python means. The conversion functionality is added to the OpenaiFilesApi class.

Summary of Changes

Added uploadFineTuneFile method to SoakpProxy class to upload files for model fine-tuning to the OpenAI API.

Added POST /openai/fine-tunes/upload endpoint to OpenaiFinetunesApi class to handle file upload requests.

Added convertToJSONL method to OpenaiFilesApi class to convert uploaded files into JSONL format.

Step 4: ⌨️ Coding

File	Instructions	Progress
`src/SoakpProxy.ts`	Modify src/SoakpProxy.ts with contents: • Add a new method `uploadFineTuneFile` that takes a file and a purpose as parameters. This method should use the OpenAI API's `createFile` method to upload the file. The purpose should be 'fine-tune'.	✅ Commit `2ff492c`
`src/openai/OpenaiFinetunesApi.ts`	Modify src/openai/OpenaiFinetunesApi.ts with contents: • Add a new endpoint `POST /openai/fine-tunes/upload` that handles the HTTP request for file upload. This endpoint should call the `uploadFineTuneFile` method in the `SoakpProxy` class to upload the file to the OpenAI API.	✅ Commit `2f4bc14`
`src/openai/OpenaiFilesApi.ts`	Modify src/openai/OpenaiFilesApi.ts with contents: • Add a new method `convertToJSONL` that takes a file as a parameter and converts it into JSONL format using Python means. This method should be called before the file is uploaded to the OpenAI API.	✅ Commit `e6d29bf`	I have finished coding the issue. I am now reviewing it for completeness.

Step 5: 🔁 Code Review

Here are my self-reviews of my changes at sweep/upload-files-for-fine-tuning.

Here is the 1st review

Thanks for your contribution. There are a few changes that need to be made:

In src/SoakpProxy.ts, please ensure that openai is properly imported or is a property of the SoakpProxy class. Also, consider improving the error handling in the uploadFineTuneFile method to handle errors other than TypeError. You can find these changes on lines 394-410.

In src/openai/OpenaiFilesApi.ts, the convertToJSONL method is currently unimplemented. Please add the Python code to convert the file to JSONL format as indicated in the comment. This change is required on lines 386-395.

In src/openai/OpenaiFinetunesApi.ts, please ensure that the uploadFineTuneFile method and the middleware functions validateToken, getKeyStorage, getUser, getProxyInstance are properly implemented and are available in the context ctx. These changes are required on lines 60-65.

Please make these changes and update the pull request. If you need any help, feel free to ask.

I finished incorporating these changes.

To recreate the pull request edit the issue title or description. ^{Join Our Discord}

lehcode / soakp