Server-side token counting and message trimming

[sweep-ai[bot]]

PR Feedback (click)

• [ ] 👍 Sweep Did Well
• [ ] 👎 Sweep Needs Improvement

  Description

  This pull request includes changes to the generate.ts file in the src/pages/api directory. The changes involve adding server-side token counting and message trimming logic to the existing code.

Summary

• Added import statements for ProxyAgent and fetch from the undici library.
• Imported generatePayload and parseOpenAIStream functions from the @/utils/openAI module.
• Imported verifySignature function from the @/utils/auth module.
• Defined the APIRoute type from the astro module.
• Initialized variables apiKey, httpsProxy, baseUrl, sitePassword, and ua using values from import.meta.env.
• Created a constant array FORWARD_HEADERS containing headers to be forwarded.
• Implemented the post function that handles the API request.
• Added input validation for the messages field and returned an error response if it is empty.
• Added password validation and returned an error response if the provided password is invalid.
• Added signature validation and returned an error response if the signature is invalid.
• Initialized the encoder instance from the tiktoken-js library.
• Defined minMessages and maxTokens variables for message truncation.
• Created a countTokens function to count the tokens in a message using the encoder instance.
• Implemented message truncation logic to ensure the total number of tokens does not exceed the maxTokens limit.
• Updated the initOptions variable with the new messages array.
• Updated the headers object with the appropriate values.
• Added a check for the baseUrl and forwarded specific headers if it exists.
• Added the ua header if it is defined.
• Added a check for the httpsProxy and updated the initOptions object with a new dispatcher property if it exists.
• Made a fetch request to the OpenAI API endpoint and handled any errors that occurred.
• Returned the parsed response from the OpenAI API.

Fixes #17.

---

🎉 Latest improvements to Sweep:

• Sweep uses OpenAI's latest Assistant API to plan code changes and modify code! This is 3x faster and significantly more reliable as it allows Sweep to edit code and validate the changes in tight iterations, the same way as a human would.
• Sweep now uses the rope library to refactor Python! Check out Large Language Models are Bad at Refactoring Code. To have Sweep refactor your code, try sweep: Refactor <your_file>.py!

---

💡 To get Sweep to edit this pull request, you can:

• Comment below, and Sweep can edit the entire PR
• Comment on a file, Sweep will only modify the commented file
• Edit the original issue to get Sweep to recreate the PR from scratch

[sweep-ai[bot]]

Sandbox Executions

[sweep-ai[bot]]

Apply Sweep Rules to your PR?

• [ ] Apply: All new business logic should have corresponding unit tests.
• [ ] Apply: Refactor large functions to be more modular.
• [ ] Apply: Add docstrings to all functions and file headers.

[netlify[bot]]

❌ Deploy Preview for endless-chat failed.

Name	Link
🔨 Latest commit	0b278c6e111249382eb1edf83ce2f19b9e62f9fe
🔍 Latest deploy log	https://app.netlify.com/sites/endless-chat/deploys/6569aa9c6fed690008602cd7

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
44444444444	❌ Failed (Inspect)			Dec 1, 2023 9:44am
emphasize	❌ Failed (Inspect)			Dec 1, 2023 9:44am
endless-chat	❌ Failed (Inspect)			Dec 1, 2023 9:44am
free-chat	❌ Failed (Inspect)			Dec 1, 2023 9:44am
free-chat-personal	❌ Failed (Inspect)			Dec 1, 2023 9:44am

[senior-dev-bot[bot]]

Hi there! :wave: Thanks for opening a PR. :tada: To get the most out of Senior Dev, please sign up in our Web App, connect your GitHub account, and add/join your organization CNSeniorious000. After that, you will receive code reviews beginning on your next opened PR. :rocket:

[pr-explainer-bot[bot]]

Pull Request Report

Hey there! 👋 Here's a report on the changes made in the src/pages/api/generate.ts file:

Changes

1. Updated import statements for ProxyAgent and fetch from the undici library. (View Diff)
2. Updated import statements for generatePayload and parseOpenAIStream from the @/utils/openAI module. (View Diff)
3. Updated import statements for verifySignature from the @/utils/auth module. (View Diff)
4. Updated import statement for APIRoute from the astro module. (View Diff)
5. Updated the value of apiKey using the import.meta.env.OPENAI_API_KEY environment variable. (View Diff)
6. Updated the value of httpsProxy using the import.meta.env.HTTPS_PROXY environment variable. (View Diff)
7. Updated the value of baseUrl using the import.meta.env.OPENAI_API_BASE_URL environment variable. If the variable is not defined, the default value is set to 'https://api.openai.com'. (View Diff)
8. Updated the value of sitePassword using the import.meta.env.SITE_PASSWORD environment variable. (View Diff)
9. Updated the value of ua using the import.meta.env.UNDICI_UA environment variable. (View Diff)
10. Added a check for the messages property in the request body. If it is not present, a 400 Bad Request response is returned with an error message. (View Diff)
11. Added a check for the sitePassword and pass values. If they are not equal, a 401 Unauthorized response is returned with an error message. (View Diff)
12. Added a check for the import.meta.env.PROD flag and the signature verification. If the signature is invalid, a 401 Unauthorized response is returned with an error message. (View Diff)
13. Initialized the tiktoken-js library and obtained an encoder instance for the 'en' language. (View Diff)
14. Defined the minMessages and maxTokens variables for message truncation. Adjusted as needed. (View Diff)
15. Added a function countTokens to count the tokens for a message using tiktoken-js. (View Diff)
16. Implemented message truncation logic to ensure the total tokens do not exceed the maxTokens limit. (View Diff)
17. Updated the initOptions object with the generated payload, headers, and other options. (View Diff)
18. Added forwarding of specific headers from the request to the headers object. (View Diff)
19. Added the user-agent header if the ua variable is defined. (View Diff)
20. Added a check for the httpsProxy variable and set the dispatcher option to a ProxyAgent instance if it is defined. (View Diff)
21. Handled errors during the fetch request and returned a 500 Internal Server Error response with the error details. (View Diff)
22. Returned the parsed response from parseOpenAIStream as the API response. (View Diff)

Suggestions to Improve Code

• Consider using more descriptive variable names to improve code readability. For example, ua could be renamed to userAgent. (View Diff)

Bugs

• No bugs found.

Improvements

• No improvements found.

Rating

I would rate the code a 7 out of 10 based on the following criteria:

• Readability: The code is generally readable, but some variable names could be improved for better clarity. (View Diff)
• Performance: The code appears to be efficient and does not contain any obvious performance issues.
• Security: The code includes checks for authentication and signature verification, which is a good security practice.

Keep up the good work! If you have any questions or need further assistance, feel free to ask. 😄

---

Code Diffs

Diff-1

- import { ProxyAgent, fetch } from 'undici'
+ import { ProxyAgent, fetch } from 'undici'

Diff-2

- import { generatePayload, parseOpenAIStream } from '@/utils/openAI'
+ import { generatePayload, parseOpenAIStream } from '@/utils/openAI'

Diff-3

- import { verifySignature } from '@/utils/auth'
+ import { verifySignature } from '@/utils/auth'

Diff-4

- import type { APIRoute } from 'astro'
+ import type { APIRoute } from 'astro'

Diff-5

- const apiKey = import.meta.env.OPENAI_API_KEY
+ const apiKey = import.meta.env.OPENAI_API_KEY

Diff-6

- const httpsProxy = import.meta.env.HTTPS_PROXY
+ const httpsProxy = import.meta.env.HTTPS_PROXY

Diff-7

- const baseUrl = ((import.meta.env.OPENAI_API_BASE_URL) || 'https://api.openai.com').trim().replace(/\/$/, '')
+ const baseUrl = ((import.meta.env.OPENAI_API_BASE_URL) || 'https://api.openai.com').trim().replace(/\/$/, '')

Diff-8

- const sitePassword = import.meta.env.SITE_PASSWORD
+ const sitePassword = import.meta.env.SITE_PASSWORD

Diff-9

+ const ua = import.meta.env.UNDICI_UA

Diff-10

+   if (!messages) {
+     return new Response(JSON.stringify({
+       error: {
+         message: 'No input text.',
+       },
+     }), { status: 400 })
+   }

Diff-11

+   if (sitePassword && sitePassword !== pass) {
+     return new Response(JSON.stringify({
+       error: {
+         message: 'Invalid password.',
+       },
+     }), { status: 401 })
+   }

Diff-12

+   if (import.meta.env.PROD && !await verifySignature({ t: time, m: messages?.[messages.length - 1]?.content || '' }, sign)) {
+     return new Response(JSON.stringify({
+       error: {
+         message: 'Invalid signature.',
+       },
+     }), { status: 401 })
+   }

Diff-13

+ const encoder = new Encoder('en');

Diff-14

+ const minMessages = 5;
+ const maxTokens = 1024;

Diff-15

+ function countTokens(message) {
+     return encoder.encode(message).length;
+ }

Diff-16

+ let totalTokens = messages.reduce((sum, msg) => sum + countTokens(msg.content), 0);
+ while (totalTokens > maxTokens && messages.length > minMessages) {
+     totalTokens -= countTokens(messages[0].content);
+     messages.shift();
+ }

Diff-17

+ const initOptions = generatePayload(request.headers.get('Authorization') ?? `Bearer ${apiKey}`, messages, model);

Diff-18

+   if (baseUrl) request.headers.forEach((val, key) => (FORWARD_HEADERS.includes(key) || key.startsWith('sec-') || key

[netlify[bot]]

❌ Deploy Preview for chat-for-free failed.

Name	Link
🔨 Latest commit	0b278c6e111249382eb1edf83ce2f19b9e62f9fe
🔍 Latest deploy log	https://app.netlify.com/sites/chat-for-free/deploys/6569aa9ca4c97f00088a2feb