Sweep: Add backoff for batch sizes in the vector DB voyage API code

[kevinlu1248]

Details

We get the following error.

voyageai.error.InvalidRequestError: Request to model 'voyage-code-2' failed. The example at index 0 in your batch has too many tokens and does not fit into the model's context window of 16000 tokens. Please lower the number of tokens in the listed example(s) or use truncation.

This occurs in openai_call_embedding. When this error occurs, we should backoff by running it twice with half the batch size each. If it still happens, we should backoff again repeatedly.

Branch

No response

[sweep-nightly[bot]]

---

Actions (click)

• [ ] ↻ Restart Sweep

---

❌ Unable to Complete PR

It looks like an issue has occurred around fetching the files. Perhaps the repo failed to initialized. If this error persists contact team@sweep.dev.

@kevinlu1248, editing this issue description to include more details will automatically make me relaunch. Please join our Discord server for support (tracking_id=39b954ca68)

For bonus GPT-4 tickets, please report this bug on Discord (tracking ID: 39b954ca68).

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description.

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

✨ Track Sweep's progress on our progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: a448bdd669)

---

[!TIP] I can email you when I complete this pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/vector_db.py#L1-L170 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/utils/ticket_utils.py#L1-L280

---

Step 2: ⌨️ Coding

Working on it...

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

✨ Track Sweep's progress on our progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: f0ec24b07f)

---

[!TIP] I can email you when I complete this pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/docs/public/redis_vector_db.py#L1-L131 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/vector_db.py#L1-L170 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/utils/openai_proxy.py#L1-L324 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/utils/ticket_utils.py#L1-L280 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/config/server.py#L1-L230 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/context_pruning.py#L1-L230 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/handlers/on_ticket.py#L1609-L1776 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_wrapper.py#L271-L494 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/sweep_bot.py#L1-L260 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/docs/pages/blogs/vector-db-implementation.mdx#L50-L100 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_function_modify.py#L562-L703 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_function_modify.py#L454-L561 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_planning.py#L1-L226 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/handlers/on_ticket.py#L566-L818 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/modify_bot.py#L390-L690

---

Step 2: ⌨️ Coding

Working on it...

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

✨ Track Sweep's progress on our progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: 2d668b9ecb)

---

[!TIP] I can email you when I complete this pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

I am currently looking into this ticket! I will update the progress of the ticket in this comment. I am currently searching through your code, looking for relevant snippets.

---

Step 1: 🔎 Searching

I'm searching for relevant snippets in your repository. If this is your first time using Sweep, I'm indexing your repository. You can monitor the progress using the progress dashboard

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

[sweep-nightly[bot]]

---

Actions (click)

• [ ] ↻ Restart Sweep

---

❌ Unable to Complete PR

I'm sorry, but it looks like an error has occurred due to a planning failure. Feel free to add more details to the issue description so Sweep can better address it. Alternatively, reach out to Kevin or William for help at https://discord.gg/sweep.

For bonus GPT-4 tickets, please report this bug on Discord (tracking ID: 47711f404b).

---

Please look at the generated plan. If something looks wrong, please add more details to your issue.

File Path

Proposed Changes

sweepai/core/vector_db.py

Modify sweepai/core/vector_db.py with contents: • Import the InvalidRequestError exception from the voyageai package at the top of the file • In the openai_with_expo_backoff function: - Wrap the call to openai_call_embedding in a try/except block - Catch the InvalidRequestError in the except block - When caught, split the batch in half - Recursively call openai_with_expo_backoff on each half - Concatenate the results of the two recursive calls and return that • This will implement the backoff strategy by automatically reducing the batch size by half whenever the InvalidRequestError occurs due to the batch being too large. It will keep reducing until each sub-batch is small enough to get processed.

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description.

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

✨ Track Sweep's progress on our progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: 05c700ff68)

---

[!TIP] I can email you when I complete this pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/vector_db.py#L1-L170

---

Step 2: ⌨️ Coding

• [ ] sweepai/core/vector_db.py

  Modify sweepai/core/vector_db.py with contents:
  • Import the InvalidRequestError exception from the voyageai package at the top of the file
  • In the openai_with_expo_backoff function: - Wrap the call to openai_call_embedding in a try/except block - Catch the InvalidRequestError in the except block - When caught, split the batch in half - Recursively call openai_with_expo_backoff on each half - Concatenate the results of the two recursive calls and return that
  • This will implement the backoff strategy by automatically reducing the batch size by half whenever the InvalidRequestError occurs due to the batch being too large. It will keep reducing until each sub-batch is small enough to get processed.

---

Step 3: 🔁 Code Review

Working on it...

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

---

Actions (click)

• [ ] ↻ Restart Sweep

---

❌ Unable to Complete PR

I'm sorry, but it looks like an error has occurred due to a planning failure. Feel free to add more details to the issue description so Sweep can better address it. Alternatively, reach out to Kevin or William for help at https://discord.gg/sweep.

For bonus GPT-4 tickets, please report this bug on Discord (tracking ID: e2f47dfa8b).

---

Please look at the generated plan. If something looks wrong, please add more details to your issue.

File Path

Proposed Changes

sweepai/core/vector_db.py

Modify sweepai/core/vector_db.py with contents: • Import the InvalidRequestError exception from the voyageai package at the top of the file • In the openai_with_expo_backoff function: - Wrap the call to openai_call_embedding in a try/except block - Catch the InvalidRequestError in the except block - When caught, split the batch in half - Recursively call openai_with_expo_backoff on each half - Concatenate the results of the two recursive calls and return that • This will implement the backoff strategy by automatically reducing the batch size by half whenever the InvalidRequestError occurs due to the batch being too large. It will keep reducing until each sub-batch is small enough to get processed.

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description.

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

✨ Track Sweep's progress on our progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: acf8249f89)

---

[!TIP] I can email you when I complete this pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/vector_db.py#L1-L170

---

Step 2: ⌨️ Coding

• [ ] sweepai/core/vector_db.py

  Modify sweepai/core/vector_db.py with contents:
  • Import the InvalidRequestError exception from the voyageai package at the top of the file
  • In the openai_with_expo_backoff function: - Wrap the call to openai_call_embedding in a try/except block - Catch the InvalidRequestError in the except block - When caught, split the batch in half - Recursively call openai_with_expo_backoff on each half - Concatenate the results of the two recursive calls and return that
  • This will implement the backoff strategy by automatically reducing the batch size by half whenever the InvalidRequestError occurs due to the batch being too large. It will keep reducing until each sub-batch is small enough to get processed.

---

Step 3: 🔁 Code Review

Working on it...

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

✨ Track Sweep's progress on our progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: 90431ab6c5)

---

[!TIP] I can email you when I complete this pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/vector_db.py#L1-L170

---

Step 2: ⌨️ Coding

• [ ] sweepai/core/vector_db.py

  Modify sweepai/core/vector_db.py with contents:
  • Import the InvalidRequestError exception from the voyageai package at the top of the file
  • In the openai_with_expo_backoff function: - Wrap the call to openai_call_embedding in a try/except block - Catch the InvalidRequestError in the except block - When caught, split the batch in half - Recursively call openai_with_expo_backoff on each half - Concatenate the results of the two recursive calls and return that
  • This will implement the backoff strategy by automatically reducing the batch size by half whenever the InvalidRequestError occurs due to the batch being too large. It will keep reducing until each sub-batch is small enough to get processed.

---

Step 3: 🔁 Code Review

Working on it...

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

🚀 Here's the PR! #3413

See Sweep's progress at the progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: 6f7e7878e4)

---

[!TIP] I can email you next time I complete a pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/vector_db.py#L1-L170

---

Step 2: ⌨️ Coding

• [X] Modify sweepai/core/vector_db.py ✓ https://github.com/sweepai/sweep/commit/597daeb5d83dc21e63cb82dafecf932be06c8fd5 Edit

  Modify sweepai/core/vector_db.py with contents:
  • Import the InvalidRequestError exception from the voyageai package at the top of the file
  • In the openai_with_expo_backoff function: - Wrap the call to openai_call_embedding in a try/except block - Catch the InvalidRequestError in the except block - When caught, split the batch in half - Recursively call openai_with_expo_backoff on each half - Concatenate the results of the two recursive calls and return that
  • This will implement the backoff strategy by automatically reducing the batch size by half whenever the InvalidRequestError occurs due to the batch being too large. It will keep reducing until each sub-batch is small enough to get processed.

--- 
+++ 
@@ -9,6 +9,7 @@
 from redis import Redis
 from tqdm import tqdm
 import voyageai
+from voyageai import InvalidRequestError

 from sweepai.config.server import BATCH_SIZE, REDIS_URL, VOYAGE_API_KEY
 from sweepai.utils.hash import hash_sha256
@@ -129,21 +130,35 @@
         embeddings = np.array(embeddings)
         return embeddings  # all embeddings are in cache
     try:
-        # make sure all token counts are within model params (max: 8192)
+        try:
+            # make sure all token counts are within model params (max: 8192)

-        new_embeddings = openai_call_embedding(batch)
-    except requests.exceptions.Timeout as e:
-        logger.exception(f"Timeout error occured while embedding: {e}")
-    except Exception as e:
-        logger.exception(e)
-        if any(tiktoken_client.count(text) > 8192 for text in batch):
-            logger.warning(
-                f"Token count exceeded for batch: {max([tiktoken_client.count(text) for text in batch])} truncating down to 8192 tokens."
-            )
-            batch = [tiktoken_client.truncate_string(text) for text in batch]
             new_embeddings = openai_call_embedding(batch)
-        else:
-            raise e
+        except requests.exceptions.Timeout as e:
+            logger.exception(f"Timeout error occured while embedding: {e}")
+        except Exception as e:
+            logger.exception(e)
+            if any(tiktoken_client.count(text) > 8192 for text in batch):
+                logger.warning(
+                    f"Token count exceeded for batch: {max([tiktoken_client.count(text) for text in batch])} truncating down to 8192 tokens."
+                )
+                batch = [tiktoken_client.truncate_string(text) for text in batch]
+                new_embeddings = openai_call_embedding(batch)
+            else:
+                raise e
+    except InvalidRequestError:
+        # Split the batch in half
+        half = len(batch) // 2
+        first_half = batch[:half]
+        second_half = batch[half:]
+
+        # Recursively call on each half
+        first_half_embeddings = openai_with_expo_backoff(first_half)
+        second_half_embeddings = openai_with_expo_backoff(second_half)
+
+        # Concatenate the results and return
+        new_embeddings = np.concatenate((first_half_embeddings, second_half_embeddings))
+        return new_embeddings
     # get all indices where embeddings are None
     indices = [i for i, emb in enumerate(embeddings) if emb is None]
     # store the new embeddings in the correct position

• [X] Running GitHub Actions for sweepai/core/vector_db.py ✓ Edit

  Check sweepai/core/vector_db.py with contents:

Ran GitHub Actions for 597daeb5d83dc21e63cb82dafecf932be06c8fd5:
• Vercel Preview Comments: ✓

---

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/add_backoff_for_batch_sizes_in_the_vecto_11fb1.

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[sweep-nightly[bot]]

🚀 Here's the PR! #3414

See Sweep's progress at the progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: a3c0dd25ab)

---

[!TIP] I can email you next time I complete a pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/docs/public/redis_vector_db.py#L1-L131 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/vector_db.py#L1-L170 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/utils/openai_proxy.py#L1-L324 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/docs/pages/blogs/vector-db-implementation.mdx#L50-L100 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/config/server.py#L1-L230 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/handlers/on_ticket.py#L1609-L1776 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_wrapper.py#L271-L494 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_planning.py#L1-L226 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/modify_bot.py#L390-L690 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/docs/pages/blogs/vector-db-implementation.mdx#L1-L50 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_function_modify.py#L562-L703 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/context_pruning.py#L1-L230 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/core/chat.py#L138-L416 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/utils/ticket_utils.py#L1-L280 https://github.com/sweepai/sweep/blob/abfa7eec269f0378b3d938afe2ddfc6f9ada8ad1/sweepai/agents/assistant_function_modify.py#L1-L333

---

Step 2: ⌨️ Coding

• [X] Modify sweepai/core/vector_db.py ✓ https://github.com/sweepai/sweep/commit/9e53f55ae80cf8370e394f76aac7bbc3dbac453a Edit

  Modify sweepai/core/vector_db.py with contents:
  • Modify the openai_call_embedding function to catch the voyageai.error.InvalidRequestError exception
  • When this error is caught, recursively call openai_call_embedding with the batch split in half
  • Keep recursively calling with smaller batch sizes until the error is no longer thrown
  • Concatenate the results of the recursive calls to return the full batch of embeddings
  • Adjust the batch size import from sweepai/config/server.py if needed to a smaller starting size

--- 
+++ 
@@ -9,6 +9,7 @@
 from redis import Redis
 from tqdm import tqdm
 import voyageai
+from voyageai import error as voyageai_error

 from sweepai.config.server import BATCH_SIZE, REDIS_URL, VOYAGE_API_KEY
 from sweepai.utils.hash import hash_sha256
@@ -88,11 +89,20 @@
 # @redis_cache()
 def openai_call_embedding(batch, input_type: str="document"): # input_type can be query or document
     if VOYAGE_API_KEY:
-        client = voyageai.Client()
-        result = client.embed(batch, model="voyage-code-2", input_type=input_type)
-        cut_dim = np.array([data for data in result.embeddings])
-        normalized_dim = normalize_l2(cut_dim)
-        return normalized_dim
+        try:
+            client = voyageai.Client()
+            result = client.embed(batch, model="voyage-code-2", input_type=input_type)
+            cut_dim = np.array([data for data in result.embeddings])
+            normalized_dim = normalize_l2(cut_dim)
+            return normalized_dim
+        except voyageai.error.InvalidRequestError as e:
+            if len(batch) > 1:
+                mid = len(batch) // 2
+                left = openai_call_embedding(batch[:mid], input_type)
+                right = openai_call_embedding(batch[mid:], input_type)
+                return np.concatenate((left, right))
+            else:
+                raise e
     client = get_embeddings_client()
     response = client.embeddings.create(
         input=batch, model="text-embedding-3-small", encoding_format="float"

• [X] Running GitHub Actions for sweepai/core/vector_db.py ✓ Edit

  Check sweepai/core/vector_db.py with contents:

Ran GitHub Actions for 9e53f55ae80cf8370e394f76aac7bbc3dbac453a:
• Vercel Preview Comments: ✓

---

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/add_backoff_for_batch_sizes_in_the_vecto_1e531.

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[kevinlu1248]

🚀 Here's the PR! #3426

See Sweep's progress at the progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: 01fcd4110d)

---

[!TIP] I can email you next time I complete a pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/core/vector_db.py#L1-L227 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/docs/pages/blogs/vector-db-implementation.mdx#L50-L100 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/utils/openai_proxy.py#L1-L324 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/docs/public/redis_vector_db.py#L1-L131 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/agents/assistant_wrapper.py#L271-L494 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/platform/app/api/openai/create/route.ts#L1-L77 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/platform/app/api/openai/edit/route.ts#L1-L70 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/utils/convert_openai_anthropic.py#L1-L109 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/utils/multi_query.py#L1-L101 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/config/server.py#L1-L240 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/utils/openai_listwise_reranker.py#L1-L380 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/core/context_pruning.py#L1-L230 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/docs/pages/blogs/generating-50k-embeddings-with-gte.mdx#L1-L50 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/agents/assistant_wrapper.py#L1-L270 https://github.com/sweepai/sweep/blob/30a02a1ee570a64fbc23fdff1189451cf83526f4/sweepai/core/reflection_utils.py#L1-L168

---

Step 2: ⌨️ Coding

• [X] Modify sweepai/core/vector_db.py ✓ https://github.com/sweepai/sweep/commit/c9ce39092b6cca17029ab93acd4238753750ca55 Edit

  Modify sweepai/core/vector_db.py with contents:
  • In the openai_call_embedding function, wrap the existing code in a try/except block
  • In the except block, catch the voyageai.error.InvalidRequestError
  • If this specific error is caught and the batch size is greater than 1: - Log an error message indicating the token count was exceeded for the batch and it will be retried by splitting the batch in half - Calculate the max token count of the texts in the batch using the Tiktoken class from sweepai/utils/utils.py - Split the batch list in half - Recursively call openai_call_embedding on the left and right halves of the split batch - Concatenate the results of the recursive calls and return that
  • Else if any other exception occurs, raise it
  • Import the Tiktoken class from sweepai/utils/utils.py at the top of the file

--- 
+++ 
@@ -18,6 +18,26 @@
 from sweepai.utils.hash import hash_sha256
 from sweepai.utils.openai_proxy import get_embeddings_client
 from sweepai.utils.utils import Tiktoken
+
+import json
+import multiprocessing
+from typing import Generator
+
+import backoff
+import numpy as np
+import openai
+import requests
+from loguru import logger
+from redis import Redis
+from tqdm import tqdm
+import voyageai
+import boto3
+from botocore.exceptions import ClientError
+from voyageai import error as voyageai_error
+
+from sweepai.config.server import BATCH_SIZE, REDIS_URL, VOYAGE_API_AWS_ACCESS_KEY, VOYAGE_API_AWS_ENDPOINT_NAME, VOYAGE_API_AWS_REGION, VOYAGE_API_AWS_SECRET_KEY, VOYAGE_API_KEY, VOYAGE_API_USE_AWS
+from sweepai.utils.hash import hash_sha256
+from sweepai.utils.openai_proxy import get_embeddings_client

 # Now uses Voyage AI if available, with asymmetric embedding
 # CACHE_VERSION = "v2.0.04" + "-voyage" if VOYAGE_API_KEY else ""
@@ -144,13 +164,15 @@
     # and check the token count manually, but it requires an extra dependency. 
     try:
         return openai_call_embedding_router(batch, input_type)
-    except (voyageai_error.InvalidRequestError, ClientError) as e: # full error is botocore.errorfactory.ModelError: but I can't find it
-        if len(batch) > 1 and "Please lower the number of tokens in the batch." in str(e):
+    except voyageai_error.InvalidRequestError as e:
+        if len(batch) > 1:
             logger.error(f"Token count exceeded for batch: {max([tiktoken_client.count(text) for text in batch])} retrying by splitting batch in half.")
             mid = len(batch) // 2
-            left = openai_call_embedding(batch[:mid], input_type)
-            right = openai_call_embedding(batch[mid:], input_type)
-            return np.concatenate((left, right))
+            left_half = batch[:mid]
+            right_half = batch[mid:]
+            left_result = openai_call_embedding(left_half, input_type)
+            right_result = openai_call_embedding(right_half, input_type)
+            return np.concatenate((left_result, right_result))
         else:
             raise e
     except openai.BadRequestError as e:

• [X] Running GitHub Actions for sweepai/core/vector_db.py ✓ Edit

  Check sweepai/core/vector_db.py with contents:

Ran GitHub Actions for c9ce39092b6cca17029ab93acd4238753750ca55:
• Vercel Preview Comments: ✓

---

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/add_backoff_for_batch_sizes_in_the_vecto_9668b.

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.