Multi API solution? - Githubissues

haitranvua commented 1 year ago

I am using your source code and it is working great. However, I am using a free OpenAI API, I am often limited in some uses (because the .srt file is quite large, 500~600 subtitle lines). I came up with this I don't know if it's stupid, but if I have multiple free APIs, is there a way to edit the source code so that each time it is limited, the function will automatically change to the next API.

Cerlancism commented 1 year ago

Can I confirm, provide error stack text or screenshot on what is the error thrown when the limits hits till it crashes?

I think it is possible to write a command line script, such as in bash to do it, but I would not recommend:

Remove OPENAI_API_KEY from .env completely
Set OPENAI_API_RPM=3 in .env
Store an array of API keys in variable
export OPENAI_API_KEY
Run the command, everytime it crashes with non-zero exit code change to the next key from the array variable
Rerun the command

Code example (not tested): https://chat.openai.com/share/04313a49-1783-4901-a244-26ed0ed47f4a

haitranvua commented 1 year ago

This is the error

[Translator] BatchSize decrease 100 -> 10 SizeThreshold 10
[Translator] Estimated Usage - Tokens: 4426 $ 0.009 Wasted: 4426 $ 0.009 100% Rate: 4009.66 TPM 0 RPM
1. Bienvenue à une autre leçon, la troisième de la série sur les effets volumétriques.
2. Comme toutes les leçons du dernier module, celle-ci sera très, très pratique.
3. Nous aborderons le scénario du coucher de soleil, qui, à un niveau basique, ne sera pas très différent du premier coucher de soleil que nous avons fait.
4. Cependant, il y aura une petite variation liée aux effets volumétriques ici.
5. Dans la leçon précédente, nous avons littéralement peint avec différents brouillards volumétriques, mais maintenant nous allons tout faire en 3D complète.
6. Nous apprendrons, par exemple, comment créer ces effets volumétriques près du sol et comment en introduire de nouveaux qui ne soient pas uniformes partout.
7. Nous apprendrons comment tout configurer et vous expliquerons comment le faire sans nuire à la scène.
[Translator] Lines count mismatch 10 7
[Translator] batch [
  '1. Welcome to another lesson, the third one in the volumetric effect series.',
  '2. Like all lessons from the last module, it will be very, very practical.',
  "3. We'll approach the sunset scenario, which on a basic level won't be too different from",
  '4. the first sunset we did.',
  '5. However, there will be a small twist related to volumetrics here.',
  '6. In the previous lesson, we literally painted with different volumetric fogs, but now we',
  '7. are going to do it all in full 3D.',
  '8. We will learn, for example, how to make that volumetrics close to the ground and how to',
  '9. introduce some new ones that makes it not uniform everywhere.',
  '10. We will learn how to set it all up and tell you how to do it without killing the scene.'
]
[Translator] transformed [
  '1. Bienvenue à une autre leçon, la troisième de la série sur les effets volumétriques.',
  '2. Comme toutes les leçons du dernier module, celle-ci sera très, très pratique.',
  '3. Nous aborderons le scénario du coucher de soleil, qui, à un niveau basique, ne sera pas très différent du premier coucher de soleil que nous avons fait.',
  '4. Cependant, il y aura une petite variation liée aux effets volumétriques ici.',
  '5. Dans la leçon précédente, nous avons littéralement peint avec différents brouillards volumétriques, mais maintenant nous allons tout faire en 3D complète.',
  '6. Nous apprendrons, par exemple, comment créer ces effets volumétriques près du sol et comment en introduire de nouveaux qui ne soient pas uniformes partout.',
  '7. Nous apprendrons comment tout configurer et vous expliquerons comment le faire sans nuire à la scène.'
]
[Translator] Single line mode
1. Bienvenue à une autre leçon, la troisième de la série sur les effets volumétriques.
1 "Welcome to another lesson, the third one in the volumetric effect series." -> "Bienvenue à une autre leçon, la troisième de la série sur les effets volumétriques."
[Error_TranslationPrompt] 2023-09-08T13:25:48.219Z Status 429 Error Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method. {
  message: 'Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.',
  type: 'requests',
  param: null,
  code: 'rate_limit_exceeded'
}
[Error_TranslationPrompt] Retries 1 Delay 1000
[Cooldown] ChatGPTAPI 48136 ms
[Error_TranslationPrompt] 2023-09-08T13:26:40.589Z Status 429 Error Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method. {
  message: 'Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.',
  type: 'requests',
  param: null,
  code: 'rate_limit_exceeded'
}
[Error_TranslationPrompt] Retries 2 Delay 8000
[Error_TranslationPrompt] 2023-09-08T13:26:51.981Z Status 429 Error Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method. {
  message: 'Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.',
  type: 'requests',
  param: null,
  code: 'rate_limit_exceeded'
}
[Error_TranslationPrompt] Retries 3 Delay 27000
[Error_TranslationPrompt] [openaiRetryWrapper] Max Retries Reached 2023-09-08T13:27:19.004Z {
  currentTry: 4,
  error: RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.
      at Function.generate (file:///content/chatgpt-subtitle-translator/node_modules/openai/error.mjs:49:14)
      at OpenAI.makeStatusError (file:///content/chatgpt-subtitle-translator/node_modules/openai/core.mjs:263:21)
      at OpenAI.makeRequest (file:///content/chatgpt-subtitle-translator/node_modules/openai/core.mjs:303:24)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async file:///content/chatgpt-subtitle-translator/src/translator.mjs:112:40
      at async retryWrapper (file:///content/chatgpt-subtitle-translator/src/helpers.mjs:35:20)
      at async openaiRetryWrapper (file:///content/chatgpt-subtitle-translator/src/openai.mjs:52:12)
      at async Translator.translatePrompt (file:///content/chatgpt-subtitle-translator/src/translator.mjs:89:26)
      at async Translator.translateSingle (file:///content/chatgpt-subtitle-translator/src/translator.mjs:180:28)
      at async Translator.translateLines (file:///content/chatgpt-subtitle-translator/src/translator.mjs:235:21) {
    status: 429,
    headers: {
      'alt-svc': 'h3=":443"; ma=86400',
      'cf-cache-status': 'DYNAMIC',
      'cf-ray': '8037868e8ed544fc-ATL',
      connection: 'keep-alive',
      'content-length': '522',
      'content-type': 'application/json; charset=utf-8',
      date: 'Fri, 08 Sep 2023 13:26:51 GMT',
      server: 'cloudflare',
      'strict-transport-security': 'max-age=15724800; includeSubDomains',
      vary: 'Origin',
      'x-ratelimit-limit-requests': '200',
      'x-ratelimit-limit-tokens': '40000',
      'x-ratelimit-remaining-requests': '0',
      'x-ratelimit-remaining-tokens': '39915',
      'x-ratelimit-reset-requests': '23h54m7.966s',
      'x-ratelimit-reset-tokens': '127ms',
      'x-request-id': '5e38d9d5772fb615ba01660b0bc160cd'
    },
    error: {
      message: 'Rate limit reached for default-gpt-3.5-turbo in organization org-FIX73AJ9J6dpxwYWUEN8QT6k on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.',
      type: 'requests',
      param: null,
      code: 'rate_limit_exceeded'
    },
    code: 'rate_limit_exceeded',
    param: null,
    type: 'requests'
  },
  maxRetries: 3
}

this is command i used !node cli/translator.mjs --stream --temperature 0 --file "test.srt" --to French --no-use-moderator

This is my .srt file https://we.tl/t-o038s707vP

Cerlancism commented 1 year ago

Seems it is getting the line mismatching pretty badly.

Allow line mismatching and fix misalignments manually --no-line-matching
Use the older model --model "gpt-3.5-turbo-0301"

haitranvua commented 1 year ago

"Thank you very much for your support, changing the model has helped me no longer be limited. However, I often encounter errors that make the translation incomplete. Can you tell me the reason why?" p.s "I have noticed that after a batch of translations is completed, in the next batch, errors occur intermittently."

``60 "We've got a dark scene here and now we want to introduce some artificial unrelating lighting here." -> "Chúng tôi có một cảnh tối ở đây và bây giờ chúng tôi muốn giới thiệu một số ánh sáng không tự nhiên" [Translator] Label mismatch 61 undefined 61 "Well, let's start by saying that I would like to add such a fairly solid corona sphere here." -> "[Flagged][Model] Well, let's start by saying that I would like to add such a fairly solid corona sphere here. -> "

[Translator] Label mismatch 62 undefined 62 "Somewhere here on the right-hand side." -> "[Flagged][Model] Somewhere here on the right-hand side. -> "

[Translator] Label mismatch 63 undefined 63 "We can see how strongly it casts light already, of course, and we need to adjust all this." -> "[Flagged][Model] We can see how strongly it casts light already, of course, and we need to adjust all this. -> "

[Translator] Label mismatch 64 undefined 64 "We will definitely reduce the intensity." -> "[Flagged][Model] We will definitely reduce the intensity. -> " `

Cerlancism commented 1 year ago

That is the issue you have fix manually especially when using --no-line-matching

One way to fix this:

Cross-check manually. I use Google Sheets, paste in the original .srt (copied from Aegisub, which writes CSV to the clipboard), paste in the output progress .csv, and compare it with another translation service, such as the =GOOGLETRANSLATE formula

ishaan-jaff commented 1 year ago

@Cerlancism @haitranvua

i'm the maintainer of LiteLLM we allow you to maximize your throughput/increase rate limits - load balance between multiple deployments (Azure, OpenAI) I believe litellm can be helpful here - and i'd love your feedback if we're missing something

Here's how to use it Docs: https://docs.litellm.ai/docs/routing

from litellm import Router

model_list = [{ # list of model deployments 
    "model_name": "gpt-3.5-turbo", # model alias 
    "litellm_params": { # params for litellm completion/embedding call 
        "model": "azure/chatgpt-v-2", # actual model name
        "api_key": os.getenv("AZURE_API_KEY"),
        "api_version": os.getenv("AZURE_API_VERSION"),
        "api_base": os.getenv("AZURE_API_BASE")
    }
}, {
    "model_name": "gpt-3.5-turbo", 
    "litellm_params": { # params for litellm completion/embedding call 
        "model": "azure/chatgpt-functioncalling", 
        "api_key": os.getenv("AZURE_API_KEY"),
        "api_version": os.getenv("AZURE_API_VERSION"),
        "api_base": os.getenv("AZURE_API_BASE")
    }
}, {
    "model_name": "gpt-3.5-turbo", 
    "litellm_params": { # params for litellm completion/embedding call 
        "model": "vllm/TheBloke/Marcoroni-70B-v1-AWQ", 
        "api_key": os.getenv("OPENAI_API_KEY"),
    }
}]

router = Router(model_list=model_list)

# openai.ChatCompletion.create replacement
response = router.completion(model="gpt-3.5-turbo", 
                messages=[{"role": "user", "content": "Hey, how's it going?"}])

print(response)

Cerlancism / chatgpt-subtitle-translator

Multi API solution? #10