Losing indexes with a large number of translations

mateuszjaworskiwakacje commented 7 months ago

Hi - I've run into the problem that I'm getting errors with a lot of translations:

WARNING:root:No original text found for index 378
WARNING:root:No original text found for index 379
WARNING:root:No original text found for index 380
WARNING:root:No original text found for index 381
WARNING:root:No original text found for index 382
WARNING:root:No original text found for index 383
WARNING:root:No original text found for index 384
WARNING:root:No original text found for index 385
WARNING:root:No original text found for index 386
WARNING:root:No original text found for index 387
WARNING:root:No original text found for index 388
WARNING:root:No original text found for index 389
WARNING:root:No original text found for index 390
WARNING:root:No original text found for index 391
WARNING:root:No original text found for index 392
WARNING:root:No original text found for index 393
WARNING:root:No original text found for index 394
WARNING:root:No original text found for index 395
WARNING:root:No original text found for index 396
WARNING:root:No original text found for index 397
WARNING:root:No original text found for index 398
WARNING:root:No original text found for index 399

The result further is a translation turnout:

msgid "number"
msgstr "Bokning av plats för buss."

msgid "Booking date."
msgstr "Tillvaltvätt."

msgid "Booking data."
msgstr "Bokning av VIP-plats."

msgid "Number of people."
msgstr "VIP Access."

Additionally, it is not possible to run the program without the bulk parameter:

INFO:root:Discovered .po file: ../mysoft-test/locale/sv/LC_MESSAGES/django.po
INFO:root:Attempting to infer language for .po file: ../mysoft-test/locale/sv/LC_MESSAGES/django.po
INFO:root:Inferred language for .po file: ../mysoft-test/locale/sv/LC_MESSAGES/django.po as sv
INFO:root:Translating text 1/380 in file ../mysoft-test/locale/sv/LC_MESSAGES/django.po
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:root:Raw API response: 0: Det går inte att redigera denna bokning.
ERROR:root:Error processing file ../mysoft-test/locale/sv/LC_MESSAGES/django.po: cannot unpack non-iterable int object

steglicd commented 7 months ago

Exact same issue here, tried installation from git and pip.

pescheck-bram commented 3 months ago

Can you deliver some files that we can test it?

In version 0.1.12 it is possible to run it without the --bulk parameter

pescheck-bram commented 2 months ago

Closing for now please let us know if this is still an issue, later and we will reopen it.

pescheck-bram commented 1 month ago

Okay this should be all fixed with release 0.2.5:

Major Functional Updates

Enhanced Batch Processing: Improved precision in batch functions, adjusting batch size for optimal performance and accuracy.
Retry Mechanism Introduced: Implemented retries across multiple scenarios to enhance reliability, including retries specifically for translation and parsing issues.
AI Translation Enhancements: Enhanced context handling for AI translations, including retries when certain explanatory phrases are detected.
Contextual Awareness for AI: Updated the AI's context to explicitly recognize its role as a translator, ensuring more accurate and context-aware translations.

Technical Details

Updated Batch Size: Batch size adjusted from 50 to 40 to better manage API limits and improve processing efficiency.
JSON Array Parsing: Introduced a structured JSON array format for parsing, improving data handling and ensuring consistent translation output.
Batch and Single Translation Improvements: Refined the bulk and single translation methods to better handle chunk processing, including retries for individual translations when bulk translations fail.
Improved Error Handling: Enhanced error handling and logging to provide more detailed information about translation failures and retries.
Translation Context Improvements: Updated the translation prompt context to provide clearer and more direct instructions for the AI, reducing the likelihood of errors and unnecessary explanations.
Enhanced PO File Handling: Added methods to manage and update PO file entries more effectively, including handling fuzzy translations and validating translation quality.

Example Translation Context:

f"Translate the following list of texts from English to {target_language}. "
"Provide only the translations in a JSON array format, maintaining the original order. "
"Each translation should be concise and direct, without explanations or additional context. "
"Keep special characters, placeholders, and formatting intact. "
"If a term should not be translated (like 'URL' or technical terms), keep it as is. "
"Example format: [\"Translation 1\", \"Translation 2\", ...]\n\n"
"Texts to translate:\n"

Additional Changes:

PO File Management: Introduced new utilities to disable fuzzy translations and infer file languages based on folder structure.
Unit Test Enhancements: Updated unit tests to reflect the new bulk and single translation logic, including improved mock handling and test coverage.
Dependency Updates: Updated dependencies to include the latest versions of openai and tenacity libraries, ensuring compatibility with the latest API features.

This update brings significant improvements in precision, reliability, and overall functionality of the translation system, ensuring smoother operations and more accurate results.

https://github.com/user-attachments/assets/872a01d5-e036-4678-bb68-92d79ace21e2

pescheckit / python-gpt-po