OpenAI MachineTranslationError: Could not parse assistant reply

erikhugintech commented 4 weeks ago

Describe the issue

When attempting to OpenAI translate an entire component, I get "MachineTranslationError: Could not parse assistant reply".

A sample log:

celery-translate stderr | [2024-06-07 10:19:28,345: ERROR/ForkPoolWorker-31] stasism-applications/dodge-dare/pt_PT: failed automatic translation: Could not parse assistant reply
celery-translate stderr | [2024-06-07 10:19:28,345: ERROR/11392] stasism-applications/dodge-dare/pt_PT: failed automatic translation: Could not parse assistant reply
celery-translate stderr | [2024-06-07 10:19:28,344: ERROR/ForkPoolWorker-31] Could not fetch translations from OpenAI
celery-translate stderr | [2024-06-07 10:19:28,344: ERROR/11392] Could not fetch translations from OpenAI
celery-translate stderr | [2024-06-07 10:19:28,343: WARNING/ForkPoolWorker-31] Machinery error: MachineTranslationError: Could not parse assistant reply
celery-translate stderr | [2024-06-07 10:19:28,343: WARNING/11392] Machinery error: MachineTranslationError: Could not parse assistant reply
celery-translate stderr | Sair
celery-translate stderr | ==WEBLATE_PART==
celery-translate stderr | Bem Jogado!
celery-translate stderr | ==WEBLATE_PART==
celery-translate stderr | Quantos golpes precisas levar antes de perder
celery-translate stderr | ==WEBLATE_PART==
celery-translate stderr | [2024-06-07 10:19:28,343: WARNING/ForkPoolWorker-31] Failed to parse assistant reply: Vidas Iniciais
celery-translate stderr | Sair
celery-translate stderr | ==WEBLATE_PART==
celery-translate stderr | Bem Jogado!
celery-translate stderr | ==WEBLATE_PART==
celery-translate stderr | Quantos golpes precisas levar antes de perder
celery-translate stderr | ==WEBLATE_PART==
celery-translate stderr | [2024-06-07 10:19:28,343: WARNING/11392] Failed to parse assistant reply: Vidas Iniciais

I already tried

[X] I've read and searched the documentation.
[X] I've searched for similar filed issues in this repository.

Steps to reproduce the behavior

Go to Product/Component/Portuguese (Brazil)
Select Tools/Automatic Translation
Mode: Add as translation
Search filter: Untranslated Strings
Source: Machine Translation
OpenAI (preconfigured and working fine for most languages/components)
Click Apply

Expected behavior

Expected that parsing should succeed.
Expected that the parsing failure should provide a detailed description of where parsing failed and why.

Screenshots

No response

Exception traceback

No response

How do you run Weblate?

Docker container

Weblate versions

Weblate: 5.5.5
Django: 5.0.6
siphashc: 2.4.1
translate-toolkit: 3.13.0
lxml: 5.2.2
pillow: 10.3.0
nh3: 0.2.17
python-dateutil: 2.9.0.post0
social-auth-core: 4.5.4
social-auth-app-django: 5.4.1
django-crispy-forms: 2.1
oauthlib: 3.2.2
django-compressor: 4.4
djangorestframework: 3.15.1
django-filter: 24.2
django-appconf: 1.0.6
user-agents: 2.2.0
filelock: 3.14.0
rapidfuzz: 3.9.0
openpyxl: 3.1.2
celery: 5.4.0
django-celery-beat: 2.6.0
kombu: 5.3.7
translation-finder: 2.16
weblate-language-data: 2024.5
html2text: 2024.2.26
pycairo: 1.26.0
PyGObject: 3.48.2
diff-match-patch: 20230430
requests: 2.31.0
django-redis: 5.4.0
hiredis: 2.3.2
sentry-sdk: 1.45.0
Cython: 3.0.10
misaka: 2.1.1
GitPython: 3.1.43
borgbackup: 1.2.8
pyparsing: 3.1.2
ahocorasick_rs: 0.22.0
python-redis-lock: 4.0.0
charset-normalizer: 3.3.2
Python: 3.12.3
Git: 2.39.2
psycopg: 3.1.19
psycopg-binary: 3.1.19
phply: 1.2.6
ruamel.yaml: 0.18.6
tesserocr: 2.7.0
boto3: 1.34.104
zeep: 4.2.1
aeidon: 1.14.1
iniparse: 0.5
mysqlclient: 2.2.4
Mercurial: 6.7.3
git-svn: 2.39.2
git-review: 2.4.0
PostgreSQL server: 16.2
Database backends: django.db.backends.postgresql
PostgreSQL implementation: psycopg3 (binary)
Cache backends: default:RedisCache, avatar:FileBasedCache
Email setup: django.core.mail.backends.smtp.EmailBackend: email-smtp.eu-west-1.amazonaws.com
OS encoding: filesystem=utf-8, default=utf-8
Celery: redis://ops-redis.j8y5uz.0001.euw1.cache.amazonaws.com:6379/1, redis://ops-redis.j8y5uz.0001.euw1.cache.amazonaws.com:6379/1, regular
Platform: Linux 4.14.343-260.564.amzn2.x86_64 (x86_64)

Weblate deploy checks

SystemCheckError: System check identified some issues:

CRITICALS:
?: (weblate.E003) Cannot send e-mail (timed out), please check EMAIL_* settings.
        HINT: https://docs.weblate.org/en/weblate-5.5.5/admin/install.html#out-mail

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-5.5.5/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-5.5.5/admin/backup.html

Additional context

The translated parts in the log seem to correspond to the final strings in the JSON file used as a source:

    "DODGEDARE_SCENARIOSCOMPLETEDTEXT": "You completed:",
    "DODGEDARE_STARTINGLIVES_TEXT": "Starting Lives",
    "DODGEDARE_STARTINGLIVES_TEXTTOOLTIP": "How many times you need to be hit before you lose",
    "DODGEDARE_WELLDONETEXT": "Well Done!",
    "HT_EXIT": "Exit"
}

nijel commented 3 weeks ago

Weblate does this to translate more strings at once. It appears that OpenAI omitted some strings from the response.

erikhugintech commented 3 weeks ago

@nijel Is it possible to discard the problematic translations, but keep the rest? Currently, one such error in my 600 string initial automated translation prevents the remaining correctly translated strings from being saved.

nijel commented 3 weeks ago

Yes, b4d31853d2748dcb58813037163fb56219a57e64 will do that.

nijel commented 2 weeks ago

In the long term, it might be better to use JSON mode instead of this simple joining of strings. Example can be found at https://medium.com/@sean37gl/how-to-leverage-chatgpts-new-json-mode-to-build-better-user-experiences-db8f6963f608

WeblateOrg / weblate