Auto Translation using Google Translate V3 doesn't preserve New Line characters

ChiragMoradiya commented 1 year ago

Describe the issue

When a String contains a new line character (\n). It's "Google Translate V3" auto-translation, replaces new-line character with a single space.

But, AWS Translation preserves new-line characters properly.

See Screenshots for the reference.

I already tried

[X] I've read and searched the documentation.
[X] I've searched for similar issues in this repository.

Steps to reproduce the behavior

Given a string's english value is Hello\nWorld
Then it's auto translation in Hindi Language, using "Google Translate V3" is हैलो वर्ल्ड (ISSUE)
And It's auto translation in Hindi Language, using AWS is नमस्ते\nवर्ल्ड

Expected behavior

Auto Translation in Hindi, using "Google Translate V3" should be हैलो\nवर्ल्ड

Screenshots

Google Translate V3 API:

AWS API:

Exception traceback

No response

How do you run Weblate?

Docker container

Weblate versions

Weblate: 4.17
Django: 4.2
siphashc: 2.1
translate-toolkit: 3.8.6
lxml: 4.9.2
Pillow: 9.5.0
nh3: 0.2.11
python-dateutil: 2.8.2
social-auth-core: 4.4.2
social-auth-app-django: 5.2.0
django-crispy-forms: 2.0
oauthlib: 3.2.2
django-compressor: 4.3.1
djangorestframework: 3.14.0
django-filter: 23.1
django-appconf: 1.0.5
user-agents: 2.2.0
filelock: 3.11.0
rapidfuzz: 3.0.0
openpyxl: 3.1.2
celery: 5.2.7
django-celery-beat: 2.5.0
kombu: 5.2.4
translation-finder: 2.15
weblate-language-data: 2023.4
html2text: 2020.1.16
pycairo: 1.23.0
pygobject: 3.44.1
diff-match-patch: 20200713
requests: 2.28.2
django-redis: 5.2.0
hiredis: 2.2.2
sentry_sdk: 1.21.1
Cython: 0.29.34
misaka: 2.1.1
GitPython: 3.1.31
borgbackup: 1.2.4
pyparsing: 3.0.9
pyahocorasick: 2.0.0
python-redis-lock: 4.0.0
charset-normalizer: 3.1.0
Python: 3.11.3
Git: 2.30.2
psycopg2: 2.9.6
psycopg2-binary: 2.9.6
phply: 1.2.6
ruamel.yaml: 0.17.21
tesserocr: 2.6.0
boto3: 1.26.124
zeep: 4.2.1
aeidon: 1.12
iniparse: 0.5
mysqlclient: 2.1.1
Mercurial: 6.4.2
git-svn: 2.30.2
git-review: 2.3.1
Redis server: 7.0.11
PostgreSQL server: 15.3
Database backends: django.db.backends.postgresql
Cache backends: default:RedisCache, avatar:FileBasedCache
Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.mailgun.org
OS encoding: filesystem=utf-8, default=utf-8
Celery: redis://cache:6379/1, redis://cache:6379/1, regular
Platform: Linux 5.19.0-1024-aws (x86_64)

Weblate deploy checks

Output of `docker compose exec --user weblate weblate weblate check --deploy`

System check identified some issues:

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-4.17/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-4.17/admin/backup.html
?: (weblate.I031) New Weblate version is available, please upgrade to 5.0.2.
    HINT: https://docs.weblate.org/en/weblate-4.17/admin/upgrade.html

System check identified 3 issues (1 silenced).

Additional context

There seems an issue in REST API invocation in https://github.com/WeblateOrg/weblate/blob/c7915cc0954da39169621ce3bbfa19ba189583fe/weblate/machinery/googlev3.py#L63C16-L63C16

It sends whole String as an element in the Array. e.g. ["Hello\nWorld"].

If it sends request as multiple array elements, split by new-line characters. e.g. ["Hello","world"]. And then join response elements back by new-line character, then this issue might be resolved.

NOTE: There is an additional cavity, if the request array contains any blank string; then Google treats this as an invalid request. So, such elements should be removed from the request element.

nijel commented 1 year ago

Splitting the text needs to be evaluated as well as it might negatively impact translation if Google treats segments as independent. For example, in Czech this should be translated as "Ahoj\nsvěte" while independent words would be translated as Ahoj, svět.

Meanwhile, I've added highlighting of whitespace in machine translation results in https://github.com/WeblateOrg/weblate/pull/10147, so that it is clearly visible.

github-actions[bot] commented 1 year ago

This issue has been put aside. It is currently unclear if it will ever be implemented as it seems to cover too narrow of a use case or doesn't seem to fit into Weblate.

Please try to clarify the use case or consider proposing something more generic to make it useful to more users.

ChiragMoradiya commented 1 year ago

So, it looks like this should be reported as a Bug to Google Transtate API service, instead here.

nijel commented 1 year ago

I don't think it's realistic to expect machine translation to always keep newlines at the right place.

ChiragMoradiya commented 1 year ago

I agree it won't always keep. But, here the issue is that, it always removes. It's never preserved.

We have added many mark-down texts as a Weblate Strings. And by not preserving new-line characters, their formatting gets screwed with auto-translation.

pickfire commented 11 months ago

How we did it in another project https://github.com/arvin-pantas/django-autotranslate/commit/85f0d8d6567070411b8abefa0c32391f2f52fb47

nijel commented 11 months ago

@pickfire Thanks, that is useful!

WeblateOrg / weblate