search

WeblateOrg / weblate

Web based localization tool with tight version control integration.
https://weblate.org/
GNU General Public License v3.0
4.63k stars 1.02k forks source link

"Generate MO files" add-on appends \x04 character to every translation key #6756

Closed m-aciek closed 3 years ago

m-aciek commented 3 years ago

Describe the issue

After updating translation resource and committing to the repository with "Generate MO files", the MO file generated has "/x04" appended at the end of every translation key.

I already tried

Steps to reproduce the behavior

  1. Update translations in a Gettext project with "Generate MO files" add-on enabled
  2. Go to manage -> Commit, manage -> Push
  3. Load MO file with Gettext in the application
  4. See Gettext not finding a translation for any translation key and fallbacking to msgid
  5. Introspect loaded catalog to see that every translation key there has \x04 appended

Expected behavior

  1. See Gettext finding a translation for any translation key and not fallbacking to msgid
  2. Introspect loaded catalog to see that every translation key is in line with translation template

Screenshots

After pulling repository with Weblate commits and loading.

Zrzut ekranu 2021-11-5 o 10 16 09

After generating manually and loading.

% msgfmt locales/en/LC_MESSAGES/resource.po -o locales/en/LC_MESSAGES/resource.mo
% msgfmt -V    
msgfmt (GNU gettext-tools) 0.21
Zrzut ekranu 2021-11-5 o 10 11 10

Exception traceback

No response

How do you run Weblate?

Docker container

Weblate versions

Copying over versions from About Weblate page as I have user access only:

Weblate 4.8.1 Django 3.2.7 siphashc 2.1 translate-toolkit 3.4.1 lxml 4.6.3 Pillow 8.3.2 bleach 4.1.0 python-dateutil 2.8.2 social-auth-core 4.1.0 social-auth-app-django 5.0.0 django-crispy-forms 1.12.0 oauthlib 3.1.1 django-compressor 2.4.1 djangorestframework 3.12.4 django-filter 2.4.0 django-appconf 1.0.4 user-agents 2.2.0 filelock 3.0.12 setuptools 58.0.4 jellyfish 0.8.2 openpyxl 3.0.7 celery 5.1.2 kombu 5.1.0 translation-finder 2.10 weblate-language-data 2021.5 html2text 2020.1.16 pycairo 1.16.2 pygobject 3.38.0 diff-match-patch 20200713 requests 2.26.0 django-redis 5.0.0 hiredis 2.0.0 sentry_sdk 1.3.1 Cython 0.29.24 misaka 2.1.1 GitPython 3.1.24 borgbackup 1.1.17 pyparsing 2.4.7 pyahocorasick 1.4.2 python-redis-lock 3.7.0 Python 3.9.2 Git 2.30.2 psycopg2 2.9.1 psycopg2-binary 2.9.1 phply 1.2.5 chardet 4.0.0 ruamel.yaml 0.17.16 tesserocr 2.5.2 akismet 1.1 boto3 1.18.44 zeep 4.1.0 aeidon 1.9 iniparse 0.5 mysqlclient 2.0.3 git-svn 2.30.2 git-review 2.1.0 Redis server 5.0.7 PostgreSQL server 11.12

Weblate 4.6.1 was generating MO files correctly.

Weblate deploy checks

No response

Additional context

Potentially it's a bug in some Weblate's dependency, but I don't know the internals.

I use MacOSX and I have LF line endings.

nijel commented 3 years ago

How does your PO file look like? How is your translation component configured?

m-aciek commented 3 years ago

I'm not able to upload a PO file in the issue, so I'm inlining it's content:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2021-09-29 13:24+0200\n"
"PO-Revision-Date: 2021-11-05 07:43+0000\n"
"Last-Translator: Maciej Olko <maciej.olko@gmail.com>\n"
"Language-Team: English <https://domain/projects/"
"project/api/en/>\n"
"Language: en\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=n != 1;\n"
"X-Generator: Weblate 4.8.1\n"

msgid "categories.descriptions.special_personal_data"
msgstr ""
"This shortened."

msgid "categories.dialog.descriptions.paragraph1"
msgstr ""
"This shortened"

I'm using continuous localization flow with gettext monolingual PO files.

Please let me know if I should provide any more details.

m-aciek commented 3 years ago

Fact that may play a role here is that before the generation made by Weblate, I broke the MOs on main branch by doing search and replace in binary files, which corrupted the "magic numbers" in them.

nijel commented 3 years ago

Seems like e4bb7fdbaeb411747814307caaeb32f517984938 reverted the logic for conversion of monolingual PO files...

github-actions[bot] commented 3 years ago

Thank you for your report; the issue you have reported has just been fixed.