search

WeblateOrg / weblate

Web based localization tool with tight version control integration.
https://weblate.org/
GNU General Public License v3.0
4.63k stars 1.02k forks source link

Component language gets deleted after upload new language source file #4861

Closed adriandiazgar closed 3 years ago

adriandiazgar commented 4 years ago

Describe the bug

The custom language zh_CN disappears when uploading again the source file with a replacing mechanism.

To Reproduce the bug

Steps to reproduce the behavior:

  1. Make sure language zh_CN exists on Weblate, if not create it.
  2. Create a project "mobile-project"
  3. Add a new component from scratch android-app language English (en), file format Android Resource
  4. Add translations for zh_CN XML file based
  5. Check that the language is present in the component (for example via de API: api/translations/mobile-project/android-app/zh_CN/)
  6. Re-upload the strings.xml file to the Project, so new strings get populated using the replace mechanism.
  7. Check that the language is not present anymore in the component (for example via de API: api/translations/mobile-project/android-app/zh_CN/) --> NOT FOUND

Expected behavior

When on point number 4 a new strings.xml get uploaded, only add the new strings to the different languages but not get rid of any of them.

Server configuration and status

Weblate installation: Hosted service

weblate list_versions

On Git checkout:

./manage.py list_versions

` * Weblate: 4.3

nijel commented 4 years ago

Do you get any errors in Weblate logs?

Are the files removed from the repository? Check content of DATA_DIR/vcs/mobile-project/android-app on your Weblate server.

github-actions[bot] commented 4 years ago

This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but purchasing the support subscription is not only more responsible and faster for your business but also makes Weblate stronger. In case your question is already answered, making a donation is the right way to say thank you!

adriandiazgar commented 4 years ago

Hi, @nijel thanks for your quick response.

Once the new strings.xml file is uploaded. The zn_CN.xml file still remains in the server local repo but Weblate can't retrieve it from the API neither the UI.

Thanks in advance, Adrián

nijel commented 4 years ago

Does it appear under different name? Or there are no translations shown by Weblate? Anyway the Weblate logs should indicate why it has been removed...

adriandiazgar commented 4 years ago

I think it gets moved to zn_Hans which is an alias for zn_cn (lowercase) however the language disappearing (getting moved) is zn_CN (CN Uppercase).

nijel commented 4 years ago

I think it gets moved to zn_Hans

Can you take a look? And take a look at the actual files?

Before c355570449a (included in the 4.3.2 release) there could have been some inconsistencies how newly created files and existing files were parsed and this could lead to different naming of the translations in the UI. Still, the actual files should stay there and should not be removed...

jojurajan commented 3 years ago

Hi Nijel, We found the actual issue as to why this was happening. Even if we provide the language code for creating a translation, it takes the filepath of generated XML file and due to the conversions happening at https://github.com/WeblateOrg/weblate/blob/8f6e12d967cd48882abbd36e4b72cfec06811261/weblate/lang/models.py#L170 and https://github.com/WeblateOrg/weblate/blob/8f6e12d967cd48882abbd36e4b72cfec06811261/weblate/lang/models.py#L149 the language code gets set as zh_Hans instead of zh_CN which was provided in the create request.

The issue can be reproduced by adding any chinese locale that has a traditional/simplified version. e.g. zh_HK, zh_CN, zh_SG.

nijel commented 3 years ago

The fuzzy_get first tries exact matches, so in case the language exists it is used. In case no match is found, it falls back to trying aliases and other heuristics. See https://docs.weblate.org/en/latest/admin/languages.html#parsing-language-codes for more detailed explanation.

jojurajan commented 3 years ago

The language provided to fuzzy_get is zh-rCN which fetches zh_Hans, instead of zh_CN which was provided for the translation creation via API. I suppose this happens because zh_CN and zh_Hans map to the same Android filename zh-rCN.xml.

I hope this clarifies the issue that we are facing after using latest 4.3.2 Docker image which also includes this commit -> https://github.com/WeblateOrg/weblate/commit/c355570449acf5766858ba78d07bfc729ce2ed51.

nijel commented 3 years ago

Thanks, now I can reproduce this.

Anyway, what is wrong on using zh_Hans inside Weblate?

jojurajan commented 3 years ago

We are migrating data from an existing platform and we need to use the old language codes for backward compatibility.

github-actions[bot] commented 3 years ago

Thank you for your report, the issue you have reported has just been fixed.

nijel commented 3 years ago

We are migrating data from an existing platform and we need to use the old language codes for backward compatibility.

You can use anything in the file names, it will be just mapped for Weblate. It's better to stick with Weblate defined languages on Weblate side for correct behaviour of some features (for example machine translation using third-party services).