Initializing a component from a .zip file does not respect the template file

luebbe commented 3 years ago

To Reproduce the issue

Create a component from the attached .zip file pot_not_honored.zip
Set pot_not_honored.pot as the source template for new translations (why isn't it suggested automatically? The .po file(s) are detected)
When the component is created, go to "English" translation, which shows: two string four words ✔️
Go to "German" translation, which shows: one string two words, 100% translated ❌ Expected: two strings, four words as in English, 50% translated.
Go to "English" translation Files -> upload translation Extract pot_not_honored.pot from the zip and upload it again with "update source strings"
Go to "German" translation, which now shows: 50% translated, two string four words ✔️

IMHO the .pot file should always be the reference for the translations. It may well be that .po files haven't been updated for a while and thus are missing msgids or containing outdated msgids. The attached .zip only tests the first case, not the second.

nijel commented 3 years ago

is a bug in the translation finder, I will fix it.
is expected, see https://docs.weblate.org/en/latest/faq.html#why-does-weblate-still-show-old-translation-strings-when-i-ve-updated-the-template

github-actions[bot] commented 3 years ago

Thank you for your report; the issue you have reported has just been fixed.

In case you see a problem with the fix, please comment on this issue.
In case you see a similar problem, please open a separate issue.
If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

luebbe commented 3 years ago

2. is a bug in the translation finder, I will fix it.
Fine 👍 Thank you.
3. is expected, see https://docs.weblate.org/en/latest/faq.html#why-does-weblate-still-show-old-translation-strings-when-i-ve-updated-the-template
That's why I put a green check-mark next to 3. The real problem is:

Go to "German" translation, which shows: one string two words, 100% translated ❌ Expected: two strings, four words as in English, 50% translated.

Or did Github screw up the numbering when you quoted the reproduction steps and you are referring to my Step 4?

Is this really good behaviour? It doesn't make sense to me. We're starting with a blank sheet of paper here. Why whould I upload a .pot file in the .zip when its contents are ignored by Weblate? My expectation is that, when a .pot file (or any template FWIW) is provided together with the .po files, msgmerge should be run against this .pot file. Uploading the .pot file again leads to the expected behaviour.

So this means that I should activate the: https://docs.weblate.org/en/latest/admin/addons.html#addon-weblate-gettext-msgmerge plugin for all gettext based components?

But IIRC this can only be done after the component has been created. Aren't we running into a chicken/egg problem here? I create a project from a .zip that contains .pot and .po files. The add-on description says that it will be triggered, when upstream changes are detected. But the data is already in Weblate. There are no upstream changes, the .pot contains the strings to translate but some .po files don't match the .pot (yet). So the add-on is not going to be triggered and the translators think that everything is fine, while in fact it isn't.

nijel commented 3 years ago

Yes, GitHub changed rendering to ordered list and messed up numbers.

Weblate has no clue whether the difference between the PO files are intentional or not. We try not to break things, and enabling msgmerge by default could break things if it is not intended.

The addon is triggered on installation as well, so everything should be fine.

luebbe commented 3 years ago

OK, I understand your reasoning. In order not to have too many subjects in this thread I'll leave the addon aside for now.

My goal is to make setting up a weblate translation project from a developer's working copy and interacting with it as painless as possible. The Weblate UI should just be used for translation. No configuration should be done via the UI. Everything should done via a (homebrew) API client. Right now I don't see how I coud find the id of the msgfmt addon via the API.

I'm trying to understand what is going on, when a component is initialized from a .zip file:

Stage 1

The .zip file is uploaded
The .po files are parsed and the translation pairs are stored in the database
The .pot file is just used to detect that there is a template and it is not parsed at all

Is this correct?

Result:

The uploaded po files are untouched, the state of the translations vs the source language is unknown and possibly inconsistent, but no translation is lost.

Stage 2

Upload the .pot file again using the option "update sources". I assume that now msgmerge is run under the hood and the language pairs are synchronized with the .pot file.

Result:

The uploaded po files are touched and the state of the translations vs the source language is known and consistent. Possibly old, but now obsolete, translations are lost.

The result after Stage 2 is the desired state after Stage 1 for me. I'm working with translations for more than 25 years now, using different web based, desktop, proprietary and open source translation tools and it is the first time that I come across this behaviour. If you say that the default is not to touch anything, I'd turn this into a feature request:

"Add an option to the component initialization to run msgmerge immediately".

Basically combine Step 1 and Step 2, so the option could be named "update source", like in file upload or API POST /api/translations/(string: project)/(string: component)/(string: language)/file/ method – Upload method (... source ...)

What do you think about this?

nijel commented 3 years ago

I understand your point - it makes perfect sense if you use xgettext to automatically update PO files. On the other side, the PO files are often used just a storage for the translations as that is widely used format and there is no xgettext involved. The POT file is used only for starting new translations and further management of the strings happens by other means.

In case you are expencting this behavior for all PO files, you can add the addon to DEFAULT_ADDONS - it will make it automatically install on all newly created components.

luebbe commented 3 years ago

Yes, you're right. The translations in the Open Source projects that I'm involved in (e.g. TortoiseSVN, PyScripter) as well as the translations at work are all centered around xgettext or other extraction tools that create .pot files from different sources. This is why this behaviour is so cruicial to me. The pot files are the reference :)

I'll try the default_addons. Thank you for the hint.

luebbe commented 3 years ago

Sorry, but it still doesn't work.

The addon is added to the default add ons with the options previous:true, fuzzy:true, no_location:false. A source of error was here that the JSON API shows booleans in all lowercase and the DEFAULT_ADDONS configuration requires them to be capitalized.

But after that was solved it looks like we're good to go:

When I go to weblate/api/addons, the list is empty ✔️ I go through the steps 1..4 of the initial ticket description with identical results ❌ I go to weblate/api/addons, the addon is shown for one component ✔️

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "component": "https://weblate/api/components/test-lo/pot_not_honored/",
            "name": "weblate.gettext.msgmerge",
            "id": 21,
            "configuration": {
                "configuration": {
                    "previous": true,
                    "no_location": false,
                    "fuzzy": true
                }
            },
            "url": "https://weblate/api/addons/21/"
        }
    ]
}

Notice that true,false,true are reflected as in the DEFAULT_ADDONS configuration.

When I look at it again, I have the impression that our admin has made an error when adding the configuration. I see two nested configuraitions

Now I go to the addon configuration of the "pot_not_honored" component and see the following:

grafik

Where are the true,false,true settings gone?

So AFAICT ~~two~~ one things are wrong:

msgmerge doesn't run when a component is initialized from a zip file, even though the plugin is activated for the component when the component is created ~~2. the plugin settings on the gui and in the api don't match~~

nijel commented 3 years ago

This looks wrong, there should not be two "configuration" blocks, the keys should be part of the first one:

            "configuration": {
                "configuration": {
                    "previous": true,
                    "no_location": false,
                    "fuzzy": true
                }
            },

msgmerge doesn't run when a component is initialized from a zip file, even though the plugin is activated for the component when the component is created

Any errors in the logs? Any alerts on the newly created component indicating a problem?

luebbe commented 3 years ago

This looks wrong, there should not be two "configuration" blocks, the keys should be part of the first one:
            "configuration": {
                "configuration": {
                    "previous": true,
                    "no_location": false,
                    "fuzzy": true
                }
            },

Yes, that's what I noticed yesterday and wrote above. I asked our admin to fix it, which he just did:

Went through steps 1-4 again with the same wrong result.

The add-on configuration looks good to me now:

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "component": "https://weblate/api/components/test-lo/pot_not_honored/",
            "name": "weblate.gettext.msgmerge",
            "id": 23,
            "configuration": {
                "previous": true,
                "no_location": false,
                "fuzzy": true
            },
            "url": "https://weblate/api/addons/23/"
        }
    ]
}

msgmerge doesn't run when a component is initialized from a zip file, even though the plugin is activated for the component when the component is created

Any errors in the logs? Any alerts on the newly created component indicating a problem?

I don't have access to the server (docker), so I have to tell my admin exactly in which log he has to look for what.

The only alert on the component is that the license information is missing. This is what is visible to me from the insights:

grafik

Translation status is shown as 100% English (source language) shows 2 phrases 4 words German shows 1 phrase 2 words, expected 2 phrases 4 words

nijel commented 3 years ago

See https://docs.weblate.org/en/latest/contributing/debugging.html#weblate-logs for info on logging.

luebbe commented 3 years ago

Here's some filtered output of the log during component creation from zip

Created with: docker-compose logs weblate |egrep -i '(Test-LO|pot.not.honored|weblate.gettext.msgmerge)' >log

I don't see anything evil here. I see that the addon is enabled immediately. I also see that it couldn't be enabled for the glossary. I also see no trace that the addon ever executed.

Before and after are just GET requests on the component

[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:52,658: INFO/450] test-lo/pot_not_honored: updating repository
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:52,676: INFO/450] test-lo/pot_not_honored: update took 0.02 seconds
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:52,678: INFO/450] test-lo/pot_not_honored: repository updated from dc1bbc3edaee020c149a34c6bbdcdbe937ed253b to dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:52,678: DEBUG/450] test-lo/pot_not_honored: update: dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:55,274: INFO/450] test-lo/pot_not_honored: updating repository
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:55,289: INFO/450] test-lo/pot_not_honored: update took 0.01 seconds
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:55,290: INFO/450] test-lo/pot_not_honored: repository updated from dc1bbc3edaee020c149a34c6bbdcdbe937ed253b to dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:16:55,291: DEBUG/450] test-lo/pot_not_honored: update: dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:17:02,900: INFO/451] test-lo/pot_not_honored: updating repository
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:17:02,918: INFO/451] test-lo/pot_not_honored: update took 0.02 seconds
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:17:02,921: INFO/451] test-lo/pot_not_honored: repository updated from dc1bbc3edaee020c149a34c6bbdcdbe937ed253b to dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:17:02,922: DEBUG/451] test-lo/pot_not_honored: update: dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:17:02,979: INFO/451] test-lo/pot_not_honored: enabling addon weblate.gettext.msgmerge
[36mweblate_1   |[0m uwsgi stderr | [2021-04-21 12:17:03,006: INFO/451] test-lo/pot_not_honored: skipped push: upstream not configured
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,098: INFO/461] test-lo/pot_not_honored: skipped push: upstream not configured
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,098: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: skipped push: upstream not configured
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,171: INFO/461] test-lo/pot_not_honored: rebase remote into repo dc1bbc3edaee020c149a34c6bbdcdbe937ed253b..dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,171: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: rebase remote into repo dc1bbc3edaee020c149a34c6bbdcdbe937ed253b..dc1bbc3edaee020c149a34c6bbdcdbe937ed253b
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,228: DEBUG/461] test-lo/pot_not_honored: running post_update addon: weblate.gettext.msgmerge
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,228: DEBUG/ForkPoolWorker-1] test-lo/pot_not_honored: running post_update addon: weblate.gettext.msgmerge
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,266: INFO/461] test-lo/pot_not_honored: weblate.gettext.msgmerge addon skipped, new base was not updated
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,266: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: weblate.gettext.msgmerge addon skipped, new base was not updated
[36mweblate_1   |[0m uwsgi stderr | [pid: 448|app: 0|req: 6/44] 172.19.0.3 () {58 vars in 1002 bytes} [Wed Apr 21 12:17:03 2021] GET /progress/test-lo/pot_not_honored/ => generated 14654 bytes in 140 msecs (HTTP/1.1 200) 9 headers in 629 bytes (1 switches on core 0)
[36mweblate_1   |[0m nginx stdout | 172.19.0.3 - - [21/Apr/2021:12:17:03 +0200] "GET /progress/test-lo/pot_not_honored/ HTTP/1.1" 200 5272 "https://weblate.*****/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0"
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,299: INFO/461] test-lo/pot_not_honored: checking pot_not_honored.pot (en) [1/2]
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,299: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: checking pot_not_honored.pot (en) [1/2]
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,310: INFO/461] test-lo/pot_not_honored/en: processing pot_not_honored.pot, new file
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,310: INFO/ForkPoolWorker-1] test-lo/pot_not_honored/en: processing pot_not_honored.pot, new file
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,418: INFO/461] test-lo/pot_not_honored/en: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,418: INFO/ForkPoolWorker-1] test-lo/pot_not_honored/en: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,442: INFO/461] test-lo/pot_not_honored: checking de/LC_MESSAGES/pot_not_honored.po (de) [2/2]
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,442: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: checking de/LC_MESSAGES/pot_not_honored.po (de) [2/2]
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,453: INFO/461] test-lo/pot_not_honored/de: processing de/LC_MESSAGES/pot_not_honored.po, new file
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,453: INFO/ForkPoolWorker-1] test-lo/pot_not_honored/de: processing de/LC_MESSAGES/pot_not_honored.po, new file
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,469: INFO/461] test-lo/pot_not_honored/de: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,469: INFO/ForkPoolWorker-1] test-lo/pot_not_honored/de: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,495: INFO/461] test-lo/pot_not_honored: checking triggered alerts
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,495: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: checking triggered alerts
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,504: INFO/461] test-lo/pot_not_honored: updating stats caches
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,504: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: updating stats caches
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,529: INFO/461] test-lo/pot_not_honored: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,529: INFO/ForkPoolWorker-1] test-lo/pot_not_honored: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,752: WARNING/461] test-lo/glossary: could not enable addon weblate.gettext.msgmerge, not compatible
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,752: WARNING/ForkPoolWorker-1] test-lo/glossary: could not enable addon weblate.gettext.msgmerge, not compatible
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,971: INFO/464] test-lo/glossary: rebase remote into repo 806a90e438acefea50b01d4e75db524ec13cdab3..806a90e438acefea50b01d4e75db524ec13cdab3
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:03,971: INFO/ForkPoolWorker-2] test-lo/glossary: rebase remote into repo 806a90e438acefea50b01d4e75db524ec13cdab3..806a90e438acefea50b01d4e75db524ec13cdab3
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,061: INFO/464] test-lo/glossary: checking triggered alerts
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,061: INFO/ForkPoolWorker-2] test-lo/glossary: checking triggered alerts
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,069: INFO/464] test-lo/glossary: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,069: INFO/ForkPoolWorker-2] test-lo/glossary: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,310: INFO/461] test-lo/glossary: skipped push: upstream not configured
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,310: INFO/ForkPoolWorker-1] test-lo/glossary: skipped push: upstream not configured
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,326: INFO/461] test-lo/glossary: checking de.tbx (de) [1/1]
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,326: INFO/ForkPoolWorker-1] test-lo/glossary: checking de.tbx (de) [1/1]
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,335: INFO/461] test-lo/glossary/de: processing de.tbx, new file
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,335: INFO/ForkPoolWorker-1] test-lo/glossary/de: processing de.tbx, new file
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,363: INFO/461] test-lo/glossary/de: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,363: INFO/ForkPoolWorker-1] test-lo/glossary/de: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,368: INFO/461] test-lo/glossary: checking triggered alerts
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,368: INFO/ForkPoolWorker-1] test-lo/glossary: checking triggered alerts
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,376: INFO/461] test-lo/glossary: updating stats caches
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,376: INFO/ForkPoolWorker-1] test-lo/glossary: updating stats caches
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,383: INFO/461] test-lo/glossary: updating completed
[36mweblate_1   |[0m celery-celery stderr | [2021-04-21 12:17:04,383: INFO/ForkPoolWorker-1] test-lo/glossary: updating completed

WeblateOrg / translation-finder

Initializing a component from a .zip file does not respect the template file #95

Stage 1

Stage 2