Help translate the web interface via weblate #3183

Open aparcar opened 4 years ago

Hi, for testing I created an OpenSource account on weblate which gives a nice interface to translate the PO files. To improve the process I'd install the weblate service integration (also available for GitLab an other services) which automatically a) add newly added strings to their translation web interface and b) creates pull request once every now and then when updates via weblate happend. If anyone is against this approach please let me know. Right now it looks like only 52% percent is translated.

I did a registration with github and proposed a change on a translation. I don't want to get too deeply involved for now. But to get a feel for how it works, can you please explain to me how this change I have made then goes into the master branch on github?

I played a bit with the tool and it works surprisingly good!

From what I understand it is possible to manually trigger a commit (like https://github.com/openwrt/luci/pull/3190/) or set it to create a PR every x hours based on the recent changes. Instead of PRs we could also allow it to directly commit those changes, however I'd leave this off for now.

Cool thing is we can also host it on our own if weblate changes their policies, which I don't see coming anytime soon but still a requirement I'd say.

Thanks for clarification. I think directly commit is not an option.

I have done some translation in luci-app-statistics and saved this. CPU-Frequenz and Adressfamilie But nothing happens. No creation of a new pullrequest in the openwrt/luci :-( Do I miss something?

Currently there are the two options to auto commit every 24h or perform it manually - as I just did. This however ended up in a somewhat messy PR as people started to translate also Portuguese and Russian files.

I set it up that from now on PRs are squashed by author, meaning whatever you do in 24h will become a single commit.

It is possible to add the weblate git repository via

git remote add weblate https://hosted.weblate.org/git/openwrt/luci/

Seems to work well. Please continue with this in case further changes are needed.

What is the mechanism here? I have received ~100 email during the night from PR 3197...

Subject: Re: [openwrt/luci] Update from Weblate (#3197)
From: Weblate (bot)
To: openwrt/luci
Cc: Push
Reply-To: openwrt/luci
Date: Today 06:41

@weblate pushed 2 commits.
    ea20df6 Added translation using Weblate (Korean)
    8c98352 Translated using Weblate (German)

Each new translation via weblate gets spammed to all LuCI committers?

(I guess we can unsubscribe this PR to get rid of the spam, but sounds strange if all future translations are still tied to that one PR. )

@hnyman sorry for that! I will see how to disable these notifications! The weblate service updates daily a PR with a single commit per author. This way we don't end up with x unmerged PRs for translations.

@feckert wrote

@aparcar How can I add missing apps in weblate? Specifically I am talking about luci-app-mwan3?

The weblate team initially setup the service but made a small mistake, I fixed that and the missing components should now be available.

Another 300 mails, but the storm is over and whoever survives the 200k new lines of PO files in the next git pull should look into a nicely translated future. I'll leave this open for further requests.

I see that when translating an App (e.g. DDNS), a string that I modify is "Enabled" and is also modified in all apps with that word to translate. I mean, in one App it means one thing and in another it means something else even if it is the same word in English. I don't know if it can be deactivated or dodged.

Hi @castillofrancodamian please don't crosspost issues, I responded to your forum.openwrt.org message.

How is translation supposed to work for release branches? I see that there were some updates to openwrt-19.07 after it was branched off (e.g. dd0c93224 and 83a7292a7), but not all translation updates seem to end up in the 19.07 branch.

I fixed several translation mistakes in 19.07.0-rc1 and it would be nice to have the fixes in the final 19.07 release.

I think this depends on @jow- , I'd be okay with moving all translations over to 19.07 just before release freeze or set a public date for a 19.07 translations freeze.

I planned to do some scripting to sync 19.07 translations with master ones by using the master po files as translation catalogs for the branch ones. That should ensure consistency

Hmm. I synced translations in master and that seem to have caused a few conflicts at the current weblate PR #3355

These weblate PRs have no clear author to be contacted, so I will likely fix those conflicts myself.

But I am not sure what would be the correct workflow in future. Probably merging the current weblate PRs before syncing translations files?

There has also not been much discussion how these should be handled daily. So far @aparcar or @jow- have mainly applied PRs, partially manually, I think.

Thus I wonder what is the actually the correct workflow with these weblate PRs:

Simple merge of the PR without any changes at the merge stage?
They can be merged at any time?

I don't know what's the best approach but would be happy not being in charge of it :roll_eyes:

Maybe things are simplified by using the weblate github integration. Most issues (seem) to appear by weblate being out of sync, while the app (hopefully) triggers a rebase on every commit to luci.git.

would be happy not being in charge of it

Well, you introduced it...

Can you at least answer my previous questions:

what is the actually the correct workflow with these weblate PRs:

Simple merge of the PR without any changes at the merge stage?

They can be merged at any time?

Apparently the conflicting commits from yesterday have been removed automatically. (Spanish, if I remember right). Probably they have been somehow pushed back for translation/checking. Is that supposed to happen?

would be happy not being in charge of it

Well, you introduced it...

Fair enough. I think I wanted to say to distribute the merge capabilities.

what is the actually the correct workflow with these weblate PRs:

Simple merge of the PR without any changes at the merge stage?

I wonder if the SoB line is important for translation updates. If not, the GitHub PR merge should be fine. The problem seem to be syncing.

They can be merged at any time?

I think so, however it bloats the git log. Maybe we should only do so once a week? Maybe I got your questions wrong.

From what I understand using the Weblate-GitHub-App allows to set a push rate to daily or weekly, so the entire merging could be done automatically. I guess part of the issue is that when using PRs they stay open for multiple days and are out of sync at some point.

Is weblate borderline unusable for anyone else here? Their servers / software are always incredibly slow for me.

@MartB I think it depends where you are, I tried it before in Honolulu and it barely worked, using it now in Leipzig works fairly well. Maybe we as a bigger project should consider to donate some money for faster servers. I'm against self hosting as it'd be yet another service to maintain... But this is just my personal though.

@aparcar Im sitting in germany <100km away from Frankfurt, so yeah not sure whats going on there. Submitting translations/suggestions and only displaying a certain language takes ages to load for me. Not sure what amount of donations would change their mind given that they have a premium service. The slow performance could be due to a deliberately low resource allocation from their side (i understand that from am business pov).

Self hosting would be a possibility but also requires money, time and an active maintainer/sysadmin.

@MartB please report this problem upstream to the developer.

It seems, that Weblate has conflict and it could not merge the new translations. Can someone check it and try to solve the conflict?

Well, the conflicts seem to be mainly due to the recent "typo corrections", either spelling or case changes...

Looks like the typo corrections have caused some 50 conflicts there, as they have been made in another repo at the same time as there has been translation going on in weblate.

Example of conflicts:

#: applications/luci-app-adblock/luasrc/model/cbi/adblock/overview_tab.lua:251
<<<<<<< HEAD
msgid "Topic for adblock notification e-mails."
msgstr ""
=======
msgid "Topic for adblock notification E-Mails."
msgstr "Sujet pour les e-mails de notification adblock."
>>>>>>> weblate/master


#: applications/luci-app-nut/luasrc/model/cbi/nut_server.lua:91
<<<<<<< HEAD
msgid "Maximum Retries"
msgstr ""
=======
msgid "Maxium Retries"
msgstr "Legtöbb újrapróbálás"
>>>>>>> weblate/master

And some of those conflicts seem to be just header meta data conflicts. Too bad.

@aparcar @jow- are there any config options on how often weblate tries to refresh upstream LuCI sources?

As a vanilla weblate user, I have not yet figured out how often/why/when weblate actually tries to pull new sources from us. I fixed a dozen conflicts yesterday and merged that, and weblate took maybe 20 hours to notice that we have new upstream LuCI sources. Now I have fixed the next conflicts from the translations done meanwhile. I currently wonder if it will take again almost a full day for the changes to get noticed at weblate, so that new conflicts will get generated in the meanwhile...

Likely the "free plan" (or whatever we currently have) has a pretty slow update polling frequency, but still sometimes weblate seems to try updating several times per hour, so the updating frequency varies.

So, can those who have some admin rights to weblate, please check if there is something to be configured to enable a somewhat more regular update trials.

Ps. who has any admin rights to the OpenWrt weblate instance? aparcar and jow?
(At least jow has been somehow able to lock/free the weblate instance during his conflict fixes.)

It's me and jow, I guess you can have admin access as well.

The current frequency is likely based on the "create a PR every 24 hours with the latest changes" setting. In other words that's when it tries, fails and notifies.

Based on the documentation I added a webhook, however the it appears to be broke. I did not figure that as it doe not report an error even tho it fails.

I'll report an error on the weblate repo, but can"t yet figure out why it wont find the openwrt/luci repository...

All right I found the problem. Apparently Weblate is a bit picky regarding leading slashes. Once removed it works.

Here the (really) successful hook

Yep. Now it seems to also receive web notifications, as it reacted to the PR merge a few minutes ago:

Ps. who has any admin rights to the OpenWrt weblate instance? aparcar and jow? (At least jow has been somehow able to lock/free the weblate instance during his conflict fixes.)

It's me and jow, I guess you can have admin access as well.

Might be good. I tried yesterday to force weblate pull via wlc and weblate api, but I missed the necessary authorisation.

Of course, if weblate will pick changes more quickly in future and do frequent rebases, there will be less reasons for manual interaction.

Great that you figured out the reason for the webhook's inactivity.

@hnyman I added you as an admin. Thanks for pinging me before!

Glad it works now. I'll report another issue upstream that leading slashes shouldn't cause trouble...

@castillofrancodamian Please stop mass-"translating" technical abbreviations/terms with identical content as the original English string, like "BSSID", "B43 + B43C" etc..

Having the identical string in a language (marathi, bulgarian, catalan, whatever) only enlarges the size of the translation package without providing any added value except having a higher "translated" percentage shown for that language in weblate.

That added size just wastes storage space.

That it is available to translate those things is also not useful. Wouldn't it be better to remove those strings in the translation?

@jow- What would be the best way to programmatically remove these nonsense "translations" of numbers or arconyms?

They are now included in the proposed weblate PR #3504 ... If the PR gets approved, we will have hundreds of msgstr strings translated identically to msgid If the PR gets rejected, we will probably have some conflict with weblate.

Best way might be accept the PR but then programmatically remove the msgstr strings that are equal to the msgid string. (not sure how the weblate system would react to that, but goes over my perl skills in any case)

Please stop mass-"translating" technical abbreviations/terms with identical content as the original English string, like "BSSID", "B43 + B43C" etc..

Having the identical string in a language (marathi, bulgarian, catalan, whatever) only enlarges the size of the translation package without providing any added value except having a higher "translated" percentage shown for that language in weblate.

That it is available to translate those things is also not useful. Wouldn't it be better to remove those strings in the translation?

Some of the strings get semi-automatically picked up for translations e.g. from typical drop-down box item lists without any thought of somebody really doing a "translation" on them.

Some of them are sometimes (rarely) actually translated like this:

msgid "RX"
msgstr "接收"

I get it. The idea of "translating" these numbers was that those who translate the other languages skip them to save time. But I also understand that it occupies space.

Well, the good thing in your actions is that it made me to look at the already existing translations, and I noticed that there are some similar unnecessary ones already in some languages (e.g. Chinese).

It might make sense to purge them, so I wonder if there is an easy way to get rid of them. Maybe some gettext command/parameter, but I haven't yet figured out how to purge msgstr that are identical to msgid.

Perhaps some algorithm or script in which known numbers and acronyms are omitted.

Hi!

I'm one of the Brazilian Portuguese translators for the openwrt project. Just finished all the translation and I've noticed a bunch of exclamation marks all over the project, some times weblate complains that the project it's out of sync with the repository and warns me that all my work will be lost.

Can you guys please do something about it? Thank you.

PS: Can I suggest to correct our language name in the project? Our language name is Brazilian Portuguese and not Portuguese (Brazil).

Thank you.

Hi @wtuemura this is sadly nothing we can influence directly. It is an upstream issue within their code here. Please open an issue there.

Thank you @aparcar I'll.

@aparcar Please link to the issue(s). Very interested to know how it is wrong. Language name (location) is the convention used elsewhere.

@comradekingu I did not open any issue but asked @wtuemura to do so. I'm not sure if Weblate accepts renaming the language identifier to "Brazilian Portuguese" as it looks like a correct display name and I don't think Weblate offers options for dialects.

Regarding the out of sync issue, is this still happening? I remember a broken webhook causing trouble but this was fixed some time ago upstream.

@comradekingu @aparcar Looks like they are following a different format(?)
I used to know as en-us as "American English", pt_BR as "Brazilian Portuguese" now is "English (United States)", "Portuguese (Brazil)".

American English, Brazilian Portuguese, Canadian English so on and so forth are Language Variants.

Maybe Weblate is using langtag instead of variant as of BCP47?

PS: There is no issue report yet.

@aparcar @jow- I seem to not have rights to add new translation components.

luci-app-sqm should be added for translation (as it was transferred from the packages feed to the LuCI feed).

~~@hnyman The weblate pricing table says regarding "number of components": Negotiable I'll write the dev an email and ask for more...~~

There is no limit of components, the problem is the component discovery (addon). It detects components via a regex similar to /luci-app-{component}/po/{lang}/{component}.po where in SQMs case "only" the the pot template was available.

I reported that problem to the Weblate dev, for now we should simply provide /po/en/{component}.po so it is detected. Weblate then automatically populate po files for all missing languages.

The commit https://github.com/openwrt/luci/commit/731eace63f78d8eef64990b0ff3c6ab4e7dda219 from @hnyman fixed the issue and the component is now available for translation.

Looking at https://github.com/openwrt/luci/issues/3984 I wonder if there is a clean dependency magic that allows to install something like luci-ru which automatically pulls other translation dependencies?

Looking at #3984 I wonder if there is a clean dependency magic that allows to install something like luci-ru which automatically pulls other translation dependencies?

@aparcar It's just a thought. But wouldn't it be possible to generate a meta package and install in the pre install script all translations from the language of the installed packages? I've often wondered the same thing

@jow- @aparcar

I noticed that some apps, like luci-app-opkg, are partially closed for translation: most of their source strings are marked with "source in review" and the target language strings can't be edited.

I have traced this to the fact that some apps contain unnecessary "English" translations, where one or more strings (or comments in weblate?) have been entered, and then weblate automatically thinks that the English translated string file should be used as the "source" instead of the msgid strings in the language .po. And as the English .po file is mostly empty, weblate declares all not-found source strings in other .po files as suspicious and refuses to allow translation for them.

Weblate also complains about the duplicate languages frequently after each refresh from LuCI github:

I tried to reset opkg's English translation (by clearing upstream the one "translated" string), but that did not help. Weblate still allows only a few strings to be translated, while all others are locked due to "source in review". Apparently it remembers that opkg has been "translated" to English at some point.

I see three different solutions:

We remove all "en" English translations from LuCI. There is no real reason to have them as the original source code is in English.
- We might need to revamp the translation selection logic in LuCI, so that English would not be really there, or it would be like "English (no translation)". I tried to look at the LuCI source, and I do not see any real need for English, as the default setting is empty in config file, and the selection defaults to "auto" if there is no setting. ( I wonder why we at all do create the empty i18n-xxx-en files ??? Is there some historical reason for that, jow ?)
We duplicate all strings in "en" English .po, just for weblate purposes. But that would be an additional step for the i18n-sync or similar tool.
We somehow manage to make weblate to forget about English translation. But so far I have been unable to find options for that.

Looking at #3984 I wonder if there is a clean dependency magic that allows to install something like luci-ru which automatically pulls other translation dependencies?

@aparcar @hnyman I have played with the idea but unfortunately found no way to make it easy. Has anyone else tried it? I took a look at it in luci.mk and tried to find out if LUCILANG\<XX> is set to m or y.

If the value is y then this language is add as an dependecy to the package and is installed as an dependency on opkg install luci-app-<package_name>

If the value is m then nothing is changed the language luci-i18n-\<package_name>-\<lang> has an dependency to the package as before.

This works only for image generation if you build your own images. If you set the language you want to have in your image y on build then this get installed as an dependency of this package. So you do not have to install the default i18n´s for your application anymore this is done by opkg.

openwrt / luci

Help translate the web interface via weblate #3183