Use Crowdin for translations

[stefanhaller]

We don't have any tooling for keeping track of which newly added english texts need to be translated into other languages. Nobody really knows how out of date our translations are, or how to find out. Instead of investing in such tooling ourselves, I suggest we look into using crowdin.com instead.

We recently started using Crowdin at work, and I was pleasantly surprised at what a nice and powerful platform this is. Basically you upload your english text file, and then contributors can swarm on typing in translations for various languages, all with a pretty nice web interface. There's a dashboard showing how many texts are left to be translated for each language, and also which texts have "quality issues" (an example being that the english text ends with a period, but the translation doesn't, and many others). And of course it can pre-translate things from Google Translate or DeepL automatically.

This requires that we have data files (json or yaml) instead of code for our translations, so we'd need to change our code a bit to serialize things to/from json or yaml. That doesn't sound very hard to me.

Crowdin is not cheap, but it has a free plan for open source projects which I think is very attractive.

[jesseduffield]

Sounds good to me. Re: text files, I assume we would end up generating code from them with 'go generate', is that right?

[stefanhaller]

Not really. For the english texts we'd need a script that marshals it to json so that we can upload it to crowdin (that json file doesn't have to be committed to git). For the translations we'd just download them as json files from crowdin, commit them to git, and then unmarshal them at runtime.

One of the many advantages of doing it this way is that developers no longer have to care about the translations at all. In commits such as 0df5cb1286a we had to touch all languages so that things compile again; this will no longer be necessary.

[jesseduffield]

If we unmarshal at runtime, how does that work when we need to distribute a binary? This is something that I'd want to have baked into the binary which is why I assumed we'd use code generation to do that unmarshalling rather than handling at runtime.

I'm not tracking how the end-to-end flow works: if I want to add a new translated string in a PR, do I need to go into crowdin and update the translation there? Or do I update the json file locally and push it to crowdin? Or does CI do that for me?

[stefanhaller]

Ah right, I forgot that we can't just add something to our Resources folder. 😄

In that case, yes, we'll need a go generate thing that generates code from the json files.

As for the workflow: developers would not usually be concerned with crowdin, they'd just edit the english translation struct in the code as today.

Then, occasionally somebody would run the script that generates json from the english struct, and upload that to crowdin. We could also look into automating this step on CI, I think it should be possible.

For the other direction, somebody would occasionally (e.g. just once right before a release is cut) download the translated json files and make a PR to commit them; at the same time the go generate would be run to convert them back into structs.

[stefanhaller]

I'm happy to work on getting some (or all) of this going, but I don't think I'll have time for that before mid November.

[jesseduffield]

Okay, sounds good!

[hasecilu]

An alternative to Crowdin is Weblate which is open source, free for open source projects and self-hostable if needed.

As translator I find it very user friendly and it has very nice GitHub/GitLab integration.

Anyway, choose according to the project's needs. I'm interested on Spanish so I'll be waiting for it.

[stefanhaller]

I'm planning to start working on this very soon now.

Re: json files, I'm going back to my original suggestion of bundling the json files into lazygit and unmarshalling them at runtime. This is the easiest way of dealing with it. I recently learned about the embed feature of go, which makes this very easy. PR will hopefully come soon.

Re: crowdin vs. weblate: I'm having trouble deciding which one is better for us. I have a little bit of experience with crowdin myself (through my day job), and zero experience with weblate. Anybody here who knows both and can help us choose? Are there even other options that I didn't consider?

[stefanhaller]

PR for loading the non-English translation sets from json is here: #3649

We still need to decide on which translation service to use. Another candidate seems to be https://poeditor.com. Any help with choosing one is much appreciated!

[hasecilu]

Hi, I don't manage translation system so I'm not particularly knowledgeable in that specific regard but I have participated as translator on different platforms as Weblate, CrowdIn and Transifex.

I have seen more recurrently the use of Weblate & CrowdIn on FOSS projects, both of them are pretty fine, good UI, show recommendations, can be configured to use translation providers, both can have glossary (which is very handy for large projects). Personally I like how the Weblate bot adds stats on commit message (example of commits made by Weblate bot) so I tend to prefer it.

[stefanhaller]

@hasecilu This is useful input, thanks. I'm not so worried about administrating the system, I guess the effort for that will be comparable for all of them. Sounds like it's largely a matter of taste then.

[comradekingu]

Crowdin doesn't have a free plan. It has a non-monetary cost plan for non-commercial open source projects. It is also laden with one-way revenue exchanges, so it isn't at all gratis.

The Libre plan of https://hosted.weblate.org/hosting/ isn't gratis either as it requires mentioning in the README, but that is a fair proposal.

[stefanhaller]

Crowdin doesn't have a free plan. It has a non-monetary cost plan for non-commercial open source projects. It is also laden with one-way revenue exchanges, so it isn't at all gratis.

Can you elaborate on this? As far as I can see, lazygit's crowdin project is using the free open-source plan. What is the problem with this that I am missing?

[stefanhaller]

Ping @comradekingu; I'm very interested in hearing more about this, can you explain?

[comradekingu]

@stefanhaller The word "free" doesn't mean comissioned work in exchange for service.

I read the terms and conditions every so often to point out updated pitfalls, and this should still be relevant. https://github.com/NightscoutFoundation/xDrip/discussions/3420#discussioncomment-10513307

I have a PR for Tolgee to remove the CoC there, but that wouldn't make the project viable. As I read the terms I could be liable for 1 million in damages since I have helped Weblate (pro bono) a lot.

https://tolgee.io/docs/terms_and_conditions/terms_and_conditions 6.5. Non-Compete. You must not to develop, sell, or otherwise make available to any third party any software or service that serves the same or similar purpose or has the same or substantially similar functionality as the Service or software used in provision of the Service and you must not assist any other person in such activity. The restriction applies in the USA and EEA, during the term of the Agreement and for a period of 2 years thereafter. If You breach the provisions of this Section 6.5, You shall pay Us a liquidated damages EUR 1 million.

While I think it is unenforceable and totally invalid, I also can't risk that, so I don't know more about the platform. I'd be happy to recommend more good alternatives, but it is Weblate and Pontoon atm, with Pontoon functioning worse as a tool.