online translation

[koem]

title was: Translations: automatically include missing strings in values-*/strings.xml

I could write a script that reorders and updates all translation files (values-*/strings.xml):

• adds missing translations with untranslated english text and maybe a comment you can search for when translating
• puts content of every of that files into the same order as the english one

idea 1 for output

  <string name="live_map_button">sOmE cRaZy JaPaNeSe SyMbOLs</string>
  <string name="caches_nearby_button">Nearby</string> <!-- please translate me! and delete this comment then ->
  <string name="advanced_search_button">sOmE oTheR cRaZy JaPaNeSe SyMbOLs</string>

idea 2 for output

  <string name="live_map_button">sOmE cRaZy JaPaNeSe SyMbOLs</string>
  <string name="translate_me__caches_nearby_button">Nearby</string>
  <string name="advanced_search_button">sOmE oTheR cRaZy JaPaNeSe SyMbOLs</string>

Pro:

• translators can easily see what's missing
• translators don't have to look up the english text in another file
• idea 1 may be even a little faster when looking up strings

Contra:

• Big pull request that should be merged fast. May be it will produce conflicts in ongoing local developments. So we should tell everybody to expect merge problems in strings.xmls that are under local development when merged.
• idea 1 doesn't react on changes in the english text when the script is not run after every change of values/strings.xml - but idea 2 solves that

[mucek4]

Make 2 scripts. 1st to make a temp file with only missing translations (from currently translated version and english version) 2nd to make a final translation file ordered correctly.

[koem]

1st script is there since a year or so: findmissingtranslations.sh :)

[samueltardieu]

The strings are globally in order, so I don't expect a huge reordering if you do so.

However, please do not include valid XML strings with untranslated versions even with mangled, symbols. You can include XML comments though:

<!-- untranslated <string name="caches_nearby_button">Nearby</string> -->

This way, one can run the script, then look for "untranslated" in the text editor and translate the strings.

Ideally, we should also use a commit hook to prevent XML files with "<!-- untranslated" from being committed.

[koem]

Great idea!

But the commit hook is a bit too harsh:

missing translations: cs: 61 da: 711 de: 7 es: 287 fr: 123 hu: 97 it: 75 ja: 64 nb: 619 nl: 176 pl: 72 pt: 96 sk: 94 sv: 202

But: these numbers are only correct if the corresponding file is in the same order as the english file. Linux diff...

[samueltardieu]

But the commit hook is a bit too harsh

Why? It would only prevent committing untranslated strings. People choosing to translate only a few strings would have to use partial commits (git add -p for example).

[koem]

we are just chatting about that in the irc... wanna join?

[koem]

summary from irc-chat: (http://shorty.pfc-clan.de/logs/cgeo/2013-02-11.txt from 09:11:05 to 10:28:55)

• files are checked in with idea 3 from samuel: "<!-- untranslated <string name="
• no commit hook
• script can be run every now and then by anyone with a linux console - if I do the coding. No real perl/python skills here

does anyone know who the translators are and how we can tell them?

[ckroehnert]

You could include a message (how they could use the scripts) in the top of each string file, so they will see it at the next fetch/merge.

[Bananeweizen]

Good that you avoided adding valid strings. Cause some people like me rely on having the missing strings shown by Android lint instead of some shell script. That would have been broken otherwise.

+1 for a reordering script. That not only makes it easier to compare files, but it also allows to rename/move strings in the English file and then to have them moved accordingly in the other files. Until now I've avoided such ID cleanups just for that reason.

[SammysHP]

Recently I found this service: https://webtranslateit.com/ Well, it's not free, but it can import, handle and export Android string resource files. Such a service is much more simple than our experiments with scripts. Does anyone know a free (web)tool like webtranslateit?

[YraFyra]

What about http://crowdin.net ? I'm involved in a translation there, and it is very easy to use for contributors.

[RomNexus6]

koem asked about knowing translators name. I'm the one who did the IT. Add me to your list.

[koem]

ok, I'll check crowdin.net. It's free for open source.

[koem]

crowdin:

Very nice collaboration tool. Good layout, good workflows, but:

• shows strings which are the same in both languages (like OK, Version, Twitter) as not translated. There is no way to see, if they are really missing or just the same as in english.
• no git / github integration
• generates xml-files without respect to the original strings.xml (en) - comments are gone
• comments are not shown - it would be better, if you can see where you are, which block of strings you are translating. To see the neighboring strings and the empty lines in the strings.xml

Because of the first contra -> I will send them a "bug report"

If someone wants to look at my test project: http://crowdin.net/project/koem-crowdin-test/de Then "translate" and then you can see all the strings (red dot) which are translated but shown as not translated

[koem]

reply mail from github - I asked them if they plan to make their own online translation feature:

Hi Köm,

Thanks for your feedback! We're always working to improve GitHub and we consider every suggestion we receive. I've added your idea about international translations to the Feature Request List™ for the team to see. The time is probably right for us to start looking at this for the future.

Thanks, -Austin

[koem]

@YraFyra: crowdin is now one of the TOP 2 of all the systems I checked (details later). What is your workflow with crowdin? How to you get your language files back into your repository? And how do you get your language files into crowdin if there are changes made not using crowdin?

[YraFyra]

I have only used Crowdin as a contributor, ie in GpsStatus. It is very easy to translation as you get a suggested text by Bing. Integration wih github should be able to get using some regular Jenkins job to download translations and git push...

[Bananeweizen]

I've also used crowdin today and it felt quite nice. If you want to test for yourself and actually contribute to a (non open source) project, then look at the translations for http://project-gc.com/ and try to fix them, whenever you see some bad translation on the website of that project.

Translation link: http://crowdin.net/project/Project-GC

[koem]

Progress update / status: I think I've tested all major systems out there now. The two in focus are:

a) crowdin: really good interface and free for open source projects. No github integration, but API for use with wget and curl. One drawback: download of translated files only every 30 mins.

b) weblate: ugly but with git integration. Demo page online. Has to be installed on server. Every single translated string leads to a commit (I read somewhere). Couldn't install it on my machine, too many dependencies. Does someone wants to give it a try and install it?

I like crowdin.

But we (me) need to script up and downloading+commiting of files - no problem.

But we also need to trigger some things - upload when there is a new version of english strings.xml / and maybe trigger manually the download of new translations, or just cron it. can we do this?

[Bananeweizen]

Yes we can. Crowdin has a huge API: http://crowdin.net/page/api. And that API can be handled by a small command line client: http://crowdin.net/page/cli-tool

So we can easily create some script around that. For the automation I would simply use the existing Jenkins server. Have a time triggered job to fetch new translations from Crowdin and to push them directly to git. Have another job to upload all translation files, which is automatically invoked after each change on our repository.

To make things not too complicated, I suggest we run this in 3 defined steps:

• Setup the crowdin project, and do manual up- and downloads.
• Automate uploads to crowdin.
• Automate downloads and merging (this might be the hardest part).

[SammysHP]

One suggestion: IMHO it's better to push new translation into a separate repository and automatically create a pull request. So we can review the changes, they are checked by Jenkins and if nobody merges the changes for some days additional commits will be appended to the existing pull request.

[koem]

I set up the project already.

We should only automatically upload english strings.xml automatically. For the other language strings.xml files we must decide if we want to edit them in crowdin OR in git. Both crowdin AND git is not good (merging). German => git. French => ?. Spanish => crowdin, ...

I can do all that, but right now I want to work on issue 750 (Settings screen).

When done, we should ask for translators on facebook.

And we could need some english natives, to check our german influenced english strings...

[SammysHP]

We need a solution that pulls from git each time somebody wants to change something.

[Bananeweizen]

We need a solution that pulls from git each time somebody wants to change something.

I don't think so. As koem wrote above, if we have exactly one direction per language, everything is easy to manage. And I would agree that we should translate English and German in git only, every other language in crowdin only. Now two way merging by default

Of course, that still allows a developer to manually merge an Italian translation update in git.

[SammysHP]

What if somebody sends us a pull request with English and another language different than German (together with e.g. a new feature implementation)?

[Bananeweizen]

I don't know the crowdin system myself that much, but I guess that we can upload single files at any time. Therefore in that case I would first review the open changes in crowdin and merge them into the repo, and then merge the pull request. By that order we should always see the conflicts directly in the pull request and can solve them on the git side.

Honestly, I also don't know if that works well or not. I think we should just try it. If there are a lot of problems, we have to solve them then. But maybe such conflicts occur very seldom, then we don't need to have a perfect solution for this case.

[koem]

Yes, that's how it works if we get translations via git. Download, merge, upload. But once a language is in crowdin, we shouldn't do translations in git anymore for that language. But there are sometime things you can only do on your dev machine, like renamings (find & replace).

I set up the project. Just one language now - spanish. If you want to do something: login, search for c:geo and join. I don't know about rights for users yet. I don't want anyone to have administrator rights. So let's see how that works.

[koem]

@Bananeweizen : the cli-tool is crap I think. At least on my machine, the ruby version didn't work and the java version took more than a minute to load. But with wget and curl I was able to do everything fast.

[Bananeweizen]

As it is not really possible to just join a project, the easiest way to get c:geo into your own list of projects is

• search c:geo in the top search box
• open the Spanish translations, vote one of the up or down (I used the app name).

Afterwards c:geo will be in your personal list.

[Bananeweizen]

@koem Do you have access to the Jenkins server? Or do you want to create the script part as such and then @mucek4 or I can create a job on Jenkins (including triggers by commits or pull requests and all that stuff).

[koem]

@Bananeweizen No, I dont have access to the server. But I think I can create the scripts on my laptop and then someone can put them there.

But not in the next 2-3 weeks - wanna finish work on settings first.

[koem]

@Bananeweizen, @rsudev I gave you the status of a proofreader. Can you manage the project settings now? http://crowdin.net/project/cgeo/settings

[rsudev]

@koem No, I can't

[koem]

from crowdin support, just to keep the information:

Yes, you surely can add more managers to your project the following way:

1. Login to Crowdin;
2. Choose "Resources" tab from the head menu;
3. Select "Managers" option from the dialog that will appear; 4.Press "Add manager" button;
4. Enter emails of people you want to manage your project.

so @rsudev @Bananeweizen - I need the mail addresses, you used to log in to crowdin -> spam ät petoria punkt deutschland

[rsudev]

Now I am a 'manager' and can see/change the settings, thanks. BTW, I think we could even put German translation to Crowdin. It might be easier WRT string re-ordering, changed/new source strings and stuff like that. And there might be other translators that can benefit from the presence of the German translation.

[Lineflyer]

Do I need to be a manager to contribute? I don't think so, so I will stay a "normal user" unless I could do something useful on crowdin to support the translation work.

[rsudev]

No, you don't. Everybody can contribute currently. Once we are set-up we can spread the word and hopefully get some more up-to-date translations. In our company even English is a 'translated' language, that is created from the developer-English by native speakers. Perhaps we can at some time move to such a work flow as well.

[Lineflyer]

In our company even English is a 'translated' language, that is created from the developer-English by native speakers.

Could be useful indeed :)

[samueltardieu]

@koem Could you add me as well?

[koem]

Add german: that actually is also not such a bad idea. We should give it a try when the self made git integration is set up. The original title of this issue was something with "finding missing translations"

[koem]

Add english: this is an amazing and idiot proof idea! we need a base language in crowdin. This can only be dev-english (values/strings.xml) . But a second real English is values-en/strings.xml then. When it lacks some translations every now and then, android will grab them from dev-english file!

[Lineflyer]

One remark: If possible we should include the PlayStore description as well. Currently this is done in

https://github.com/cgeo/cgeo/wiki/App-description-for-PlayStore https://github.com/cgeo/cgeo/wiki/Plugin-description-for-PlayStore

which is not very nice in terms of processing it.

[koem]

reminder: check what happens when new values/strings.xml is uploaded with changed texts. Will crowdin mark the translations for which the english text has changed?

[samueltardieu]

Note that this should not happen. Or only for cosmetic change with no semantic change.

[RomNexus6]

I'm sorry, but I don't understand what happened. Is c:geo translation available on crowdin or must we translate by git as before? thanks.

[Bananeweizen]

@RomNexus6 We are in the transition phase, somehow. We have a project setup on crowdin, but we don't have the scripts for automatically merging those translations ready. So until we announce a change, translation via git is still perfectly fine.

[koem]

I'm gonna start writing the scripts now. Any special wishes?

Script 1: upload new values/strings.xml

• file will be taken from master
• script will be started periodically or triggered if possible

Script 2: package and download translations (values-*/strings.xml)

• when there are changes and compile is ok -> push to master directly?
• script will be started periodically, maximum every 30 min, because of restrictions in crowdin services

[SammysHP]

No, only pull request.

[koem]

Scripts are done. There is one thing left: uploading changed language files (values-*/strings.xml) to crowdin (automatically).

• normally translations shall ONLY be edited in crowdin
• but there can be exceptions, e.g. renaming of string IDs, global renamings like send to c:geo -> send2c:geo
• when this is done and the changed files are uploaded to crowdin, it may happen that changes done in crowdin in the meantime may be overwritten

I can only think of these not so cool solutions:

• defining maintenance times (e.g. 2am to 3am UTC) and using an upload script via cronjob -> nobody works in crowdin at these times!
• alarming all translators before doing the uploads (manually, via script)
• doing it anyway automatically and fast -> let's hope nobody looses some translations

[SammysHP]

Translators usually live in other timezones. ;)