Language packs and translations

[eddieajau]

Need to talk to the TT's about how decoupled extensions are going to work in terms of translations.

[infograf768]

The only viable solution I see atm is to always include the extensions language strings in the full CMS. This means that any change in an extension language strings would only be done when we release a new version of the CMS. This will let TTs include the strings whether they are used or not.

[eddieajau]

@infograf768 we can't have any files for these decoupled extensions in the CMS at all (that's what decoupled means). We'll need to work through the issues and see if we can come up with innovative and efficient solutions.

[wilsonge]

Is there any reason why we can't use transifex to manage our translations (we could even try and link the existing Joomla translating component into their API???).

Then posting latest versions of language packs becomes our responsibility and also gives us the ability to separate out any language packs that we don't want in the core system!

[eddieajau]

So I am not familiar with how translation files are maintained anymore, but in terms of distribution, I expect all language files for all translations will live in this repository. And this will be the case for each extension suite. How that can work with how Transifex works, I'm not sure but I'd be happy to talk to them about ways we can make this as efficient and painless as possible.

[Hackwar]

A viable solution would be our own translation site, which I described earlier to other people in the community. We can't keep the translation files in the CMS and especially since com_weblinks wont be the last extension that we make standalone and hopefully since there will be new standalone, community maintained extensions, we have to implement our own translation site.

[eddieajau]

A viable solution would be our own translation site, which I described earlier to other people in the community.

When it's available, I'll be sure to use it :) But we need this all done and dusted by May 28.

[Bakual]

Transifex (more precisely OpenTranslators) works quite well for extensions. It's what many extension developers already use for quite some time. You can also automate the up- and download of the language files. I think Michael does that already with JIssues.

The real problem here is not so much technical but more about who will do it. Currently we have excellent language teams which make sure that the translations are up to date for the releases or shortly after. We have that for around 60 languages if I'm right. That's only because we have dedicated teams and a single place where the files are located, using their preferred tools to translate the strings. Moving to Transifex may work, but may mean that we lose on quantity or quality because the teams will concentrate on the remaining core files. Also Transifex has some limitations as I understand it.

I think with the current translation structure, we may have to keep the language files managed using the core language packs. That will work the same it does currently for the webinstaller plugin. It's not exactly nice but will work. I think changing the translation process will be a big task which likely goes beyond the timeline for 3.4 :smile:

[infograf768]

The TT coordination has already explained multiple times why we can't use Transifex: The number of strings in a said language can change depending on the plural forms. Also some strings are explained by comments and others may or may not be commented. Therefore the comments have to display.

Also has to be taken into account the fact that TTs may be working on 3 instances of a language at the same time. For example: 2.5.19, 3.2.3 and 3.3.0 dev as of today... Then they have to make the packs (updates or new) and release.

@Wilsonge Could you explain "also gives us the ability to separate out any language packs that we don't want in the core system!" ?

@eddieajau The way it works as of now is that core lang packs are posted by TTs on a specific repository on joomlacode and then a cron job makes them available to users in the admin interface via Update or Install languages. Each Team choses the way they work together (Github, SVN, Text Editor, Webtranslateit, com_localise corrected manually,Transifex own project manually corrected, etc.). Separating some "core supported" extensions lang files from the rest of core would mean that TTs have to produce their work differently. I am afraid of the confusion in the workflow and the ability by some to use github.

[eddieajau]

Separating some "core supported" extensions lang files from the rest of core would mean that TTs have to produce their work differently. I am afraid of the confusion in the workflow and the ability by some to use github.

Yep. I'm asking you to keep an open mind and embrace this as a problem we should try to solve. Let's exhaust all possibilities before throwing in the towel.

[Hackwar]

I'm not saying to use Github. I'm not saying that we should make it more complicated for them. What I am saying is, that they have to adapt to our translation workflow most likely. We simply can't make everybodys wish come true and implement every possible workflow out there.

[eddieajau]

Sure. I think it's a good idea but I would not want it to hold up this particular feature (if you can call it that). And yes, I'd like everyone to think out of the box on every aspect of how this extension gets to market.

[nicksavov]

Language strings are part of the b/c promise of the new development strategy, thus they can't be removed in J3.

[dongilbert]

@nicksavov, language strings for a component that is removed from core can also be removed. There is no BC issue here. The BC promise on language strings is that the keys won't change.

@infograf768 Wouldn't it make the lives of the core TT's a lot easier if they had a lot fewer extensions to translate? If we offload the translation of decoupled components to a 3rd party, such as Transifex or some other provider, wouldn't that make it the whole process simpler for everyone?

[toretto]

I have been asked by my collegues of OpenTranslators to clear up some misunderstandings about Transifex (which you can use without OpenTranslators' help, by the way)

1. OpenTranslators and Transifex are separate things. OT is a Joomla community project, USING Transifex as a translation tool.
2. Transifex does support comments on translations files. It automatically imports them from .PO files. For Joomla's .ini files you can add them manually or use the Source String API.
3. The multiple version story is irrelevant, since you're not translating JOOMLA but a single extension, where one version will suffice. Otherwise, you could add different sources per extension.
4. Using Transifex and CTransifex making language packs is extremely simple. Or you could use the Transifex client (CTransifex more for "end users", really easy but really powerful).

On top of that, Transifex offers a ton of advantages for both translators and developers, but you've got to keep an open mind to appreciate them.

Based on our three years of experience translating Joomla extensions with Transifex, and looking at the feedback of our 2500+ volunteers, we think ew can conclude that Transifex makes translating extensions easier, and Transifex has been very open to suggestions we've made in the past. Their tool keeps improving every day. No need to get stuck in the past.

[wilsonge]

Could you explain "also gives us the ability to separate out any language packs that we don't want in the core system!" ?

So transifex means we can separate out the language files a bit which should mean we have more flexibility in packaging them together. We can package them all together split them up into core + individual components etc. however we like :)

JM can you explain a bit more about The number of strings in a said language can change depending on the plural forms. because it's fairly clear that translations comments aren't a problem in transifex :)

[nicksavov]

I'm pretty sure what JM is referring to is the ability to quickly uncomment a line, then translate its string, e.g. checkout lines 37-49 of: https://github.com/joomla/joomla-cms/blob/master/administrator/language/en-GB/en-GB.plg_captcha_recaptcha.ini#L37

Is that possible?

For the quote about plurals, check out: http://docs.joomla.org/J3.2:Making_a_Language_Pack_for_Joomla#the_fr-FR.localise.php

Define specific plural functionality for some languages where the value of the string can change depending on the count (Russian for example).

[infograf768]

@dongilbert Would be much easier for TTs to not have to translate anything... Seriously now: It is the decision of the PLT, when decoupling anything or for new "core supported" extensions to go on trusting the TTs OR to choose a system where any volunteer, who may or may not translate stuff the best way and in sync with the rest of that language usage and grammar, will provide translations when they have time... I have seen 3pd translations done on Transifex (OT) using Google Translate... Unreadable.

That proposal was already made a few years ago for the whole core and was turned down as we can't depend on single unknown volunteers for core stuff translating. => Language translation is NOT like code. My opinion is that any "core supported" extension should be translated by the TTs to make sure we get the quality needed and that we get translations for ALL core packs. To "offload", as you say, would therefore mean that all the TTs would have to use a specific platform only for these files (as I said, Transifex does not fit and we have been waiting for ages for major/senior Joomla devs to spare a bit of their time to help us concerning languages translations...

@toretto The experience of a 3pd dev who does not use plurals and does not have to make sure that the indications in a commented line are read and understood or lines uncommented when translating are evidently very different from core. And btw, forget .po. We use .ini.

@wilsonge Please look at https://github.com/joomla/joomla-cms/blob/staging/administrator/language/en-GB/en-GB.plg_captcha_recaptcha.ini from line 37 on. also a recent one: https://github.com/joomla/joomla-cms/blob/staging/administrator/language/en-GB/en-GB.plg_twofactorauth_totp.ini line 20 to 24 If one does not read the comments or be able to uncomment, they can't adapt their translations.

Concerning plurals, this is the Russian plural:

public static function getPluralSuffixes($count)
    {
        if ($count == 0) {
            $return = array('0');
        } else {
            $return = array(($count%10==1 && $count%100!=11 ? '1' : ($count%10>=2 && $count%10<=4 && ($count%100<10 || $count%100>=20) ? '2' : 'MORE')));
        }
        return $return;
    }

An this is the Scottish gaelic one

public static function getPluralSuffixes($count) {
        if ($count == 0 || $count > 19) {
            $return =  array('0');
        }
        elseif($count == 1 || $count == 11) {
               $return =  array('1');
        }
        elseif($count == 2 || $count == 12) {
               $return =  array('2');
        }
        elseif(($count > 2 && $count < 12) || ($count > 12 && $count < 19)) {
                $return =  array('FEW');
        }
        return $return;
     }

Now compare with en-GB:

public static function getPluralSuffixes($count) {
        if ($count == 0) {
            $return =  array('0');
        }
        elseif($count == 1) {
            $return =  array('1');
        }
        else {
            $return = array('MORE');
        }
        return $return;
    }

This means that when we have in en-GB: COM_BANNERS_BANNERS_N_ITEMS_CHECKED_IN_0="No banner successfully checked in" COM_BANNERS_BANNERS_N_ITEMS_CHECKED_IN_1="%d banner successfully checked in" COM_BANNERS_BANNERS_N_ITEMS_CHECKED_IN_MORE="%d banners successfully checked in"

In Russian they need: COM_BANNERS_BANNERS_N_ITEMS_CHECKED_IN_0="Ни один баннер не был разблокирован" COM_BANNERS_BANNERS_N_ITEMS_CHECKED_IN_1="%d баннер успешно разблокирован" COM_BANNERS_BANNERS_N_ITEMS_CHECKED_IN_2="%d баннера успешно разблокировано" COM_BANNERS_BANNERS_N_ITEMS_CHECKED_IN_MORE="%d баннеров успешно разблокировано"

This means one string more in both cases for plurals. These, at last I saw, can't be added in Transifex and comments can't be read or lines uncommented

[Hackwar]

We can't remove com_weblinks from the core. What we can do, is making it possible to uninstall com_weblinks and provide a seperate update package for com_weblinks. Simply said: com_weblinks will be installed by default, but you can uninstall it and it is guaranteed, that a Joomla update does not add in those files again.

[Hils]

"And btw, forget .po. We use .ini."

@JM - isn't the JTracker Joomla project using .pot files? on Transifex?

[infograf768]

JTracker is not joomla...

[Hils]

I apologise JM - I used the name of the .pot file rather than the project

https://github.com/joomla/jissues/blob/master/templates/JTracker/g11n/templates/JTracker.pot

[javigomez]

I have been collecting ideas about what a translation system that can be seen here: https://github.com/joomla-projects/translate-joomla

Anyway, after thinking and thinking I always hit a wall: I think there is not the perfect solution there is always challenges. So my suggestion is that we start solving one by one the problems.

For me the very first step would be:

• Move translation packages from joomlacode to github, see this example repository: https://github.com/joomla-projects/en-US_j3. This move has several benefits:
  • allow translators to use the Github UI online editor, that is much more easier than using Joomlacode SVN, given that many of our translators are no developers (see http://docs.joomla.org/Using_the_Github_UI_to_Make_Pull_Requests)
  • Github ACL can provide us the possibility to define who can accept translations, and also allow community translations
  • But also Github can just act as a repository, so we could add later support for Transifex, Crowdin or whoever service the Translation Team prefers. And if that service doesn't support plurals we can always do it by hand in Github
• One translation repository could contain all kind of files: joomla CMS, com_weblinks... it will be up to a tool that generate language packages to separate them

The challenge for this move will be to:

• define the structure of a translation repository
• build the server scripts to create the packages
• Others...

For the future I always see something like what we have done with JIssues, a Framework app that works with our Git repos in the back and have a nice UI in the front.

I can help to do that move but I would need help because I don't feel able to do all this alone. Is anyone willing to?

My proposal would be to do the en-US package https://github.com/joomla-projects/en-US_j3, and use it as a proof of concept.

[dongilbert]

We can't remove com_weblinks from the core.

Yes we can. The PTL has already decided. Removing com_weblinks from core is perfectly acceptable. If it remains in core, and you remove those files, then update, those files will be replaced. You can't have your cake and eat it too (sorry for the old expression).

Removing com_weblinks is approved by the PLT and will take place in version 3.4, provided we work out these issues. It won't be deleted for existing installs meaning it won't break BC if someone is using it. It will not, however, be provided in the default Joomla! CMS installation package for new installs.

@infograf768 In my few years being with the Joomla Project, I pretty consistently see posts about the troubles the TT's have due to time constraints and the problem devs have with having a code freeze 2 weeks in advance of a release so TT's can catch up. I was simply offering a partial solution to these seemingly constant annoyances. If you want to continue on in frustration as it has been for the past 3 years (since I've been around), then that's fine, but don't say no one tried to help.

[infograf768]

@dongilbert I do not see how putting everything on Transifex (or any online tool) could help concerning time constraints... These will remain, whatever the way we work. The problem devs have with code freeze is mostly unrelated to language. I have always managed to commit some language strings before a PR/patch was fully finished, when necessary. We do most certainly need a Translation web app that fits our needs but also taking care of the com_localise component which needs a lot of love to be really useful. We would already save much time if that component was, at last, fully functionning.

@javigomez It is a good idea, but, as I told you in the TT Coordination chat, we would need to create a full github project per language and per J version. New files should be added when necessary all over the place (as quite a few TTs will not get git-aware enough to make PRs/Commits). Then we would not be able to make github work as a translation tool anyway and TTs will still need to get .diff. We would also need a way to build packs for a release ONLY when the Language coordinator thinks it is ready, but still be able to produce packs for testing in Joomla before releases.

To all: Until a real workable solution is proposed where we would not lose any language for any "core supported" extension, I suggest we keep the weblinks inis in core so that the language packs contain them.

[Bakual]

I do not see how putting everything on Transifex (or any online tool) could help concerning time constraints

It could help if the language strings are uploaded to the tool as soon as we commit them. Meaning you don't have to wait for language freeze to start translating. You would see them as soon as they are available in the repo. Of course some TTs likely will only start translating when we freeze language, but they at least could start earlier if they wanted. To my understanding that isn't possible currently, except if they watch the repo themself.

[eddieajau]

@nicksavov https://github.com/joomla-extensions/weblinks/wiki should answer your b/c concerns

@all the point here is to ensure we have the language files arranged in the easiest way for translators to deal with them. Selecting a tool to use is not the issue. However, if there are existing tools, we should take them into account in working out how to arrange the files. Does that make sense?

I'd also ask people to be willing to compromise on everything except the goals (that is, no extension files can remain in the core CMS download, new 3.4 installs must still work and you can't break Weblinks on existing sites when they upgrade).

[infograf768]

These are the goals you define. In this specific case I stand to my proposal that we should keep the weblinks ini in core until a working solution is effective.

Also, an aspect is apparently missing in this "strategy" for decoupled or "core supported" extensions, and that is the help. What will happen with it? Shall the "core supported" extensions get their help from the wiki as the rest of the CMS?

[infograf768]

Forget the last comment. I see now there is a documentation discussion.

[eddieajau]

In this specific case I stand to my proposal that we should keep the weblinks ini in core until a working solution is effective.

JM, this is not an option. All files are being removed from the core. What is the issue you are concerned about that you need language files to stay in the core?

Also, an aspect is apparently missing in this "strategy" for decoupled or "core supported" extensions, and that is the help.

I'm ten steps ahead of you :) See #3

[infograf768]

"JM, this is not an option. All files are being removed from the core. What is the issue you are concerned about that you need language files to stay in the core?"

If they stay in core, the diffs will show to TTs they can keep them in their packages. But we indeed could do as we did for geshi in some lang packs and keep including the files although geshi is no more provided. It would not solve updates to weblinks, if any, but better than nothing.

[eddieajau]

If they stay in core, the diffs will show to TTs they can keep them in their packages

I don't think the old "language package" is going to work once we decouple all these extensions. Also, we will have other extensions included under /joomla-extensions/ that were never added to the core anyway. So we are going to have to come up with another solution. Adding language files to the core for uninstalled extensions is not an option.

[toretto]

"I have seen 3pd translations done on Transifex (OT) using Google Translate... Unreadable."

JM, one of the benefits of Transifex is that you can change translations with problems or make a suggestion. It's an open process, really. We look forward to your feedback on those problematic translations.

[eddieajau]

Ok, let's try something different. Regardless of what tool is used, if all the translations are included in this repository, how am "I" going to know if it's ok to merge a Pull Request for a complete translation, or minor changes?

[phproberto]

Just a note: this is probably the biggest problem with removing extensions from core. Until this is decided we cannot move.

Translations is one of the things that made Joomla to lead the multilanguage market and that's thanks to the current people/structure. So let's try to listen to those people who know about it and do as much as possible to make them happy.

We can think on it as the new issue tracker. If current solutions doesn't work for our specifications we need to build our own tool.

I'm joining @javigomez in the https://github.com/joomla-projects/translate-joomla project to try to help with this problem. I hope some of you do the same.

[infograf768]

@eddieajau You are not... The only person we can trust the best is a TT coordinator for the language. This is why we would need a totally different approach. Separating core lang files from "core supported extensions" lang files needs a different workflow. Don't forget that we need updates as soon as possible from new releases. It is already hard to get in time for core, so separating core supported extensions would make it even harder

[marcospeebles]

So actually you're not at all preocupied by translations, but trust. No tool can give you that. Method will.

[eddieajau]

@infograf768 I never said it would be easy. But everyone's comfort zone is going to be stretched, not just yours. Let's work the problem because if we can solve it, we have a process "template" for every other extensions developer out there and I think that would be a huge win. This is going to be a challenge but lets take the attitude that we can solve all these problems in new and innovative ways.

Start throwing around some ideas about how the process can change.

[Skullbock]

I don't see why we have to reinvent the wheel every time we have a need in the project. There is a perfectly viable solution, Transifex, that solves the problem of "how do we manage the translations" problem

It works already for a lot of Joomla Extensions developers, who, arguably, have the same issues that com_weblink has in terms of language strings.

Just to point out a few things:

• Transifex deals with .ini files, not only .pot files. In fact, a lot of existing extensions already use transifex as their main repository and tool for language files
• Github is NOT the right tool for translations. It's code focused, not language focuses. Let's use the right tool for the right job.
• Transifex has both an excellent api and an excellent command line tool to manage translations. It can be easily integrated in any packacing / releasing workflow.
• It allows for "managers" to validate translations. This helps a lot in case of bad translations.

Just my 2c

[eddieajau]

Github is NOT the right tool for translations.

Of course it's not. But the completed language files (and you can use vi for all I care) need to be stored in Github to be able to make the build (the zip file that the user downloads). Given that, there are two choices:

1. We merge all available translations with the Weblinks package in this repository and Joomla handles the build and distribution process via the JED for you. This means we'll need to develop a new workflow to add translations via Pull Requests before the package is shipped.
2. Only the English files are shipped with the Weblinks package and each translator is responsible for managing and distributing their own language packs. All translation is done after the package is shipped and anyone can do whatever they like to produce them.

You will still do your core CMS language packs the same as always, but they will be a little lighter because they don't include the files for the Weblinks extensions.

I personally think option 1 is more convenient for the end-user, but it's less convenient for us because we have more work sorting out the new details (but this is a good problem to be solving).

[Skullbock]

I don't know if either of @eddieajau solutions include this, but transifex integrates nicely with github.

[eddieajau]

@Skullbock we don't care what flavour of tool is used (use a text editor if you want, it doesn't matter) - that is not what is being discussed. This is about what happens with the files when they are translated because the English files for Weblinks will not be shipped with the CMS from 3.4 onwards (and other components will be decoupled in 3.5). Making changes to this repository structure to make translation more convenient is also an option.

[phproberto]

My personal opinion is the same I had when the transition to github was proposed: use the existing tools instead of building our own wheel. But if those who currently manage translations think that it doesn't fit for our needs we don't have another option that develop our own solution.

We have to rely on our experts for each task. We cannot bypass them each time we don't like their opinions. Frontenders have to decide about frontend, translators about translations....

So let's try to find/discuss a solution that works and if not there isn't another option than build the wheel.

[Bakual]

@Skullbock JM already pointed out some issues with Transifex. Like it's hard to add more strings for plurals when needed. Let's just recognize Transifex isn't perfect (yet).

It allows for "managers" to validate translations. This helps a lot in case of bad translations.

This is true, however seldom used. But for core it could be a solution to cover the quality part. The workflow here is that one person proposes a language change and a second one needs to review it. It's also to note that in Transifex, not everyone can make changes. The person needs to be approved to the team first. OpenTranslators manages those teams, but you don't necessary need to use OpenTranslators for core extensions. You could as well build the TTs as they are today. Thus I think the quality isn't really a problem of Transifex itself and could be handled with right setup.

[Hils]

"It is already hard to get in time for core, so separating core supported extensions would make it even harder"

Translating extensions is different from translating core. Yes, core translations relate to Joomla releases and are a 'one off' job. Extensions are not. It can be a continuous automated process completely independent from any Joomla release. The difficulty the 'official' Translation Teams have is the inability to review easily, no Translation Memory for consistency and often relying on one person for the translation and review process.

@all Joomla could rewrite an existing wheel (and thank you to @javigomez & @Roberto for your work on this) but Transifex is a proven and constantly evolving automated but controllable system which even RedHat are using (and RedHat have also have written their own translation tool but consider Transifex to be superior). Transifex collects the feedback of thousands of translators across 10,000 projects, and continue to improve their product every day. They are also very open to suggestions for further improvements. They have around 400 languages enabled (OT currently has 89 teams with around 20 waiting due to being XX langs therefore no core translation, which uses xx-XX)

It doesn't matter to Transifex where the .ini files are stored in GitHub, they provide a Client to deal with that automatically and Daniel's cTransifex could automatically deal with them from there, if required.

[Skullbock]

@Bakual I Agree on the Quality issue.

As for the plurals, i really don't see an issue. What's stopping the translation file to add a new language string? Or from adding those strings directly in the base en-GB file, even if the translation is the same?

[infograf768]

@Skullbock Do you have all 51 languages that come with Joomla 3.x shipping with your extensions when using Transifex (OT or not)? Do you use these in your extension? https://github.com/joomla-extensions/weblinks/issues/2#issuecomment-39862370 Are you sure the syntax and consistency of the translations you have match with core packs?

@eddieajau Please refrain from posting that kind of stuff: "But everyone's comfort zone is going to be stretched, not just yours." You should know better... I have never cared about my "comfort". I have always cared (understatement) that we get as many languages as possible for Joomla in the best conditions and quality possible, and get follow up/updates along the versions and this since Mambo times. Let's say we have 60 core language packs. If we get only 30 for these "core supported extensions", I would consider it a failure for the project.

[Hils]

@Bakual "OpenTranslators manages those teams,"

This isn't factual. OT doesn't manage the teams or, except for a couple at the moment, approve translators. OT is a hub of independent translators and independent developers who gather together to increase translation resources for everyone who joins - like a cooperative.

[Bakual]

What's stopping the translation file to add a new language string?

Transifex is stopping them :smile: To my knowledge you can't have more strings in the translated file than you have in the source language. You would have to manually add them after you pulled down the file. Each time you do that. That's not going to work. But one could ask Transifex for a solution.

Or from adding those strings directly in the base en-GB file, even if the translation is the same?

You would have to cover each possible case in en-GB, which doesn't make much sense to me.

[Bakual]

This isn't factual. OT doesn't manage the teams or, except for a couple at the moment, approve translators. OT is a hub of independent translators and independent developers who gather together to increase translation resources for everyone who joins - like a cooperative.

@Hils True. I meant it right but have worded it incorrectly :smile:

[Hils]

@Bakual - no problem! Just wanted it right publicly :)