[4.0] [RFC] Remove weblinks language files from core

[Bakual]

When we decoupled the weblinks package, we left the language files in core, mainly to make it easier for translators. For 4.0 it would be great if we can find a way to move those files to the respective weblinks extensions. Personally, I see two issues we have to solve:

• Ideally, language files (at least the en-GB source) from extensions are shipped within their own folder structure. Eg in /components/com_weblinks/language/. This allows to ship all available translations even if the respective language isn't installed (yet). It also allows overriding by placing the adjusted file into the global language folder (eg in /language). In case of weblinks we would need the fact that there are existing files from core and the various language packs present in the global folder, and those override the ones shipped in the extension. We would need to find a way to solve that. The other approach is to still install the language files from weblinks into the global language folder. This is possible and probably simpler, but means if a language isn't installed, the translation will not be copied and overrides can only be done using the override.ini file.

• The other (bigger) thing to solve is how translators will be supposed to fetch the files and translate them. Currently they just work from the files in core, and happily translate the weblinks ones as well since those are in core as well. If we move them to the weblinks repo, they have to be explicitely fetched from there and may get forgotten. In Crowdin, I can solve that as we can have files from different sources there without issues. But as we all know, not all translators use Crowdin (for various reasons).

• We also need to decide if the translations should go to the weblinks repo or if they should remain part of the CMS language pack. The latter is probably the simpler approach.

Pinging @infograf768 and @imanickam for input.

[yvesh]

Ideally, language files (at least the en-GB source) from extensions are shipped within their own folder structure. Eg in /components/com_weblinks/language/.

Hm, personally that's not feeling right to me, if you install weblinks the core file is overwritten anyways by the package one. (They should be in sync ofc).

We also need to decide if the translations should go to the weblinks repo or if they should remain part of the CMS language pack

Yep, as you said the later one is the simpler approach and we don't loose a lot of translations (and translators). For moving translations to an own project, we would need an automatic workflow for crowdin and web links.. Taking the translations from crowdin is easy, but many languages are not translated there..

The question is: Is losing translations / translators worth the cleanup? :)

[Bakual]

For moving translations to an own project, we would need an automatic workflow for crowdin and web links..

That part is simple. I already have a cronjob running (on my own server, but that's another topic) which syncs the cms repo to Crowdin each night. It's just a matter of extending that script so it also does the same for the weblinks repo.

The question is: Is losing translations / translators worth the cleanup? :)

It's not only a cleanup, it currently complicates the release process for weblinks as it is still tied to the CMS as soon as we have language changes. Eg there is currently a PR open which if accepted need a sync of the language files back to the CMS so translators can translate it. And that would have to be done before language freeze. Since that didn't happen for 3.7.3, weblinks is now stuck until at least 3.7.4 is released. It's quite a mess actually. So we really should get the weblinks language files out of the CMS.

[imanickam]

Following are my thoughts.

(a) Once an extension has been decoupled from core Joomla (in this case Web Links), the extension should be treated just like any other third party extension. The reason being the extension may not follow the same release pattern as that of core Joomla!

(b) The decoupled extensions' language files have to find the appropriate places (ie. admin language files in the directory \administrator\language\xx-YY and site language files in the directory \language\xx-YY).

(c) As for as logistics for the translations (of the decoupled extensions are concerned), they can be done just like the way we have been doing for the installation related language strings (the file xx-YY.ini in the folder \installation) and third party extensions that are integrated with Joomla (e.g., TinyMCE). In this approach, whether a user wants it or not, all the language files of the extension would be installed (case in point, the TinyMCE language files).

(d) IMHO, the appropriate question would be, when the extension does not follow the release pattern of core Joomla and the extension’s language string changes are there, what needs to be done?

(i) One approach would be to release the next version of the full language pack with the language strings changes of the extension. This would work only if the extension is compatible with the version Joomla! This would be an easier method. With this approach, the core language files will have the decoupled extensions’ language strings as well.

(ii) Another approach would be to create separate language packs for the decoupled extensions just like core language packs. It would be an additional work. But, it is a cleaner approach. This approach would work well, if the extensions do not follow the release pattern of Joomla! Also, in the tool such as Crowdin, if the language pack creation is automated, this could be a way of the future.

On Sun, Jun 18, 2017 at 11:15 PM, Thomas Hunziker notifications@github.com wrote:

When we decoupled the weblinks package, we left the language files in core, mainly to make it easier for translators. For 4.0 it would be great if we can find a way to move those files to the respective weblinks extensions. Personally, I see two issues we have to solve:

-

Ideally, language files (at least the en-GB source) from extensions are shipped within their own folder structure. Eg in /components/com_weblinks/language/. This allows to ship all available translations even if the respective language isn't installed (yet). It also allows overriding by placing the adjusted file into the global language folder (eg in /language). In case of weblinks we would need the fact that there are existing files from core and the various language packs present in the global folder, and those override the ones shipped in the extension. We would need to find a way to solve that. The other approach is to still install the language files from weblinks into the global language folder. This is possible and probably simpler, but means if a language isn't installed, the translation will not be copied and overrides can only be done using the override.ini file.

The other (bigger) thing to solve is how translators will be supposed to fetch the files and translate them. Currently they just work from the files in core, and happily translate the weblinks ones as well since those are in core as well. If we move them to the weblinks repo, they have to be explicitely fetched from there and may get forgotten. In Crowdin, I can solve that as we can have files from different sources there without issues. But as we all know, not all translators use Crowdin (for various reasons).

We also need to decide if the translations should go to the weblinks repo or if they should remain part of the CMS language pack. The latter is probably the simpler approach.

Pinging @infograf768 https://github.com/infograf768 and @imanickam https://github.com/imanickam for input.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/joomla/joomla-cms/issues/16753, or mute the thread https://github.com/notifications/unsubscribe-auth/AEt-fHcMxn4jx9ZKwc8WpZn6DmRKPCnmks5sFWJJgaJpZM4N9jvl .

[brianteeman]

The logic of c above escapes me. Weblinks is nothing like tinymce. One ships with joomla one does not

[Bakual]

@brianteeman I think he just meant the logistics part. Eg the TinyMCE files are not directly translated from the english one in our repo. Mostly they are taken from the official TinyMCE Transifex project (and then posted to the TT forum where I take them). So we already have files which are taken/translated from a different source.

[imanickam]

Thomas, Thanks for the clarification.

On Mon, Jun 19, 2017 at 10:58 PM, Thomas Hunziker notifications@github.com wrote:

@brianteeman https://github.com/brianteeman I think he just meant the logistics part. Eg the TinyMCE files are not directly translated from the english one in our repo. Mostly they are taken from the official TinyMCE Transifex project (and then posted to the TT forum where I take them). So we already have files which are taken/translated from a different source.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/joomla/joomla-cms/issues/16753#issuecomment-309509790, or mute the thread https://github.com/notifications/unsubscribe-auth/AEt-fPOjLzr9VSp7XfuBamXfGRBFuKX5ks5sFq-ygaJpZM4N9jvl .

[brianteeman]

Closed see #19307

[infograf768]

See https://github.com/joomla/joomla-cms/pull/19307#issuecomment-355728213

[infograf768]

Imho, original questions by @Bakual remain undecided and #19307 should NOT be merged until a process is defined.

[Bakual]

I'm reopening this issue as there was no decision taken yet and the points are still open. Removing the strings is just a small part of the issue.

[brianteeman]

the discussion about how to manage lthe translation of language strings in the decoupled weblinks really should take place either in the translation forum or the weblinks repo.

[Bakual]

The translation forum would really be the wrong place for this. The weblinks repo would make sense. But I don't see a reason to open the same issue in another repo and have the same discussion again. It belongs as well to the CMS as it does to Weblinks because depending on the decision it affects both places. So imho it is fine to discuss here.

@wilsonge @mbabker @rdeutz Not sure who should make that decision, In the light of the recent decision by OSM to use Crowdin as the translation tool, fetching the source could be solved if the decision is to keep weblinks as part of the official language packs. However this still ties the releases of weblinks and the CMS together to a degree.

[astridx]

I have the impression that not many programmers and users are interested in the web links repo at the moment. To simplify things, it may be best to officially support only the English language and to offer the last language files of the 3 version for download. For example, on Github in the area of joomla-extensions. So anyone who has an interest in a language could keep the files up to date on github for themselves and others. So the work that was put into these language files is not in vain. We can set a hint to this language files in the component and in the JED.

[Bakual]

@astridx If you want to keep the language files on Github, it could be done in the weblinks repo itself. If we move the files into the component language folder (eg /components/com_weblinks/language/de-DE/) they can be shipped together with the component. That would work. Drawback here is that translations are tied to the release of the component. And existing translations from the language packs in the global language folder need to be deleted during install (possible with an install script).

[astridx]

If you want to keep the language files on Github, it could be done in the weblinks repo itself. If we move the files into the component language folder (eg /components/com_weblinks/language/de-DE/) they can be shipped together with the component. That would work.

I would not do it in the component language folder, because in this case the web links team has to translate all languages, Because the files will be installed if you install a version 4. So a German (who has set up the German language in his settings) do not see the translations of the new language strings in version 4.

But maybe we can put the files in a subdirectory and explain in the file Read.me which language files have to be copied to where. This has the advantage that the translation of language files other than the English language is the responsibility of the user. The web links team only needs to ensure the English language. I think this has a good cost effort ratio.

Drawback here is that translations are tied to the release of the component. And existing translations from the language packs in the global language folder need to be deleted during install (possible with an install script).

If we use a subdirectory for the "old" language files, we do not need to deleted something, are we? We only have to make clear, that we only maintain English language files with Joomla! 4. That the translation is tied to the release is in my opinion not so bad, if we only maintain the English version.

[mbabker]

Putting the language files in the component language directory and removing the files from the global directory is perfectly suitable and I would suggest is the best option without having to write complex logic to be able to add the language files for a newly installed language (since the extension install API will only install language files there for known languages whereas using the component directory just lets you copy the directory from the package). Granted this is not a production tool but com_patchtester has been doing it that way since we introduced translations.

Honestly, I'm at the point where I'd rather just re-merge weblinks to core and be done with all the politics and bickering about procedures because clearly nobody cares that the current status quo is crap (nobody will address the issue with the language management, nobody will own the package and keep it maintained with even the current 3.x code, not even discussing work for 4.0 right now) and just never again suggest decoupling anything from core because clearly the project is not in a state where it can support decoupled efforts (anyone know where the Install from Web team is?).

[brianteeman]

As you say the status quo is not good. It's really simple. Someone has to make a decision but it seems to me that no one in a position to make a decision is prepared to do so. Someone has to.

[Bakual]

@astridx You wouldn't have to translate the files yourself. That could be done using Crowdin and fetch the translated files periodically or before release.

[ghost]

I'm at the point where I'd rather just re-merge weblinks to core

+1

[astridx]

@Bakual Do we need the language files from Joomla! 4 in all languages? Does not that just make it complicated?

Please correct me if I am wrong. As far as I know, web links were decoupled because we thought the component was not used by many. We wanted to simplify Joomla!. But: The component Web links is good for learning. And to learn one language version is enough. Should not we concentrate on web links supporting the new features of each Joomla! version? And for that there are not enough programmers at the moment who have the skills. I think every extra step makes things more complicated. Even if it looks simple at first glance. If someone wants to help, he must first be trained in Crowdin. And that's another hurdle ...

[rdeutz]

Just because we are not moveing fast enough towards our goal, doesn't mean the goal is wrong. The problem over the last year was that we tried to make it right for everyone and not to step on someone's toes. It is simple, we need leadership and decision.

[infograf768]

It is simple, we need leadership and decision.

Right, a decision on the process, not only for weblinks but for any "decoupled" extension.

Let's say a specific repository is created on crowdin for weblinks lang files. TTs would be able to translate there indeed, but most TTs do not do github and they would not be able to create PRs to the weblinks various language directories. That solution, a heavy one for the weblinks package (we talked about that in the past) as it would include files for each Core language, would need to be done by someone just before each Weblinks release. Who would be the "someone"? Fetching the various language folders from a weblinks package manually and placing them where they should by single users is not thinkable.

We therefore have 2 solutions:

1. One is coding an automatic merge of the lang files from a crowdin repo to the weblinks github repo and get updated or modified langs when a weblinks new version is released.
2. Get totally rid of weblinks for 4.0, i.e. not only the lang files but also the decoupled project itself (not B/C)

[brianteeman]

Let's say a specific repository is created on crowdin for weblinks lang files. TTs would be able to translate there indeed,

As they do with core

but most TTs do not do github and they would not be able to create PRs to the weblinks various language directories.

Why do they need to - they dont with core

[infograf768]

Why do they need to - they dont with core

Think more. TTs using Crowdin make packs after downloading their stuff from Crowdin (stuff not usable as is), which they then upload in a special repository where a cron job picks them. No github concerned as the lang files are NOT added to the CMS releases, contrary to the weblinks repo where these lang files would have to be merged if the decision is to include them in the weblinks language folders.

[brianteeman]

That's easily solved and every extension developer that uses crowdin has already solved it - no need for the TT to do anything different to what they do with core

[brianteeman]

If I am not mistaken we already have a script for this created by @mbabker and in use on other joomla project extensions translated on crowdin https://github.com/joomla-projects/crowdin-sync

[dgrammatiko]

which they then upload in a special repository where a cron job picks them

This sounds like a broken workflow to me! The only source of truth should be crowdin. Use their API, as @brianteeman also points out, end of story.

[brianteeman]

So whats the next objection?

[Bakual]

No need for any script. If we set up weblinks that way, we can use native GitHub support from Crowdin. It will automatically fetch the source strings and write the translation back to a (single!) specific branch in the weblinks repo. From that branch, a PR can be created and merged.

Core works different.

[Bakual]

This sounds like a broken workflow to me! The only source of truth should be crowdin. Use their API, as @brianteeman also points out, end of story.

You can't create the update stream from Crowdin. That's what JM referred to. And we still are not at a point where JDownloads is ready to serve language packs. So it still works from JC and that Cron.

[Bakual]

If we decide to move the weblinks language files into the component language folder of com_weblinks, I can set up Crowdin so it works. I can also do the PR for a script which deletes the weblinks files from the global language folder on the next weblinks update, so outdated files would be removed. That would also mean weblinks file will be removed from the CMS and its language packs.

[horus68]

Why not implementing a solution for short therm, and get time for better solutions? This discussion can get forever, as nobody creates a system to delivery translations. Also ignore the discussion if weblinks is useful or not for you. Que question should be limited to translation and project organization.

There must be a plan to be used for any decoupled extension: The Low tech version 1- The decoupled extension will have its own github project for code files AND the en-GB master files. (the actual files under developer team care) 2- A new Crowdin project will be created to allow translations to be done in future (should include an initial upload from actual translations) 3- Crowdin gets master files updates from Github (automated step) 4- Crowdin project will share same "Translation Memory" with Crowdin Joomla Core project, for consistency. 5- Crowdin creates an installable zip with each language (refreshed each automatic project rebuild) 6- Users must download and install again new packs on each version. And we can move along!

Note: When somebody find a way to have those language packs updated in administrative area (same as the actual core packs) then it should/could be implemented. It could be the same as most of us already do now with custom extension language packs (and a XML for the update server).

[brianteeman]

Has there been any progress on defining the process yet or are we going to be stuck with unused files forever @wilsonge

[rdeutz]

Internally in the maintaince team we requested a documentation "How to handle language files and language tags". This pops always up when there is a fight about translations and it get forgotton till the next fight.

[horus68]

I can only see 2 ways to go out of here:

• To implement my "low tech" solution to allow remove the files but keep the work done until now (see my comment above)
• A plain removal of the weblinks files and ignore any translators work done for those files.

BTW... the language packs are all waiting for a system to replace the "broken and crippled" Joomlacode download system.

[brianteeman]

Thanks @rdeutz I can close this then