[PROPOSAL] New policy for language tags

[rdeutz]

Situation today

Unused language tags aren't deleted and stay for an long time within the language files. This makes the management more complicated and while adding a new language we can't easy see what has to be translated and what is already not used.

Proposal

Unused language tags stay for 2 minor releases in the language files and then they will be removed.

Example

• Langtag not used in 3.4.x
• Stay for 3.4.x, 3.5.x, 3.6.x
• Deleted in 3.7.0 (or 4.0 whatever comes first)

  Reasons

• Better management of language tags
• People will not only update the language files and not the core CMS over a couple of minor Version. And if, it might be a sign that they need to update.

[andrepereiradasilva]

IMHO i think they should be removed in the next minor release if not used.

And lower joomla versions should not update the language package to a release upper than it's joomla release.

[brianteeman]

I agree - its crazy that a new language has to translate so many unused strings

Questions

1. We would then need a way to mark a language tag as unused since version x.y.z
2. We also need to do something now to mark those strings that are already unused and not just for future strings

On 15 April 2016 at 13:42, Robert Deutz notifications@github.com wrote:

Situation today

Unused language tags aren't deleted and stay for an long time within the language files. This makes the management more complicated and while adding a new language we can't easy see what has to be translated and what is already not used. Proposal

Unused language tags stay for 2 minor releases in the language files and then they will be removed.

Example

• Langtag not used in 3.4.x
• Stay for 3.4.x, 3.5.x, 3.6.x
• Deleted in 3.7.0 (or 4.0 whatever comes first)

Reasons

• Better management of language tags
• People will not only update the language files and not the core CMS over a couple of minor Version. And if, it might be a sign that they need to update.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[andrepereiradasilva]

IMHO it's an over complicated process that could be simply resolved by not allowing old joomla version to update their language packs to a release upper than it's joomla release.

Can anyone tell me what is the problem with this approach?

If they want to update the language packs ... update joomla first!

[brianteeman]

Nothing - it is brain dead that we allow someone on Joomla 3.3 to use a language pack for 3.5 and expect it to work. There is no logical justifcation for this. But I hit my head on too many walls

On 15 April 2016 at 13:48, andrepereiradasilva notifications@github.com wrote:

IMHO it's an over complicated process that could be simply resolved by not allowing old joomla version to update their language packs to a release upper than it's joomla release.

Can anyone tell me what is the problem with this approach?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-210450612

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[andrepereiradasilva]

i agree joomla should allow install lower language versions on higher Joomla versions of course. But the opposite does not makes sense to me.

I, make a change to my comment, as i can also agree in deprecating and remove it only in the next minor release so extension can have time to check if they are using some of those deprecated language vars.

[rdeutz]

Questions

1. We would then need a way to mark a language tag as unused since version x.y.z
2. We also need to do something now to mark those strings that are already unused and not just for future strings

re 1: we could move them down to the bottom of the lang files after a ; ##### UNUSED SINCE 3.4.3 #####

re 2. I have written a script to check all files and language tags if they are used or not, it isn't perfect because of the way some language tags are created but gives you a good idea

[brianteeman]

re 2 - there is a script already from @adaneinspain sounds like you have the same problem with combination strings as him

On 15 April 2016 at 14:18, Robert Deutz notifications@github.com wrote:

Questions

1. We would then need a way to mark a language tag as unused since version x.y.z
2. We also need to do something now to mark those strings that are already unused and not just for future strings

re 1: we could move them down to the bottom of the lang files after a ;

UNUSED SINCE 3.4.3

re 2. I have written a script to check all files and language tags if they are used or not, it isn't perfect because of the way some language tags are created but gives you a good idea

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-210461366

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[MATsxm]

TBH... both scripts (sorry @rdeutz :wink: ) are far from perfect in both ways... finding strings not deprecated and not finding some that are. It is a real nightmare, believe me.

[mbabker]

I've already said in the past that if we patch up https://github.com/joomla-projects/language-update-xml-generator correctly that newer language packages would stop showing up on older releases, but thinking about it some I honestly don't think the update system can handle it.

You have to account for the fact that not every 3.x version has the same features in the update system (IIRC 3.0.1 or 3.0.2 added the min_dev_level and max_dev_level attributes on the <targetplatform> tag and 3.5.0 added support for making the version attribute regex work with the full CMS version (3.4 and earlier will only check up to the minor version)). Also the *_dev_level attributes won't work effectively with a regex (if I have a 3.[012] version regex and max_dev_level attribute set to 4, it effectively limits the versions something can install on to 3.[012].[0-4].

Then you have to be able to show the correct version in every release. And that's where the trickery or ugliness starts. Essentially https://update.joomla.org/language/translationlist_3.xml would have to list every language package release for the entire 3.x series (not that big of an issue to do, it's just code generating it). But doing that IIRC would affect what's displayed in the language install screens for the install app and extension manager, so without a core patch to filter results those lists are going to go from ~60 items to a couple thousand in a heartbeat.

So long and short, at least from the update system's perspective, I don't know what we can do to filter/limit results and I doubt the TT's are going to impose a script.php file requirement on the packages to check versions on preflight and block installs that way.

[rdeutz]

@MATsxm that's fine, just spend some hours and to have a good environment to test is a challenge in itself

[MATsxm]

Worked HOURS on this question and it is a real nightmare... it's easier when fr-FR are still supporting language packs for GeSHi... you easely know then what to delete :stuck_out_tongue_winking_eye:

Also the question about merging some sys.ini with .ini that are the same.

[rdeutz]

@mbabker showing them is only one problem, but I can also download the package and install it, so without some kind of dependency management that will not work.

That's why I tried to get the "low hanging fruit" and solve the problem on the organisation level :-)

[andrepereiradasilva]

but I can also download the package and install it, so without some kind of dependency management that will not work.

Can't the package manifest have a limit in version too? Even if not, is a Joomla problem if a user manually does that?

[mbabker]

I can also download the package and install it

If you're going out of your way to download and install something that the core UI is filtering out as non-supported on your platform you deserve all the untranslated strings that come with it 😉

[rdeutz]

@infograf768 I would like to hear your opinion on this proposal, thanks.

[Bakual]

Imho it boils down to the updater as said already. As long as our language manager proposes to install the language pack for 3.5 in a 3.3 install, I wouldn't delete the strings. You would end up with incomplete translations. Since Michael thinks our updater can't handle it properly, I would look to change it fo J4.

As for deprecating, I wouldn't move the strings around. You could just add a comment behind the string (on the same line) and translation tools like Transifex and Crowdin would pick that up and show it to the translator.

[brianteeman]

Why should we care if someone installs a language file for 3.5 into an unsupported version of joomla

On 21 April 2016 at 21:23, Thomas Hunziker notifications@github.com wrote:

Imho it boils down to the updater as said already. As long as our language manager proposes to install the language pack for 3.5 in a 3.3 install, I wouldn't delete the strings. You would end up with incomplete translations. Since Michael thinks our updater can't handle it properly, I would look to change it fo J4.

As for deprecating, I wouldn't move the strings around. You could just add a comment behind the string (on the same line) and translation tools like Transifex and Crowdin would pick that up and show it to the translator.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-213100099

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[Bakual]

Because we don't intentionally want to break existing sites.

[brianteeman]

We wouldnt be - the user would be breaking their own site by chosing to upgrade a language pack without upgrading joomla

On 21 April 2016 at 21:36, Thomas Hunziker notifications@github.com wrote:

Because we don't intentionally want to break existing sites.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-213103636

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[rdeutz]

@Bakual it seems we can't fix it technically in a overseeable future so I looked for a solution on administration level

[mbabker]

How is a site being broken? Because a string isn't localized anymore? They'll still have the English string as a fallback. Considering we have language packages that AREN'T offering a 3.5.1 version it's safe to assume then anyone using old language packages on a newer version of Joomla have broken sites by that standard.

[Bakual]

Using outdated language packs on a newer Joomla installation is another topic. Not much we can do there. Using newer language packs on an outdated Joomla installation is the topic here. Yes it would fall back to english (is that the case for whole 3.x? Don't remember), that's true. However I don't see that as a desirable effect. Yes, they should update their installation, but we all know there are outdated sites out there.

Another point is that those strings also could be used by 3rd party extensions which would leave them with untranslated keys even when the installation is up to date. There wouldn't even be an english fallback.

Imho, the risk isn't worth the reward of having less to translate. Especially since we could already mark the unused keys as deprecated and translators would be able to skip them if they wish to do so. So basically we could have the reward already without any risk involved. Why remove them then within 3.x?

[mbabker]

Using outdated language packs on a newer Joomla installation is another topic. Not much we can do there.

Except it's relevant and pretty similar behavior except switching where the words "outdated" and "newer" are in the sentence. They still fall back to English if the key doesn't exist or the raw key if strings were ever removed just like a newer version would.

[Bakual]

The difference is that with an outdated language pack you don't suddenly loose a translation when you update. While in the other case they will loose translations when they updtae the language pack (which is completely unexpected).

[brianteeman]

Again why should we care if someone updates the language pack but does not update joomla.

On 22 April 2016 at 09:13, Thomas Hunziker notifications@github.com wrote:

The difference is that with an outdated language pack you don't suddenly loose a translation when you update. While in the other case they will loose translations when they updtae the language pack (which is completely unexpected).

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-213317002

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[infograf768]

Agree with moving obsolete strings after a line like this: [Obsolete strings] Adding between the brackets since Joomla version # would create more issues than solved as one is never sure of the milestone.

[brianteeman]

Just to be clear @infograf768 you are suggesting that all unused strings are moved to the end of their respective language file after a tag to show they are obsolete and without any information about the version they were removed in (I agree that could be hard historically). Do they still need to be alpha ordered

On 22 April 2016 at 09:59, infograf768 notifications@github.com wrote:

Agree with moving obsolete strings after a line like this: [Obsolete strings] Adding between the brackets since Joomla version # would create more issues than solved as one is never sure of the milestone.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-213339089

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[infograf768]

if we do not add a j version, alpha order them there looks logical.

Using brackets[...] instead of a comment ; ... will make them stand (and in com_localise, they will show in a specific slider)

[andrepereiradasilva]

hum brackets [...] is a ini file section. sure will not have problem with that? See https://en.wikipedia.org/wiki/INI_file#Sections

[infograf768]

Yep, totally sure.

[andrepereiradasilva]

ok then

[brianteeman]

If we just move the unused strings to the end with a comment above the block will any of the translation tools see that these dont need to be translated. If not then there is no point

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9932.}

[Bakual]

Just using a bracket title and moving them to the bottom will not solve anything for translators using Transifex/Crowdin/OtherTool. It only helps those who translate directly from the file (or use com_localise, our homebrewed translation component)

For those tools you need to add a comment on the same line after the string, then it will be shown as context.

[MATsxm]

You can hide them on Crowdin at least... not a big deal if you know the ones to.

[brianteeman]

I suspected that might be the case

[MATsxm]

to clarify and on Crowdin (I didn't check the other platforms). -You can make them NOT visible for translators (only managers can see them). -Indicate a context and thus that a specific string must NOT be translated or as "optional" etc. -Integrate them in a glossary to clarify that they should not be translated (we can specify for some specific languages only and I tested it without finding the limit of the number of characters then you can add long strings).

[Bakual]

You can hide them on Crowdin at least... not a big deal if you know the ones to.

Manually, yes.

[Bakual]

Indicate a context and thus that a specific string must NOT be translated or as "optional" etc.

If we mark the strings in the source file like this: EXAMPLE_DEPRECATED_KEY="Some deprecated stuff" ; Deprecated since 3.5.0 Then that comment will show as context in Crowdin (and Transifex afaik)

[MATsxm]

You can hide them on Crowdin at least... not a big deal if you know the ones to. Manually, yes.

This is really not a big deal... even I did it on other projects :wink:

[ot2sen]

Unused and obsolete strings has nothing to do in the core language files, in the same way as unused and obsolete code has nothing to do in the core package. With +250 language files and thousands of strings, it should at all times only be valid and in use strings present in the files, for the current release.

Volunteer translators and language maintainers should not worry about how people use joomla the wrong way ~ outdated versions. We expect and recommend users to always use latest release of joomla, which include to use latest release of the language packages. So a promt delete of an unused string when found should be the standard, IMO :)

On Fri, Apr 22, 2016 at 9:29 PM, Marc-Antoine Thevenet < notifications@github.com> wrote:

You can hide them on Crowdin at least... not a big deal if you know the ones to. Manually, yes.

This is really not a big deal... even I did it on other projects [image: :wink:]

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-213562207

[brianteeman]

+100

On 22 April 2016 at 22:26, ot2sen notifications@github.com wrote:

Unused and obsolete strings has nothing to do in the core language files, in the same way as unused and obsolete code has nothing to do in the core package. With +250 language files and thousands of strings, it should at all times only be valid and in use strings present in the files, for the current release.

Volunteer translators and language maintainers should not worry about how people use joomla the wrong way ~ outdated versions. We expect and recommend users to always use latest release of joomla, which include to use latest release of the language packages. So a promt delete of an unused string when found should be the standard, IMO :)

On Fri, Apr 22, 2016 at 9:29 PM, Marc-Antoine Thevenet < notifications@github.com> wrote:

You can hide them on Crowdin at least... not a big deal if you know the ones to. Manually, yes.

This is really not a big deal... even I did it on other projects [image: :wink:]

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub <https://github.com/joomla/joomla-cms/issues/9932#issuecomment-213562207

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/9932#issuecomment-213593809

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[Bakual]

Unused and obsolete strings has nothing to do in the core language files, in the same way as unused and obsolete code has nothing to do in the core package.

If we threat it the same as the code, that would mean proper deprecate and removal with 4.0, and is what we do as of today and follows SemVer.

[mbabker]

Why is there such a strong insistence that language packages are 100% B/C forever? I can install the 2.5 packs on 3.x, and the 3.x packs on 2.5. Why doesn't Joomla support that too and go ahead and pull back all the 2.5 strings that don't exist in 3.x? IMO the "it doesn't show up in the install list so users obviously won't do it" argument is invalid; some comments up we talk about users being able to do just that even if it's figured out how to filter that screen.

Somewhere along the line you have to draw a line. You have translators continuing to translate strings which are not part of the core package for the sake of someone still running Joomla 3.1.1 being able to install the Joomla 3.5.1 language package and not have an untranslated string. Why?

I've been told the GeSHi strings are still included in some language packages. That plugin isn't even part of 3.x yet is found in 3.x packages. Why?

[Bakual]

Personally I look at language packs the same as I do at my extensions. I have to decide what my extensions minimum requirement is and enforce it. If I want my extension to be installable in 3.0 up to 3.5 then I need to make sure it's compatible with all versions. Or I can take the easy route and say my extension will only run at the latest version and enforce it in the Joomla Updater (and install script). An extension which only runs with the latest Joomla version but offers an update and is installable on all previous versions as well just doesn't look like it's properly done to me. Same with language packs. If the update shows and can be installed, I just don't expect it to "break" things. And it also never did so far. So if suddenly translations will be missing, that would look like a bug to me.

Let's start marking strings as deprecated so we see what we are talking about. Amount and type would be important factors to make a decision. Also we need to be 100% sure the string isn't used anymore. Which isn't that trivial to check as it may sound (for some strings, it will not show up in a simple search).

[Bakual]

I've been told the GeSHi strings are still included in some language packages. That plugin isn't even part of 3.x yet is found in 3.x packages. Why?

I think that's the French language which has it still. I'm not sure if there are other languages still translating that one. At least not those using Crowdin :smile:

[mbabker]

Or I can take the easy route and say my extension will only run at the latest version and enforce it in the Joomla Updater (and install script).

That's not the easy way. That's the way it should always be done (replace "latest" with "minimum"). So in theory you need minimum Joomla 3.5.1 to use the 3.5.1 language pack. That CAN be enforced easily with a script.php file, but it mandates then that TT's update a second PHP class and I've heard of general reluctance to mandate TT's work with PHP.

What can't be done easily is dealing with the updater. Sure, we can patch it for newer versions, but what's there now is probably the best you're going to get for the already released 35+ 3.x releases. And I don't agree that because the updater most likely cannot effectively block seeing newer releases is a reason why 3.x language packs need to be 100% B/C to 3.0.0 (FWIW before 3.0.1, which includes all 2.5 releases since the change wasn't backported, you couldn't even filter a release to not be displayed on specific patch versions with the updater and I will still say that this technology limit shouldn't be a primary reason why you don't do something when the updater is involved). It's mainly the collection adapter that's the issue; from what I know the single extension adapter would handle our UI requirements just fine.

TL;DR... The update system's limitations and not using common subsystems such as script.php files shouldn't be the primary reason you say "lang packs must be 100% B/C forever".

[Bakual]

That's not the easy way. That's the way it should always be done (replace "latest" with "minimum"). So in theory you need minimum Joomla 3.5.1 to use the 3.5.1 language pack. That CAN be enforced easily with a script.php file, but it mandates then that TT's update a second PHP class and I've heard of general reluctance to mandate TT's work with PHP.

I agree with that. In my own extension I use a simple code block in the preflight method which checks the version attribute in the XML and if the current Joomla version isn't compatible with that attribute it returns false, terminating the installation. Eg https://github.com/Bakual/SermonSpeaker/blob/master/com_sermonspeaker/script.php#L47-L58 We could so something similar for the language packs and translators wouldn't have to deal with PHP at all since you don't need to change anything there. Just adjust the attribute in the XML (which can be copied from the en-GB file.) For those languages using Crowdin as the translation tool, that would be simple to implement. For those using com_localise it should be doable as well I think. For those using other tools or an editor, manual work would be involved.

[infograf768]

@Bakual What kind of attribute would it be? And where? In the pkg_xx-XX.xml or both install.xml? Would it have any effect for the list of languages proposed to install/update in J UI?

[Bakual]

I'm not sure if the package installer supports a script file. I would use the one from the languagepack itself, eg in https://github.com/joomla/joomla-cms/blob/staging/administrator/language/en-GB/install.xml#L2 the version attribute. If we would only remove the keys with a minor release, then it wouldn't change anything for translators as they are supposed to change that version attribute anyway to the current minor release (as far as I know).

[mbabker]

I'm not sure if the package installer supports a script file.

The package installer has supported script files since Joomla 2.5.0.