i18n - Translation Download

[kraftner]

I've created this issue so we don't mess up #29.

Quick summary of the status quo:

There is no complete composer based solution to get translations. The only thing that comes close is https://wp-languages.github.io/ which unfortunately only supports core and a small selected number of themes and plugins.

Thoughts

When building a replacement ourselves that plays nice with wpstarter we actually have multiple problems to solve:

1. Actually getting translations at all. We'd probably get it from the API. I vaguely remember seeing a PHP client for it somewhere... We'd also need a custom installer, something like the installer that https://wp-languages.github.io/ uses.
2. If I remember correctly translations don't really have sane versioning. You just ask the API for the newest translations for a certain version of core/theme/plugin and get what they have. This may change without WP versions changing as translations get updated independently. At least that is what I remember. Correct me if I'm wrong. :wink:
3. Translations are strongly tied to the WP version. Once we have composer packages for languages the most reasonable thing would be if they were dependencies of the core package itself, but that would mean convincing John of adding it. Otherwise you always have to keep versions in sync manually, right?

   WP-CLI as an alternative

I see that there is some talk about some integration with WP-CLI in #33. Maybe to keep things simple we could just build a wpstarter step on top of wp core language update. I know that this is not an ideal solution as it relies on the internal workings of WP update, but on the other hand it may just work without much hassle. What do you think?

[gmazzap]

First thoughts:

1. First of all, I don't want languages be a blocker for WP Starter version 3, so I'm glad this is is a separate issue and even if milestone says "3.0.0" I will not be surprised if this will not be part of that release.
2. https://wp-languages.github.io/ is good enough for core. For plugins not really so. However, there are ways to download from remote locations using Composer scripts, so if necessary this can already be done with some effort on writing a custom Composer script... With WP Starter v3 this sort of things will be much easier. In the future these automatisms for downloading languages could become part of WP Starter or a separate Composer plugin...
3. Yes, to download the proper translation for core you need to know the version that is actually installed. In a Composer script this is quite easy, here: https://github.com/wecodemore/wpstarter/blob/master/wpstarter/src/Setup.php#L176-L187 how it is done in main WP Starter script. The same thing can be done for each package, in case you need to know the version of specific plugin or theme.
4. Even if I really want to make WP CLI integration much easier, I don't think it is a good solution for languages. As far as I know, languages download is only available for core, but for core we are already pretty much covered with zero effort. Moreover WP CLI requires a completely working WP install, which may not be the case on site first deploy (if .env is not created yet).

[franz-josef-kaiser]

Yesterday night I thought through various ways of offering plugin/theme language files alongside a custom composer.json delivered as ZIP. So far I have some ideas, but no idea where to host such a thing as the only proper way is to run this as service on JavaScript. In every other scenario single server resources are needed and this so far sounds like it is the only chance to find some way to host this service anywhere for free. Note that I have not found anything that allows that as GitHub pages only allows static HTML.

[kraftner]

Okay so I found the API client I remembered. Guess what, Rarst wrote it. I haven't tested it but this might be an easy way to get translations in a Composer Script/Plugin.

I am with you, if we can make this work without WP-CLI - even better. Especially because of not needing to rely on a working WP then.

About @franz-josef-kaiser thoughts on hosting I think we can't really go with a normal composer package anyway, as the version ideally needs to be dynamically inferred from the versions installed. Or am I missing something there? This is also one of my issues with https://wp-languages.github.io/ You manually have to keep versions in sync. (And even then language files may change without the version changing if I remember correctly. This would totally break the normal Composer workflow. I'm gonna research now if I remember that correctly)

[franz-josef-kaiser]

I think we can't really go with a normal composer package anyway, as the version ideally needs to be dynamically inferred from the versions installed

@kraftner What I am thinking about is dynamically generated packages. The w.org API allows to fetch plugin and theme translation files by slug and version, so they can theoretically be generated. The only problem is that I do not want to set up a system that I have to pay for, so the main blocker is finding some SaaS services that provides a plan that allows to host a Nodejs application for free. Static sadly is not enough – unless I find a workaround for GH pages.

[kraftner]

I am not sure I do understand correctly what you are saying but I guess by dynamically you still mean that I'll have to define the version of the dynamically generated language composer package so it is the same as the WP version composer package, right? If yes I feel like this is still somewhat of a hassle. It would be cool if the correct version is picked up automatically without manually needing to keep it in sync with the WP version.

[franz-josef-kaiser]

@kraftner Mainly I am talking about plugin and theme language packs. Core languages is second priority as it already works somehow.

[gmazzap]

@kraftner I don't think that API client is a good choice. It comes with a (pretty outdated) version of Guzzle as dependency.

If I add that API client to WP Starter, all websites that used it will install Guzzle and then any plugin / theme / package that uses another verion of Guzzle can't be installed because a conflict with something that is not really in use in the website...

So I think that is not an option.

To don't introduce more dependencies, in Composer scripts we will need to use Composer utilitites, like Composer\Util\RemoteFilesystem.

[kraftner]

@franz-josef-kaiser Yes, but it is the same thing there for plugins and themes, right?

"wpackagist-plugin/akismet": "~3.1",
"koodimonni-language/akismet-de_de": "~3.1"

So when I move akismet to a newer version, e.g. ~3.2 I still need to manually keep it in sync with the language package. And do that for all other plugins and themes as well. What I am talking about is handling the language with a composer script/plugin or wpstarter itself so you only need to define wanted languages once and the rest happens automatically. Something like this:

"extra": {
        "wpstarter": {
           languages: ["de_DE"]
        }
}

And then we automatically fillter out all WP dependencies (johnpbloch/wordpress, type:wordpress-plugin, type:wordpress-theme), query the API for translations for them and download those.

---

@Giuseppe-Mazzapica Yeah, tried it - seems broken anyway. But you're right anyway, this is doable with Composer Utilities alone anyway.

[franz-josef-kaiser]

@kraftner Yes, that is what I want the end result to be. Additionally, I also want to clean up languages when a developer removes a package. My main problem at the moment is that Composer queries (sends a POST-request to) a registered repository just for a package.json and does not tell what package it is actually searching for. Meaning that I am still searching for a way to build lang packages on the fly.

[kraftner]

So after some discussion and attempts I think we can conclude that having a dynamic composer repository is just not doable at all as it contradicts the basic architecture of Composer.

Which probably brings us back to the Composer Script approach. This is doable, I have done a very rough sketch I'll clean up a bit in the coming days.

One edge case I came across is that if we auto-download translations via a Composer Script we might need a way to skip some plugins/themes. People might have their own translations or have naming collisions of private and public plugins/themes.

[Chrico]

I quickly want to jump in. I also think that providing some kind of Satis is here the wrong point.

The config driven aproach from @kraftner by defining the languages could be fine. Additionally WPStarter would have to parse the plugin-slug to fetch translations via offical API:

Endpoint: https://api.wordpress.org/translations/plugins/1.0/?slug={slug}&version={version}

By defning the languages via:

"extra": {
        "wpstarter": {
           languages: ["de_DE"]
        }
}

We can then iterate over those API-responses and load everything which is needed.

An even easier way to get translations would be just to use the URL to the translation ZIP-file:

https://downloads.wordpress.org/translation/{plugin|theme}/{slug}/{version}/{language}.zip

e.G.

https://downloads.wordpress.org/translation/plugin/akismet/4.0.8/de_DE.zip

[schlessera]

WP-CLI v2.0.0, which will be released in a few days, will include the following additions:

• wp language plugin install|update|uninstall
• wp language theme install|update|uninstall
• wp i18n make-pot

In case someone wants to help with the single missing piece for proper Composer workflows, here's the issue for adding a --prune flag to language core|plugin|theme uninstall: https://github.com/wp-cli/language-command/issues/22

[schlessera]

Another option: https://twitter.com/bjornjohansen/status/1028005191604490240?s=19

[gmazzap]

So, we had today a discussion at Inpsyde regarding this issue.

In our day-to-day work we deal the vast majority of times with plugins/themes which are not hosted on wp.org.

Now, those packages most of the times are shipped with languages. Which means that in the vast majority of cases a WordPress package falls in one these two options:

• is hosted on wp.org, so languages can be downloaded quite predictably directly from wp.org
• is hosted somewhere else, with translation files included, so download it is not necessary

However, I think there are some issues which are not very well addressed by existing solutions:

Issue 1: Plugins with code hosted on GitHub (or such), but languages in wp.org

It might seems a very strange case, well it is not. Nowaday many developers uses GitHub (or such) to actively develop plugins, but then push on wp.org at release time. When this happen, and the plugin developers support Composer, they need to ship the vendor folder on wp.org. If one wants to use such a plugin with WP Starter it is way better to use the GitHub version because that (likely & hopefully) does not contain the vendor folder, which is not necessary with WP Starter, and might even be harmful. However, the GitHub (or such) version, probably will not contain languages, because the developer will very likely use wp.org to manage those, and benefit from the (potential) help from WP community for translations. When this happen there's basically no way to predicably know from where download the languages. In fact, there's absolutely no standard or predictable way to know the wp.org plugin "slug" from the Composer package name in the format "vendor/name". Current version of MultilingualPress (v2.*) is an example of this situation. (And the plugin linked in last @schlessera comment would fail for it, because Composer name and wp.org slug does not match).

Issue 2: Plugins with code hosted on wp.org, but languages stored somewhere else

As you probably know, theres's a kind of "structure" on wp.org to manage languages. Not even the plugin/theme author is authorized by default to just add/remove translations. Moreover, sometimes it is desirable to use a custom translation instead of the "official" one. In my freelancing days, when I worked with many italian clients, it was not rare the case I wanted to use an italian translation made by me, because either special requirements, or the "official" one was not really good or even missing.

Issue 3: Plugin with code somewhere, but languages somewhere else

Different "premium" plugins (so not hosted on wp.org) uses shared translation services (WordBee, to mention one) to translate the plugin. I think because in this way they can somehow "open" translation strings, and so potentially get a wider audience or/and services from professional translators, without opening the code itself. This is a rare case, I think, but still.

Issue 4: Not all plugins are on wp.org

As already said, these days a plugin from wp.org is a rarity in sites I work on. If it is true that many custom packages already includes translations, it is also true that an automated task ran on composer install does not really have a default way to be aware of that. Which means that, very likely, an attempt to download the language from wp.org will be made. Of course the subsequent 404 can be handled nicely, but when a site has, for example, 30 custom plugins and 2 custom themes and 4 languages, do we really want to attempt 128 HTTP requests at every composer update just to find they will end in a 404? Not to mention that there's no guarantee that those requests will be actually 404. If a different plugin, but with a matching slug is found on wp.org the translation will be downloaded and because of the way WordPress loads translations (priority to the ones inside wp-content) that wrong translation will be loaded in production, even if it has nothing to do with the custom plugin which is installed.

Issue 5: Code and language version too coupled with version

First of all there should be some way to easily choose the "best" possible version of a translation for a given version of the plugin. After that, it should be possible to update a languages independently from packages.

Issue 6: Optimizations

If a given version of a language exists in the project folder, that should not be downloaded again. In a site with 4 languages , 30 plugins and 2 themes, 128 downloads would be run on every Composer update, not matter if not necessary, if there's no a check for existence in place. Besides of that, one the streght points of Composer is cache, but plainly downloading lang files doesn't have that benefit. Another streght point of Composer is the lock file. This is missed in many current solutions for languages.

Issue 7: Compatibility with WP Starter

WP Starter has a setting to configure the wp-content folder localtion. A solution to integrate in WP Starter, needs to respect that setting.

To sum it up...

To make a summary, a proper solution should:

• allow to define a source of translations that is different from the source of the code (this basically covers the first 3 issues)
• treat plugins who are not in wp.org as first class citizen. It has to be designed thinking that 99% of the packages will be not on wp.org (even if that percentace is not the real one)
• be as much optimized as possible: allow avoid unnecessary HTTP requests, maybe use a cache mechanism, and possibly benefit of a lock feauture
• be compatible with WP Starter.

Some of the existing solutions do (totally or partially) this things, but it seems to me that none of them addresses enough these concerns to be considered a good solution for WP Starter.

So... what to do?

Well I don't have a clear idea. An improved version of the Bjørn Johansen's plugin linked by Alain could be a starting point.

Besides of that, I would like something that allow the package itself to declare they way it handles translations so the the upstream (WP Starter) can read that information and act accordingly.

This could reduce a lot the uptsream maintenance. Imagine if the dependencies of each and any package used in a project should be defined at the root level. That would definetively not work. The same reasoning IMO can be done for translations, moving the burden of edge case handling from the root to the package.

[gmazzap]

I thought a lot about this in these days. Considering that:

• the matter is quite complex
• theres' work being made outside WP Starter
• WP Starter 3 introduces the concept of "WP Starter extensions"

I see this being solved outside WP Starter. Being it a new WP Starter extension, or the embracing of someone else tool... that's a decision that can be made after 3.0 release.

So I decided to close this issue, but this is on the top of the list of matters to tackle after 3.0 release.