w3c / manifest

Manifest for web apps
https://www.w3.org/TR/appmanifest/
Other
653 stars 159 forks source link

Add optional `translations` member #676

Open FluorescentHallucinogen opened 6 years ago

FluorescentHallucinogen commented 6 years ago

Sometimes it's difficult to find an app or game on a specific (especially, unpopular) language.

Conversely some apps (games) are available only in one language (e.g. Chinese).

Since we want PWAs to be presented in app stores, what about adding optional "translations"/"languages"/"supported_languages" array of strings member?

Something like:

"translations": ["en","fr","it"]

It should help to find (filter/exclude) apps by supported languages.

marcoscaceres commented 6 years ago

Sometimes it's difficult to find an app or game on a specific (especially, unpopular) language.

Ok, let's start here before talking about solutions. Generally, to find an application, the user goes to a search engine (or app store) and searches for that app. The search engine may know the user's language preferences or the search engine derives the language from geoip or whatever.

Which search engine or app store is failing to do this?

CheloXL commented 5 years ago

@marcoscaceres how would I describe in my manifest that my app supports multiple languages? Right now, there is only the lang entry (and that's for specifying the primary language). I'm ok with that (after all, the app HAS to display something if the user is requesting a language not supported by the app), but it would be nice if the manifest can also tell search engines/stores right upfront which languages the app supports.

marcoscaceres commented 5 years ago

@marcoscaceres how would I describe in my manifest that my app supports multiple languages? ... but it would be nice if the manifest can also tell search engines/stores right upfront which languages the app supports.

But as you say, it falls into the "nice to have" bucket. The problem is difficult: we made and explicit design choice when creating the manifest format that it shouldn't be a place to make claims, such as "this application supports French".

Rather, we want the applications themselves to support languages in their content. Search engines can then discover that content through various means, like following links (same as a normal user).

marcoscaceres commented 5 years ago

We have some i18n guidance here: https://www.w3.org/TR/appmanifest/#internationalization

marcoscaceres commented 5 years ago

fixed a typo above, I was supposed to say "shouldn't be a place to make claims". Apologies for that.

mgiuca commented 5 years ago

I've got similar concerns about this (and just this week, we had a team having trouble localizing their manifest). I'm starting to think we could explicitly support translations of user-visible strings (like the name and description) in the manifest itself, which would implicitly give stores an indication of what languages the app supports.

The link above gives two localization suggestions:

  1. Dynamically setting the language (using in-site UI to point at a different manifest URL for each language).
  2. Using content-negotiation, or geotargeting, etc. on the server (serving a different manifest contents based on the user's chosen language).

Neither of these are great options. Option 1 interferes with the idea that "manifest == app". Depending on how the implementation keys apps off manifest URLs, changing the URL (even changing the query string) could cause the user agent to think of it as a different app. For example if you install manifest.php?lang=en on Chrome on Machine A, then install manifest.php?lang=fr on Chrome on Machine B, then Chrome Sync happens, we may consider these different apps and you'd end up with both versions on both machines. This is user-agent specific issue, but only because we haven't properly specified how apps are keyed. (#586)

Option 2 fixes that, but it means you have to have dynamic logic on the server side, which precludes any static hosting solutions like github.io. (Whereas the rest of the site UI can be localized in JavaScript, the manifest data cannot be.)

Finally, if the language solutions involve literally changing the bytes of the manifest, it means that when the user changes language, apps will remain in the originally installed language until the user agent re-pings the server with the new Accept-Language, redownloads the manifest, and updates the metadata. (On Chrome Desktop, we haven't even implemented updating yet, so you're stuck with your original language until you update.) Whereas if the manifest declaratively included all the translations, the user agent could automatically update all the UI with the correct strings as soon as the user changes language.

Therefore, I think it would be better if we actually let you put your localizations in the one manifest file itself. What are the downsides of this (besides potentially a much larger file)?

CheloXL commented 5 years ago

I'm on the same boat as @mgiuca right now. I'm using content-negotiation to provide a localized version of the manifest/app. This works fine for the user (people don't usually change the browser language on the fly... they don't change it at all, or they setup it once and be with it). But that tells nothing to a search engine. Search engines will not hit my app with different languages just to see if I'm providing content-negotiation. Why this can't be handled in the same vein as images? So, you have a "languages" entry (array of objects) where each entry can have a lang tag, dir, short_name, name and description (I'm not even sure if short_name/name are really necessary, as usually you don't translate the name of your app). Of course the main/default ones are the ones outside the languages entry. This means nothing for implementors (they don't have to do anything with this entry) but helps discoverability of the app, both in a store/by a search engine.

marcoscaceres commented 5 years ago

But that tells nothing to a search engine.

I’m still a bit confused as to which search engine or store are you referring to specifically?

CheloXL commented 5 years ago

Search engine? All of them. I'm not aware of any search engine that does language negotiation.

When a SE hits my app, my app tries to negotiate language based on accept-language header. Nothing is sent by the SE, so my app defaults to English.... and that's it. Even if the SE sends an accept-language header, it will send one. I'm quite sure that the SE will not try to see if my app changes what it returns by changing the headers.

The SE has no way to know my app supports multiple languages. There are no hints anywhere. If the manifest had a place where I can define them, the SE can read it and use that meta-information for indexing purposes.

marcoscaceres commented 5 years ago

Search engine? All of them. I'm not aware of any search engine that does language negotiation.

Sorry, I wasn't very clear: Which search engine processes/supports web manifest format?

I ask because we are reluctant to add things to the web manifest format unless we have actual commitment implementation from a search engine or store.

Put differently, imagine we add this today without explicit support from a search engine: One year later, nothing changes because no actually implements the feature.

When a SE hits my app, my app tries to negotiate language based on accept-language header.

Ok, but do you also provide a manual way for the user to switch language? For instance, I speak multiple languages, I might prefer my work stuff in English, but my social apps in Spanish.

marcoscaceres commented 5 years ago

@mgiuca wrote:

Option 2 fixes that, but it means you have to have dynamic logic on the server side, which precludes any static hosting solutions like github.io. (Whereas the rest of the site UI can be localized in JavaScript, the manifest data cannot be.)

This is true. That assumption we originally made was that the user would need to re-authenticate after the application syncs. When they re-authenticate, they would pull down their app-specific profile data, including their language preferences.

CheloXL commented 5 years ago

Sorry, I wasn't very clear: Which search engine processes/supports web manifest format?

Ah... Right now? I don't know if any. But SE originally neither indexed flash and at some point they started to.

Put differently, imagine we add this today without explicit support from a search engine: One year later, nothing changes because no actually implements the feature.

I get that, but the same can apply to other features that were suggested to be added (screenshoots, masked icons, etc). Since this is a living standard, it is expected to have features that nobody had implemented.

Ok, but do you also provide a manual way for the user to switch language? For instance, I speak multiple languages, I might prefer my work stuff in English, but my social apps in Spanish.

Right now, no. Changing the language means not only to refresh the whole app, but also expires cached assets on SW, the SW itself, etc. In a future, probably, but in any case, the app is behind a login page, so even if we let the user change the language, that means nothing for search engines. Also, the only way for a SE to be able to index different languages of the app is to have different URLs for each language/page, and while for a website that works perfectly (and I have plenty of experience on that), for an app this only add a lot of complexity to the developer.

marcoscaceres commented 5 years ago

I get that, but the same can apply to other features that were suggested to be added (screenshoots, masked icons, etc). Since this is a living standard, it is expected to have features that nobody had implemented.

So, no, fortunately. For instance, the Microsoft Store folks are the ones that requested Screenshots. And masked icons was added because it's already implemented in Android, so there was fairly clear indication from people working on the Android side that they would be willing to implement that. The same for all other things we've added: they are all implementer backed.

However, that doesn't mean we can't specify a solution. It just means that we might then need to do some work to pitch it to either browser vendors and/or search engines.

Also, the only way for a SE to be able to index different languages of the app is to have different URLs for each language/page, and while for a website that works perfectly (and I have plenty of experience on that), for an app this only add a lot of complexity to the developer.

So, I think this is the problem we need to solve generally. The HTML spec suggests using rel="alternate":

<link rel=alternate href="/en/html" hreflang=en type=text/html title="English HTML">
<link rel=alternate href="/fr/html" hreflang=fr type=text/html title="French HTML">

The question is if the above insufficient to meet the use case, or if we do need to specify yet another mechanism.

mgiuca commented 5 years ago

Good question / possible solution. I think this solves a lot of the problems outlined above:

  1. It means we have a canonical manifest URL across all languages.
  2. It means developers can present all the languages on a static hosting provider (no server-side logic).

I think a problem with that approach is that user agents may wish to download all the languages up front, rather than just the user's preferred language.

The reason a UA would do this is to create a fully internationalized representation of the app in the native OS's preferred form, so that if the user switches OS language, the web app seamlessly changes language as a native app would. For example, on Linux we'd generate a .desktop file with all the localizations in it. On Android, we'd generate an Android Manifest with the same. Thus, switching language doesn't rely on the user agent to realise that the language has changed and fetch the language-alternative resources.

So in order to create, say, the localized .desktop file, we'd have to go and fetch every single rel=alternate manifest from the server, creating a lot of requests up front.

Also, would we allow the "alternate" manifests to have just the localized strings (augmenting the base manifest), or would they need to be a complete copy of the entire manifest? That could be a lot more total bytes to download than just having all the strings in one. [Edit: From the HTML spec, it says "referencing an alternate representation of the current document", so I don't think we could interpret it as "just provide the keys you want to override", it would have to be a complete copy.]

An approach we came up with is splitting it into two separate files: the manifest and a separate "translations" file with all the translations. That way, if you need to fetch the manifest for just the icons (or another non-string resource), you don't need to fetch the potentially much larger translations file.

marcoscaceres commented 5 years ago

For example, on Linux we'd generate a .desktop file with all the localizations in it. On Android, we'd generate an Android Manifest with the same. Thus, switching language doesn't rely on the user agent to realise that the language has changed and fetch the language-alternative resources.

I think this is interesting - and I think it's something we looked at really early on in the standardization process (back in 2014). We could see about salvaging parts of #211.

mgiuca commented 5 years ago

I would be in favour of bringing in something like #211 proposes, where any string can be replaced by an object with multiple languages in it.

I am concerned with the manifest becoming too large (since Chrome, at least, now blocks on fetching the manifest before showing a tab icon, b/c we get the icon from the manifest now). So we might consider putting language-specific overrides in another file.

marcoscaceres commented 5 years ago

It would be great if you can explore alternatives... might not be a big deal in a HTTP/2 or /3 world, but splitting the file might still actually be slower in a HTTP 1.1 world (specially if all the alternatives need to be downloaded). We might need to do some experimentation.

To keep backwards compatibility, we might need to do:

{
  "alternatives": {
     "en": "path/to/en.manifest",
     "fr": "path/to/fr.manifest",
     "it": "path/to/it.manifest"
   }
}
marcoscaceres commented 5 years ago

The above also retains compatibility with the advice in the spec, I think.

marcoscaceres commented 5 years ago

@mgiuca, just wanted to follow up on...

I am concerned with the manifest becoming too large (since Chrome, at least, now blocks on fetching the manifest before showing a tab icon, b/c we get the icon from the manifest now).

I'm also concerned about maintainability. The nested object get really hard to read/write - so I'd be in favor of separate resources.

mgiuca commented 5 years ago

I'm also concerned about maintainability. The nested object get really hard to read/write - so I'd be in favor of separate resources.

As in one-file-per-language, or a separate file for translations?

I will write up an analysis of the pros and cons of each method.

marcoscaceres commented 5 years ago

yes, "one-file-per-language". I think a single translation file with multiple languages might be difficult to maintain.

mgiuca commented 5 years ago

I wrote up a doc with the various options:

https://docs.google.com/document/d/1Vwi_u7RP-HrzCz389Qp4EPKrWhanetu8ivcJmGbhwFc/edit

Please comment.

CheloXL commented 5 years ago

I like option 3, either A or B. With B you don't even have to change anything on the manifest and I think it's better for SE, but again, both are good.

One thing to note (and I don't see that in the document) is that the localized manifests could be only a subset of the original manifest, with changes only on the keys you want to localize.

So, for example, the usual keys would be title, short_desc and desc. You would end up with a complete manifest in your default language, and N manifests for each translation, that only have 3 key entries. But this opens the possibility to also have different in-app screenshoots, for example. That is key for app-stores, where you would probably want to display the screenshoots in the same language as the store itself.

I like it...

nuxodin commented 5 years ago

We have the "related_applications"-memeber Would it make sense to add a lang attribute there?

"related_applications": [{ "platform": "Web", "lang":'de-CH', "url":"example.com/de/ch/manifest.json" }, ...

marcoscaceres commented 5 years ago

Great summary, thanks @mgiuca for putting that doc together! I remain a fan of 3.

mgiuca commented 5 years ago

@CheloXL

One thing to note (and I don't see that in the document) is that the localized manifests could be only a subset of the original manifest, with changes only on the keys you want to localize.

That was the intention (which is why I showed "icons" in all the manifests) but I've added a paragraph explicitly stating this.

So, for example, the usual keys would be title, short_desc and desc. You would end up with a complete manifest in your default language, and N manifests for each translation, that only have 3 key entries. But this opens the possibility to also have different in-app screenshoots, for example. That is key for app-stores, where you would probably want to display the screenshoots in the same language as the store itself.

I actually don't think we should allow this. If you allow it, you have to mandate it (since we can't have different user agents doing different things). If we're talking about all members being localizable, then that means, for example, we have to allow the possibility that English and French have different scope URLs, and that means we might need to update our link capturing logic when the user changes their OS language. That is a huge complexity spike for us.

We could allow it for a few additional members, like screenshots and maybe icons (though even icons invites a lot of difficulty in the user agent).

@nuxodin

We have the "related_applications"-memeber Would it make sense to add a lang attribute there?

"related_applications": [{ "platform": "Web", "lang":'de-CH', "url":"example.com/de/ch/manifest.json" }, ...

As an alternative to "translations" in my option (3a)? I don't really want to do this. That implies that each translated manifest is a separate application. For one thing, these language-specific manifest files are not supposed to be complete manifests, but supplemental. For another thing, the whole point of this feature request is so that we can start thinking about different languages as views of the same app, not separate apps.

A new member for a new purpose.

kenchris commented 5 years ago

I don't really want to do this. That implies that each translated manifest is a separate application.

Yes, that would be pretty bad

CheloXL commented 5 years ago

I actually don't think we should allow this. If you allow it, you have to mandate it (since we can't have different user agents doing different things). If we're talking about all members being localizable, then that means, for example, we have to allow the possibility that English and French have different scope URLs, and that means we might need to update our link capturing logic when the user changes their OS language. That is a huge complexity spike for us.

We could allow it for a few additional members, like screenshots and maybe icons (though even icons invites a lot of difficulty in the user agent).

@mgiuca I understand, and in fact, while I didn't comment on the icons sections, I also think that icons should not be localized. Never though of having different scopes URLs. Thanks for pointing it out.

marcoscaceres commented 5 years ago

That is a huge complexity spike for us.

Ah yeah, forgot about that 😅 I think that’s why we explicitly called out the localizable members in the first place.

FluorescentHallucinogen commented 5 years ago

@CheloXL

I also think that icons should not be localized

Many companies use different logos for different markets. For example, VK (ВКонтакте), the most popular website in Russia and the 9th most visited website in the world, uses the different icons in their app for Russian-speaking and other countries:

vk-logo-ru

vk-logo-en

And yes, they have PWA: https://pwa-directory.appspot.com/pwas/5739494557089792. :blush:

mgiuca commented 5 years ago

Thanks @FluorescentHallucinogen for that useful insight.

I guess we should try to make icons localizable then. Any other fields should be localizable? screenshots, and maybe categories? (Are categories supposed to be like programmatic identifiers, so "sports" would be used across all languages and be properly translated by stores? Or should that also be localized?)

That brings us with the set of localizable members:

Malvoz commented 5 years ago

+ dir?

mgiuca commented 5 years ago

@Malvoz Oh yeah, definitely. (Edited)

Malvoz commented 5 years ago

Are categories supposed to be like programmatic identifiers, so "sports" would be used across all languages and be properly translated by stores?

If that's the case, I assume the dir member with a value of auto could resolve in an incorrect value if the categories member is the first member of the manifest and the values of the members are written RTL? Then just ignore it in the first step of (https://www.w3.org/TR/appmanifest/#dir-member)?:

The steps to programmatically determine the directionality of a member are as follows. The algorithm takes the value of a member.

  1. Find the first character (in logical order) of the value that is of bidirectional character type L, AL, or R [BIDI].
  2. If such a character is found and it is of bidirectional character type AL or R, return "rtl".
  3. Otherwise, return "ltr".
mgiuca commented 5 years ago

@Malvoz I don't think that's the case.

Firstly, categories is not "directionality-capable" so is not affected by dir at all.

Secondly, if it were, it doesn't apply to whichever member comes first in the manifest (in fact, I would consider it a bug if anything depends on the order of members in the manifest). It applies to each member individually.

Last (and this is getting a bit off topic here)... I'm not sure why this auto clause exists in the first place. This is already covered by clause P2 in UAX #9. I might file a bug to remove this apparently redundant text.

marcoscaceres commented 5 years ago

We should probably continue to treat categories like it lists enum values (i.e., just keep the categories in English). I don't have a strong opinion if they could change value based on locale, but I guess so.

aarongustafson commented 5 years ago

Are categories supposed to be like programmatic identifiers, so "sports" would be used across all languages and be properly translated by stores? Or should that also be localized?

@mgiuca When I initially conceived of categories, I included translations in the example array. I can't find the issue right now, but I demonstrated it in this example:

{
  "name": "Gojiro",
  "description": "Gojiro, a freak mutation with a cynical worldview, suffers the pain of solitude as well as several maladies experienced by entertainers, including drug abuse and suicidal tendencies.",
  "icons": [{
    "src": "images/icon.png",
    "sizes": "192x192"
  }],
  "categories": ["books", "fiction", "science fiction & fantasy", "kaiju", "怪獣"]
}

Here was my write up from when it landed too.

aarongustafson commented 5 years ago

Found it: https://github.com/w3c/manifest/issues/569#issuecomment-298650532

universewrld commented 4 years ago

My site has 2 languages - different name, different description and different logos. What method can you recommend for me to create file Manifest (PWA)?

marcoscaceres commented 4 years ago

My site has 2 languages - different name, different description and different logos. What method can you recommend for me to create file Manifest (PWA)?

If your user has the option to explicitly switch language, you can update the href of your <link rel="manifest"> to point to localized manifest.

aarongustafson commented 3 years ago

This is a topic we are trying to solve for in the context of the Microsoft Store and we recently revisited everything in this thread (and in the awesome doc @mgiuca prepared) in that context. We evaluated the options on a couple of deltas:

With all of this in mind, we were thinking it might make sense to offer developers a couple of approaches:

Approach 1: Alternate manifests

This is the current recommendation and can work well for sites that have no issues maintaining/generating multiple static manifest files or who have the capability to dynamically generate a manifest on the server side. This is 3B in Matt’s doc and, in practice, could look like this for folks who want to be declarative:

<link rel="manifest" href="manifest.json">
<link rel="alternate manifest" href="fr/manifest.json" hreflang="fr">

or in a dynamic sense, using the accept-language header via server- or client-side logic (which @marcoscaceres discussed back in 2014):

<link rel="manifest" _href="manifest.json?lang={{ ACCEPT_LANGUAGE }}">_

This approach works well and has no compat issues because it’s the currently recommended approach. It serves many use cases, but a downside is that it does not allow app catalogs to know what languages are available. It also isn’t a great solution for folks that don’t have a system in place to generate dynamic manifests on the server or as part of some build step. For those use cases, there’s Approach 2.

Approach 2: The languages (or translations) member

Based on the discussion, the languages member, indexed by language code, makes a lot of sense. It enables translations of individual fields to be done on an ad hoc basis with a straightforward fallback to the default language of the manifest itself. Adapted from Matt’s doc:

{
  "name": "Good dog",
  "description": "An app for dogs",
  "icons": [],
  "screenshots": [],
  "lang": "en",
  "languages": {
    "fr": {
      "name": "Bon chien",
      "description": "Une application pour chiens",
      "icons": [],
      "screenshots": []
    }
  },
}

This approach is simple and straightforward, allowing devs to pick and choose which keys to redefine in the alternative languages (with lang being perhaps the lone exception, but perhaps we’d want to restrict start_url and others too).

One additional thought we had, after reflecting on what y’all had discussed, was whether it might be possible to have each language definition be either an object or an external reference to a json file containing the keys. Text is pretty small, of course, but this might enable the manifest to remain as small as possible (you might imagine that each translation could include a dozen screenshots, shortcuts, etc., which might become a manageability problem in the main mainfest file). UAs could then selectively choose which manifest language file to load when parsing the manifest file, based on the user’s language setting. This would look like 3A in Matt’s document:

manifest.json:

{
  "name": "Good dog",
  "description": "An app for dogs",
  "icons": [],
  "screenshots": [],
  "lang": "en",
  "translations": {
    "fr": "manifest.fr.json"
  }
}

manifest.fr.json:

{
  "name": "Bon chien",
  "description": "Une application pour chiens",
  "icons": [],
  "screenshots": []
}

If it is possible to enable the value supplied for each language to be either a URL or an object, this kind of flexibility could be quite useful to developers and the translation management systems they are having to work with. It could even lend itself to supporting the dynamic approach we currently recommend:

manifest.json:

{
  "name": "Good dog",
  "description": "An app for dogs",
  "icons": [],
  "screenshots": [],
  "lang": "en",
  "translations": {
    "fr": "manifest.json?lang=fr"
  }
}

An additional benefit to Approach 2 is that the default manifest’s enumeration of the supported languages is easy to consume and can be used to expose that information on the app’s product description page.

Would appreciate y’all’s thoughts on the feasibility of a hybrid approach like this.

tomayac commented 3 years ago

I like the hybrid approach, since it lets everyone do what their infrastructure supports best.

One thing to point out in informal language for Approach 2 might be that no assumption may be made as to the "primary" language (that is, the language of the top-level fields not under "translations"). For some apps, English may be a secondary target.

{
  "name": "Meine Applikation",
  "description": "Eine deutsche Applikation",
  "lang": "de",
  "translations": {
    "en": {
      …
    }
  }
  …
}

This comment is basically saying if "lang" is missing, don't assume it to be necessarily "en".

espinr commented 3 years ago

Thanks @aarongustafson for linking MiniApps WG to this issue. ​

The MiniApp WG already had conversations with the i18n WG to address this same problem. No solution found so far. One of the major concerns was the potential compatibility with the Web Manifest.

The language negotiation or dynamic serving suggested by the Web Manifest cannot be implemented by MiniApps since they are packaged, and they can be retrieved using other non-HTTP protocols (or the file system).

That second approach (translations within the manifest) would enable MiniApps to implement multilingual support, keeping the compatibility with the Web Manifest. So, it's great you raised this issue.

To feed the discussion, apart from the lang, we would also need to specify each language's direction (dir) to cover all cases.

aarongustafson commented 3 years ago

@tomayac Agreed. My thought is that without lang defined, the lang value would be the language of the HTML doc that refers to the manifest (meta language).

@espinr It would be great to align. In the proposed setup, we could allow the dir member to be redefined within the TranslationItem object (or whatever we ultimately call it).

marcoscaceres commented 3 years ago

Thanks @aarongustafson for putting together that proposal! Super helpful.

I'd be supportive of having the straight translations member, but would not be in favor of having separate resources for languages (i.e., no to "an external reference to a json file containing the keys"). The problem with that it adds significant complexity:

Thus, I'd be in favor of just keeping it simple and allowing translations to override - at least for now. We can look at enhancing the behavior if we need to in the future.

aarongustafson commented 3 years ago

Thanks @marcoscaceres. In terms of the potential pitfalls you've noted (which are excellent call outs by the way), I've actually thought a bit about those and would love your opinion on them:

I agree on keeping things simple, but I also cringe thinking about the potential manageability issues that could arise with multiple translations & features (e.g., shortcuts) in the same file. It's possible folks could turn to a build process to do the final merge of multiple files into the Manifest, but I don't know I want to push people that way unnecessarily.

kenchris commented 3 years ago

I think there is a bit of fear that devs won't test all languages and errors might go unnoticed, like the PWA works for all countries but isnt installable in Iceland because language file has errors.

Maybe we should have a reporting API for any such cases?

aarongustafson commented 3 years ago

@kenchris Agreed. Seems like something we could work with folks on building tooling around too. CI/CD, etc. could validate the Manifest & any subresources. Maybe the W3C validator too.

kenchris commented 3 years ago

I was thinking about maybe using something like https://w3c.github.io/reporting/

aarongustafson commented 3 years ago

I was thinking about maybe using something like https://w3c.github.io/reporting/

I’d be totally into the Reporting API too. But if we could help people find the issues before they reach production, all the better.