search

espruino / BangleApps

Bangle.js App Loader (and Apps)
https://banglejs.com/apps
MIT License
495 stars 1.17k forks source link

Add Internationalisation to more apps #136

Closed gfwilliams closed 2 years ago

gfwilliams commented 4 years ago

CLOSED - please see https://github.com/espruino/BangleApps/issues/1311 for up to date info on what needs doing

We now have a great 'locale' module for handling language-specific units in apps - but virtually nothing uses it right now. We should have a push to add it to as many apps as possible - and also to document how to use it.

MaBecker commented 4 years ago

Just started writing a conversation with title

What does the Languages app and how can it be used in apps

gfwilliams commented 4 years ago

... which is at http://forum.espruino.com/conversations/345155

MaBecker commented 4 years ago

just updated it and changed locales to locale

MaBecker commented 4 years ago

Looks like there is no meridian function that returns a value from ampm

I guess that will do:

 meridian: d => (d.getHours() <= 12) ? locale.ampm[0]:locale.ampm[1],

just posted pr https://github.com/espruino/BangleApps/pull/145

stephenrjohnson commented 4 years ago

I'm just wondering we have access to the GPS so would it be worth on first start when we get the time, getting the country and from that guessing the timezone. Not really locale related but we can prompt the user input on changing this if they wish.

gfwilliams commented 4 years ago

I think the Web Browser itself would have a much better idea of locale than we could get from GPS. But that's a good thought - we could try and figure out what the locale was and pre-select it from the list

stephenrjohnson commented 4 years ago

There are two things, one is the locale which I suppose we could guess from the web browser settings (good idea). The other is the timezone, that is more location based. The locale and timezone don't really relate at all, just because your in the UK doesn't mean you want the language to be English. I'm more on about the timezone here (sorry for not being clear), as when the user switches on the device for the first time we set the clock using the GPS, but we could also get a long / lat from that guess the country and from that guess the timezone for this. We would need to add the timezone.db and have a system that gets s country from a long / lat, it's just an idea not sure if you want me to have a crack at it. This would be good when travelling as to set your watch to the correct timezone you just select the menu item (update timezone etc).

gfwilliams commented 4 years ago

Well, the timezone can also be got from the browser too - and is already: https://github.com/espruino/BangleApps/blob/master/js/comms.js#L87

And when using Gadgetbridge it should be updated automatically too.

So really this would just be for the case where someone went to a new country, without a phone/pc, and didn't want to set the timezone manually. I'm not sure it warrants adding a potentially quite large database of timezones to every watch (it'd have to know the actual borders, for instance https://www.timeanddate.com/time/map/)

But if you were interested in doing it, adding that functionality to the 'GPS Time' app which allows the time to be got from the GPS would be extremely cool.

stephenrjohnson commented 4 years ago

The reason is I've got an iphone so can't use the Gadgetbridge app, I'll have a look see if it's feasible, seems doable thanks to https://github.com/evansiroky/node-geo-tz/ or https://github.com/mapbox/timespace or my bet https://github.com/darkskyapp/tz-lookup/ with https://github.com/prantlf/timezone-support.

gfwilliams commented 4 years ago

Docs are now here: https://www.espruino.com/Bangle.js+Locale

paulcockrell commented 4 years ago

Could we not allow developers to specify their own locale file for their app in apps.json, this would be looked up first, then falls back to standard global locale. I think for more non standard text heavy apps relying on a global locale isn't feasible

gfwilliams commented 4 years ago

Yeah, I guess the solution is to have the locale stored in the App Loader at a minimum, and then other apps can grab that info and do what they want with it?

My concern is that realistically only an extremely small selection of apps are going to have translations. I don't think the majority of contributors are going to put in the effort (or have the language skills to reliably translate to different languages).

One option is to have /*LANG*/ by language strings as done in https://github.com/espruino/BangleApps/pull/202/files - then maybe the App Loader could look up in a global table or even Google Translate the relevant strings

srl295 commented 3 years ago

Hi! I work on ICU and also CLDR. I'd like to look at how the locale.js module can be updated automatically instead of maintained by hand.

My concern is that realistically only an extremely small selection of apps are going to have translations. I don't think the majority of contributors are going to put in the effort (or have the language skills to reliably translate to different languages).

This is a good reason to keep the translations separate from the source.

One option is to have /*LANG*/ by language strings as done in https://github.com/espruino/BangleApps/pull/202/files - then maybe the App Loader could look up in a global table or even Google Translate the relevant strings

I put some comments on that PR. Using the English text isn't the best practice for translation, though it's very common. Also, it would be better to call a function, or at least use a hash lookup, rather than have a special comment, that makes for fragile code.

srl295 commented 3 years ago

I wrote my first app https://github.com/srl295/BangleApps/tree/ctxtimer which could perhaps be a good use case for me to try out these ideas.

gfwilliams commented 3 years ago

It'd be great to be able to pull locale info (specifically translations) from an official source. My only concern is because we're dealing with a pretty constrained device we can't do things like having a full unicode charset - often the 'proper' translation would have to be modified slightly to fit with what characters are actually available for display.

/*LANG*/ ... Using the English text isn't the best practice for translation

There's a reason the PR hasn't been merged - I'm not that happy with the direction of it. I can't remember what the reason was for the PR not using the translation function, other than it making it slightly more painful to add a preprocessing step to translate, but yes - it just makes a lot of sense to use require('locale').translate.

The reason for having the English text is that internally, Bangle.js actually attempts to translate untranslated menu/prompt items, so when developers haven't bothered at all there's still a certain level of translation for common menus. It was also a convenient way of making the locale system backwards compatible so that users of old watch firmware didn't find that apps just stopped working.

If there was a common list of IDs with ID->translation that we could use, I think switching to an ID-based system that translated on upload and dropping automatic translation could make a huge amount of sense, and would still be backwards compatible.

The issue I have is that right now these apps are being developed by people for fun. There's been pretty much zero interest by developers themselves in adding translation and actually, despite it being requested and me spending a bunch of time on the locale module and docs for it it really hasn't been used much.

So really the priority for me would be finding a way to actually get apps to be made from the start with translation, and to translate what's there already - rather than creating a new incompatible internationalisation system that nobody will use either.

srl295 commented 3 years ago

It'd be great to be able to pull locale info (specifically translations) from an official source. My only concern is because we're dealing with a pretty constrained device we can't do things like having a full unicode charset - often the 'proper' translation would have to be modified slightly to fit with what characters are actually available for display.

Yes, and there are CLDR transforms available such as Latin-ASCII that could be applied selectively.

The reason for having the English text is that internally, Bangle.js actually attempts to translate untranslated menu/prompt items, so when developers haven't bothered at all there's still a certain level of translation for common menus. It was also a convenient way of making the locale system backwards compatible so that users of old watch firmware didn't find that apps just stopped working.

It makes sense, but you can allow the app to have both, the key and the English.

If there was a common list of IDs with ID->translation that we could use, I think switching to an ID-based system that translated on upload and dropping automatic translation could make a huge amount of sense, and would still be backwards compatible.

There are a number of items in CLDR that could be used.

The issue I have is that right now these apps are being developed by people for fun.

:+1: and thank you for espruino!

There's been pretty much zero interest by developers themselves in adding translation and actually, despite it being requested and me spending a bunch of time on the locale module and docs for it it really hasn't been used much.

That's understandable. But let me take a look at the locale module and see if I can polyfill the ecma402 API. The benefit there is not needing separate documentation on the locale module, but just using the standard JS api.

So really the priority for me would be finding a way to actually get apps to be made from the start with translation, and to translate what's there already - rather than creating a new incompatible internationalisation system that nobody will use either.

Makes sense. I'm just starting to take a look at this. If there's a way to get leverage and simplify things that could be good.

gfwilliams commented 3 years ago

That's understandable. But let me take a look at the locale module and see if I can polyfill the ecma402 API.

That'd be great. I recently added a modules folder and you could put Intl in there. Any apps that needed it could just pull in that polyfill. Either that or you could add it to the current apps/locale tool, but the issue there is then apps won't work until it is installed.

Weiming-Hu commented 3 years ago

Hi, is there any on-going interest in GadgetBridge to make it support displaying messages in other languages? Like Chinese? Chinese characters are currently displaying as ??? now which is annoying. I'm wondering if there is something I can contribute to? Thank you.

gfwilliams commented 3 years ago

Hi - yes, that is annoying. One issue we have is that the Bangle is really only capable of handling 8 bit character strings. I just did a quick search and it seems that there isn't really a way of getting all the Chinese characters you need into a single 8 bit codepage. Please let me know if I'm wrong though!

So... There's another option which I think may be more sensible and flexible (it'd work for all languages). It is that we make Gadgetbridge render the complete message into a bitmap, and then send the bitmap over, rather than the text?

Currently we send notifications like:

GB({"t":"notify","id":1575479849,"src":"Hangouts","title":"A Name","body":"message contents"})

But we could easily send:

GB({"t":"notify","id":1575479849,"src":"Hangouts","titleimg":"base64encodedimage","bodyimg":"base64encodedimage"})

The images are of the form shown at https://www.espruino.com/Graphics#strings so are pretty basic.

Do you think that's something you'd be interested in working on? I'd be happy to make the relevant changes in the Bangle.js code do handle this - we'd just need changes to Gadgetbridge to make it render the text out to a 240px wide bitmap

neshanjo commented 3 years ago

Hi there,

I've recently added internationalisation to multiclock and cliock and was wondering where the actual code is located that uses locales.js (just out of curiosity).

Also, for clock faces, a third date format consisting of month and dayOfMonth would be useful, since this varies in different countries. E.g. March 7 (US), 7. März (German), 7 Mars (France) - order and usage of ".". Format 1 is something similar, but includes the year which is often not needed/wanted on clock faces.

Furthermore, are there plans to localise the settings app? Some of the toggle options are already translated, while the main strings are not, which looks very odd to me (Snippet from BLE submenu settings with German locale):

Programmable       an
Passkey BETA     none
Whitelist         aus
< Zurück

Kind regards, neshanjo

gfwilliams commented 3 years ago

Hi! Thanks for your work on this!

The code that uses locales is actually in the HTML file: https://github.com/espruino/BangleApps/blob/master/apps/locale/locale.html#L181

It reads that info and then writes a locale file to Storage so it can be loaded as a module (there's also an English 'placeholder' in Bangle.js firmware for if locale isn't installed).

a third date format consisting of month and dayOfMonth would be useful

I feel like locale.month(month)+" "+dayOfMonth is probably flexible enough for this? Or so you imagine different languages would have different ways of writing it?

are there plans to localise the settings app?

This is actually part of a bigger issue of localising apps. I'd rather come up with something that can easily be applied to all apps without bloating them rather than just modifying one or two individual apps. Right now we're not getting that many app submissions anyway, and I don't want to make the process any more difficult than it already is.

There's a possible solution that could work here: https://github.com/espruino/BangleApps/pull/202/files

So effectively tag any string in the code that can be translated with /*LANG*/ and then preprocess it on upload to upload a pre-translated app. While in this case it makes the process of uploading apps painful (requiring a per-app translation), if the processing code were added inside the BangleApps loader itself then the loader could have a 'language' option and could change each app without the user ever having to be aware (with a global list of translations).

However short term, none for Passkey could be changed to no to take advantage of the existing translation that is offered I guess, and maybe there are other similar cases too?

Weiming-Hu commented 3 years ago

Hi @gfwilliams . I think rendering the bitmap is a good idea. I'm happy to work on this. But I probably won't be able to do it until mid June. But I will keep an eye on this issue.

I guess what I need to do is to:

  1. figure out how messages are currently sent from GB to JS;
  2. change the message interface to include a bitmap using some sort of conversion in Java

And then you will be able to make changes in JS app accordingly?

Thanks

gfwilliams commented 3 years ago

Hi @Weiming-Hu that's correct. Once you're producing something like described in https://github.com/espruino/BangleApps/issues/136#issuecomment-846912389 then it should be pretty easy for me to pull in the bitmap and display it

Weiming-Hu commented 3 years ago

Awesome. I will mark this and remind myself when it's time. Thank you.

neshanjo commented 3 years ago

a third date format consisting of month and dayOfMonth would be useful

I feel like locale.month(month)+" "+dayOfMonth is probably flexible enough for this? Or so you imagine different languages would have different ways of writing it?

Yes, indeed there are different ways. As I wrote in my original post: US: March 7 (month day) German: 7. März (day dot month) France: 7 Mars (day month)

The only way of doing this, is to hard-code the format in every app, see https://github.com/espruino/BangleApps/blob/1dca641ff0b4a94a888994fa137262df694c4ff5/apps/multiclock/timdat.js#L23

Anyway, if there is no need for this at other places, it's ok to keep it like this...

Thanks for the other comments, too, which I will have a look at later.

neshanjo commented 3 years ago

are there plans to localise the settings app?

This is actually part of a bigger issue of localising apps. I'd rather come up with something that can easily be applied to all apps without bloating them rather than just modifying one or two individual apps. Right now we're not getting that many app submissions anyway, and I don't want to make the process any more difficult than it already is.

There's a possible solution that could work here: https://github.com/espruino/BangleApps/pull/202/files

So effectively tag any string in the code that can be translated with /*LANG*/ and then preprocess it on upload to upload a pre-translated app. While in this case it makes the process of uploading apps painful (requiring a per-app translation), if the processing code were added inside the BangleApps loader itself then the loader could have a 'language' option and could change each app without the user ever having to be aware (with a global list of translations).

I took a look at this and also share the concerns of this commenter: https://github.com/espruino/BangleApps/pull/202/files#r542584252 I'd prefer a key-value system and some locale.t(<key>, "default string in english") calls over rewriting code based on special comments. One could provide a tool that replaces all strings by these method calls in order to make it easier to translate existing apps (using incremented numbers as keys). Language-specific key-value-pairs could go to lang_<locale>.json files in the app subdirectories. The only thing we need to make sure is that during app installation only one language file is transferred (to save space). Can we read locale.lang on the device before installing an app?

That all said, I wonder if it is generally necessary to translate apps at all. While I think it has been worth the effort to provide local speed and temperature units as well as time/date formats, I'd guess that most of the Bangle owners have decent English skills and no problems to understand the apps...

However short term, none for Passkey could be changed to no to take advantage of the existing translation that is offered I guess, and maybe there are other similar cases too?

I don't see an advantage of partially translated apps. It only confuses me to read something like "Programmable: an" with the German locale installed. Additionally, I guess if someone has limited English knowledge, he/she would rather know the words "on/off/yes/no..." than "Programmable/Quite Mode/...".

That's only my personal opinion, though. Thanks for all the great work on Bangle.js. I ordered one of the first watches available in the Espruino shop and, since then, there has been so much improvements. That's really impressive!

gfwilliams commented 3 years ago

Thanks!

locale.t(<key>, "default string in english")

We could do this. I can totally understand about the issue of changing names, it was just based on my experience of Blockly (https://github.com/espruino/EspruinoWebIDE/blob/master/blockly/en.js) being very long-winded and prone to errors. The key itself just ends up being a non-capitalised version of the original English.

I feel a bit like the 'sanity check' app could easily check either way to see if translations existed though, so the issue of names changing and breaking translation is I think less of an issue

lang_<locale>.json files in the app subdirectories

I'm not too convinced by this one though - just because I don't think the vast majority of the committers have the motivation (or knowledge in multiple languages) to translate each app. Also, given memory is at such a premium, I think we wouldn't want to have a per-app JSON file in RAM all the time (or to have to load it every time something was printed to the display), so we'll really want to do the translation on upload.

I really think a global list of translations would be a big help, and then we can have some code that runs through all the apps and provides a 'to-do' list of translations for each language. Then someone with knowledge in each language can just run through the untranslated phrases and help out to translate a bunch of different apps.

I don't see an advantage of partially translated apps.

I agree here too - yes/no/ok/etc are probably english words most people could figure out anyway.

When doing the initial code I'd hoped it'd be more useful than it actually was. I think once we get a robust set of translations set up then I'll remove the code from showMenu/showMessage/etc which should speed things up a bit anyway.

neshanjo commented 3 years ago

I don't think the vast majority of the committers have the motivation (or knowledge in multiple languages) to translate each app.

You are absolutely right. This should definitely not be done by the app developer, but by some native speakers that just contribute by adding translations. To facilitate this, a global translation file could be used and a tool/scripts that adds missing translation keys.

Also, given memory is at such a premium

I didn't take this into consideration. A translation-on-upload approach makes more sense here.

Regarding the initial idea of using the actual language strings as translation keys, you're definitely right with this:

The key itself just ends up being a non-capitalised version of the original English.

Also, translation of a file like this is much easier than when it only contains cryptic keys that one has to look up in another file.

However, I'm a bit concerned that we run into other issues:

  1. Ambiguity of language. Take for example the word "mean". It can be translated as "meinen" (signify/indicate) or "gemein" (rude) to German and might have one meaning in one app and another in the next.
  2. Dependency on context: "button" would be translated as "Knopf" if we are in the context of clothes. In the context of UI design, it would be "Schaltfläche". (There are probably better examples - "Knopf" would also be ok in the context of UIs, but you get the idea...).
  3. Different genders: In English, every "thing" is just "it", but in German, there is "er/sie/es" (he/she/it) also for every word. Thus, if in a string something else is referenced ("it is on the left side"), it is not clear how "it" should be translated.
  4. Formal vs. informal addressing: In English, we have "you" when talking to anyone. In German, there is formal addressing ("Sie") and informal addressing ("du"). If we can not guarantee that every app uses its own set of strings, we have to make sure that only one type of addressing is used, otherwise the app will be very confusing.

I know that the longer the string is, the less is the probability of running into issues 1 and 2. So maybe we can just use the proposed idea and see how well it works for the existing apps. We could include a mechanism for overriding certain strings in certain apps straight from the beginning, e.g.

{
  "GLOBAL": {
    (all the global translations)
    "Alarm" : "Wecker",
    "Hours" : "Stunden",
    "Minutes" : "Minuten",
    "Enabled" : "Aktiviert"

  },
  "trex": {
   (overrides for the app "trex")
    "Alarm" : "Alarm"
  }
}

Instead of marking all strings to be translated with "/ LANG /", I'd rather do the opposite and mark all strings that should not be translated (which are probably much less).

A good first step could be writing a scripts that extracts all strings from the existing apps and puts it into a file like above in alphabetic order.

What do you think?

gfwilliams commented 3 years ago

Sorry for the delay - not sure how I missed this first time, but that all sounds great to me!

I'm not sure I agree with marking all strings not to be translated though. Just look at a simple app like https://github.com/espruino/BangleApps/blob/master/apps/widid/widget.js

It displays no translateable text, but you'd have to tag ":", "6x8", "tr", and "widid" as not translated.

But as a first start, definitely running through and grabbing text strings would be a good idea. I'm sure we could come up with some heuristics to allow us to suggest when apps should have text strings tagged as translateable.

neshanjo commented 2 years ago

I made some progress on this topic, see referenced commit. Also attaching the output of the script here: string-constants.txt

There are a lot of strings that don't need to be translated...

gfwilliams commented 2 years ago

Thanks! That looks really promising - I think there are probably some heuristics we could use to narrow this down (checks for obvious big base64 strings). Perhaps it could even be spell-checked.

However, for something we actually deployed I think we should use an off the shelf JS tokeniser rather than making our own - for instance how does yours cope with quotes in regex or templated strings? At minimum we could use the one that's in EspruinoTools, but even that isn't foolproof.

neshanjo commented 2 years ago

Fully agree with using some external library for parsing the string literals. (Also noticed that it should read literals not constants.)

I wasn't sure if introducing an external dependeny is ok since most of the code is self-contained. Also I didn't know that regexes and template strings are supported with Espruino. Will be interesting how to deal with translating template strings.

Is there a tokenizer you can recommend? Should it be included as devDependency?

gfwilliams commented 2 years ago

Since eventually it'll have to work in the app loader itself I think we're best off using what we have already.

var lex = Espruino.Core.Utils.getLexer("my code string");

should work, and it's used in other areas of code processing too.

You can check some of the apps in bin to see how CLI app can use theEspruino` library

jeroenpeters1986 commented 2 years ago

I was also trying to add translations in Messages, but I ran into trouble as the strings that had to be translated are also a 'key' in the menu. I tried to pass them as translated vars but that didn't work. I think I read somewhere that E.show and the likes of it should already be using translations in their core, but adding the strings to the nl_NL locale language didn't translate them.

The solution used for Alarms or MultiClock (use your own translation) in https://github.com/espruino/BangleApps/issues/136#issuecomment-855907434 is acceptable to me, but I understand for Espruino/Bangle not going there. But at the moment I am not sure how to proceed properly. Is there any consensus on hwo it will/has to work? I could not find the Lexer in https://github.com/espruino/BangleApps/tree/master/bin

gfwilliams commented 2 years ago

I ran into trouble as the strings that had to be translated are also a 'key' in the menu

Could you explain? I thought we were doing stuff like:

E.showMenu({ 
  "one": ...
  "two": ...
});
// to:
E.showMenu({ 
  /*LANG*/"one": ...
  /*LANG*/"two": ...
});
// which would then get converted on upload to
E.showMenu({ 
  "eins": ...
  "zwei": ...
});
// which should still be ok?

I just added a file bin/language_scan.js which uses the Espruino tokeniser to pull out text strings (see commit above).

at the moment I am not sure how to proceed properly. Is there any consensus on hwo it will/has to work?

Well, what I feel would work well is:

jeroenpeters1986 commented 2 years ago

yes I can explain: I did something stupid, I added locale.translate() myself, and this is not necessary for any translation of menu's :-(

gfwilliams commented 2 years ago

Just to add - if you wanted to manually add /*LANG*/ to a bunch of strings in apps it'd be pretty handy as it'd give something to test against.

gfwilliams commented 2 years ago

Just pushed a bunch of changes (see above) but if you modify core/index.js in your own version to include the commented out LANGUAGE variable you can actually get it working and translating apps that you upload.

khampf commented 2 years ago

Not starting a new issue but I would like to be able to set a date format in settings which can then be used for example by clock faces, I have several faces that I would like to use and I could probably hack them into what I want but would not a selection of formats like dd-mm-yyyy, dd.mm.yy, mm/dd/yyyy and similar be useful in combination with a getDateStr() or similar be great? It would make clockfaces "right" for everyone. Or is this just a stupid idea?

gfwilliams commented 2 years ago

@khampf it's already there! I think you're after http://www.espruino.com/Bangle.js+Locale

khampf commented 2 years ago

@gfwilliams Awesome! I must try to hack some watchfaces so they are compliant. Or maybe it is time to make my very own :-) And I really need to look more into the settings for locale as I see no way to set it in the settings app of the main branch of Bangle apps. But I feel this is already a great start. Thank you for enlightening me :-)

gfwilliams commented 2 years ago

For the locale library you just install the 'Language' app in the App Loader and everything gets sorted. But yes, converting more apps would be great!