atk4 / ui

Robust and easy to use PHP Framework for Web Apps
https://atk4-ui.readthedocs.io
MIT License
440 stars 105 forks source link

Create proposal for UI translations #715

Closed romaninsh closed 5 years ago

romaninsh commented 5 years ago

Many of our users need non-english version of ATK. Also we need a guide for external add-ons, on how they should approach translations.

For now - need answers to the above questions. To be discussed in next hangout.

romaninsh commented 5 years ago

See also #315

acicovic commented 5 years ago

From my admittedly really short research, here are some things regarding gettext. If your research indicates otherwise let me know.

  1. gettext is a PHP extension that one needs to install and enable in php.ini. Thus at least theoretically it could be missing in certain environments. My shared hosting account does have it.
  2. If we want to use https://github.com/oscarotero/Gettext this means about + 50 files in ATK. As I understand it this is optional but will give us advantages.
  3. gettext offers sophisticated translation features that are suitable even for very large projects. But on the downside it can get complicated.

We should pick a solution that works everywhere. So if gettext can be missing in certain environments that is a problem. About the additional gettext library (2) in general I avoid adding too many files, except if there's a real benefit. Gettext can offer us a lot but we must see if our project will use that power. If we have a few strings that we want to translate then I think gettext might be overkill, except if there's a way for a really simple implementation.

In a non-gettext scenario here's what I would do:

  1. Create a localization file for each language (for example strings.fr.php for french).
  2. Create an $app->setLocale() function or $app->locale property that would be set before App loads.
  3. During App load and depending on the locale, the correct localization file would be loaded and variables would be added to an array or variable list. In case no locale was specified, default (strings.en.php) would be loaded. We could even avoid the file load in case of default by having the default English strings within the views themselves.
  4. All views would have a variable placeholder where the strings are now located. Those variables would be replaced with the strings loaded from our file.

This system could be enhanced with sprintf() if needed in order to change the position of certain elements. But it might not even be needed. This solution is lightweight, will work everywhere and I think it's a good fit for our project. I would also be able to implement it. Translating to another language would be as simple as copying the file, changing the fr to the language code and translate the strings within the file. Pretty simple for a PHP person.

If we're sure about gettext availability everywhere, there might be similar gettext usage scenarios that we might consider. As I haven't used gettext @abbadon1334 is in a better position to propose something.

I can provide Greek and French translations. I would not mind having my French translation inspected by a native speaker. I'm sure @abbadon1334 could provide Italian and @PhilippGrashoff German. It's a good starting point. Spanish would be a language to look for. And Chinese....

abbadon1334 commented 5 years ago

Another WALL OF TEXT is coming ( like winter )

i suggest all to read this chapter, there are some exceptions in localization that any specific library already solves and if you don't consider it and do one of your own you will have problem in the future :

INTRO

IMHO i think there is more value to add third party library that solve a specific domain and solve it well, Going in this way as a good side effect, during the usage maybe we can solve an issue or collaborate on something; In place of close our code to our implementation and make us even more asocial than average devs can be :)

ZA PROBLEM

We need in App :

SOLUTION 1 - object definition out of atk4

interface TranslatorInterface
{
    /**
     * @param string[] ...$args
     * @return string
     */
    public function translate(...$args): string;
}
class App
{
    /** @var TranslatorInterface */
    private $translator;

    private function setTranslator(TranslatorInterface $translator): void
    {
        $this->translator = $translator;
    }

    public function __(...$args): string
    {
        return $this->translator->translate(...$args);
    }
}

after this we can create the TranslatorNulled in this way, which will be the default translator. to avoid in code any checking of types.

class TranslatorNulled implements TranslatorInterface
{
    /**
     * @inheritDoc
     */
    public function translate(...$args): string
    {
        return $args[0];
    }
}

after add this 2 method we can start searching :

and redirect all to $this->app->__(...$args)

i think this is the solution with less code possible, that open atk4/ui to localization

After that we can create as many as we want implementation of TranslatorInterface and add support for GetText, other libraries or reinvent the wheel because all like to do it and be proud of it (included me)

But we give guidance and unbreakable restriction to do it giving an interface, make translator private and expose only translate method, force devs to be stick with this implementation.

i think that in future can be something like this:

$app = new App();

/* define translator */
$translator = new Atk4TranslatorGetText();
/* blah blah */
/* define translator */

$app->setTranslator($translator);
$app->run();

this solution is in line with Symfony Contracts TranslatorInterface.

if we think more on high level integration with atk4, there is another solution

SOLUTION 2 - object definition in of atk4

/**
 * Interface TranslatorInterface
 */
interface TranslatorInterface
{
    /**
     * Set language code
     *
     * @param string $code
     */
    public function setCode(string $code): void;

    /**
     * Set language code to fallback in case translation is not present
     *
     * @param string $code
     */
    public function setCodeFallback(string $code): void;

    /**
     * Add file to load in translations
     *
     * @param string $path
     */
    public function addFile(string $path): void;

    /**
     * Call at the end of initialization
     * some libraries probably need a method after all config is done
     */
    public function onInitEnd(): void;

    /**
     * @param string[] ...$args
     * @return string
     */
    public function translate(...$args): string;
}
class App
{
    public $translator_language = 'en';  // <-- can be protected?
    public $translator_fallback = 'en';  // <-- can be protected?

    /** @var TranslatorInterface */
    private $translator;

    /** @var string */
    public $translator_class = \atk4\ui\Translator\Nulled::class;  // <-- can be protected?

    /** @var string[] */
    public $translator_paths = []; // <-- can be protected?

    /**
     * @param TranslatorInterface $translator
     */
    private function setTranslator(TranslatorInterface $translator): void
    {
        $this->translator = $translator;

        /** can be only one that set code + fallback */
        $this->translator->setCode($this->app->translator_language);
        $this->translator->setCodeFallback($this->app->translator_fallback);

        foreach ($this->app->language_file as $file)
        {
            $this->translator->addFile($file);
        }

        $this->translator->onInitEnd();
    }

    private function initTranslator()
    {
        $translator = $this->translator_class;

        $this->setTranslator(new $translator());
    }

    public function __(...$args): string
    {
        return $this->translator->translate(...$args);
    }
}

all is defined in the fields of Container App. If Dev extends the app and define in extended class all the needed parameters, inner methods to setup the language will be fired and do all automation needed.

$App = new App([
'translator_class' => \atk4\ui\TranslatorGettext::class,
'translator_paths' => './locale',
'translator_fallback' => 'en'
]);
$App = new UserApp(); // <-- extended App
$App->run();

SOLUTION 3 - Total atk

N.B. but remain the problem that you cannot add something before defining layout, even if is something that is functional and not related to any kind of UI, i'm wrong?

interface TranslatorInterface
{
    /**
     * @param string[] ...$args
     * @return string
     */
    public function translate(...$args): string;
}

class App
{
    public function getLanguage() : string
    {
        return 'language code from somewhere (config | user-session)';
    }

    public function __(...$args): string
    {
        return $this->translator->translate(...$args);
    }
}

class TranslatorNulled implements TranslatorInterface {

    public function translate(...$args): string
    {
        return $args[0];
    }
}

class TranslatorReallyReallySimple implements TranslatorInterface {

    use \atk4\core\AppScopeTrait;
    use \atk4\core\ConfigTrait;
    use \atk4\core\InitializerTrait;

    public function init()
    {
        parent::init();
        /**
         * LOAD CONFIG - language
         * LOAD CONFIG - language fallback
         * LOAD CONFIG - language files
         */
    }

    public function translate(...$args): string
    {
        return $this->config->getConfig($args[0]);
    }
}

Notes

Did you see any other ways to do this?

acicovic commented 5 years ago

@abbadon1334 great research and proposal.

My proposal is a dead-simple solution which has worked for me in the past even for sites with 3-4 languages. If we want a complete translation system it's indeed better to go with a library. My only question is if we do need something that elaborate. Perhaps having it even if it's not needed is good, I guess this more is up to you and @romaninsh to discuss and decide.

Here are some comments:

abbadon1334 commented 5 years ago

My proposal is a dead-simple solution which has worked for me in the past even for sites with 3-4 languages. If we want a complete translation system it's indeed better to go with a library.

the 3rd solution, the full atk and dead simple one, is a draft of what you said, look in core/ConfigTrait is only in dev branch, is nearly all done.

Every solution i propose to implements an Interface, default for all solution it will be TranslatorNulled, that return the string without translations ( * probably we need to call sprintf to make it works with arguments).

My only question is if we do need something that elaborate. Perhaps having it even if it's not needed is good, I guess this more is up to you and @romaninsh to discuss and decide.

Is up to all who want to discuss it, i'm the last one arrived :)

In my opinion even if we go with something complete it should be easy to use to encourage translation. We should be based on an interface as you did so add-on authors can also implement it and use it. Personally I prefer non-compiled translation files because they encourage translation (you just duplicate and translate - even a non-programmer can do it).

i agree, i answer this at the end of this post.

Regarding the last note about models and persistences I don't know how you have it in your mind, but I wouldn't like to need a database just to translate 20-30 strings. If we talk about a small SQLite database with all the languages that will be distributed with ATK, then we should see which method is faster and encourages translation more. If there's a central language database, we should also see how this will work for add-ons. For example, if tomorrow I create an add-on, it's easy for me to implement an interface and create some text files with translations. However, if all translations are meant to be on a central ATK database this enforces coupling and If my add-on is not accepted as official I might have a problem. So perhaps clarifying what you meant will help us.

Anyway, i think there is a big part of people that work on MVP (atk4 is perfect for this) and probably even configure a translation system is something in plus to do.

For this purposed i propose to return directly the indexed string, because in this way if you don't want to setup the translations you don't need even the localization file.

If you want to use translations, you can substitute TranslatorNulled with another implementation or implements your own because you hate the world and want to blow up King's Landing :D

Translations

If we stick to what others do, make easier to switch to atk4, we need to find a good balance between simple and easy to be changeable because i'm sure there will be some PSR even on this, look at a beautiful and simple Framework like FatFree which is less appreciated only because has his own way of doing things.

If we want to even manage translation in a CRUD we need to find a file format that can be used in atk4/data (*) or define a atk4/Model for Translation with field_id (string) which i think is not a really bad idea, but on this i want to hear @romaninsh and @DarkSide666.

I try to think about what can be added to atk4 that already has its own translations and i have no clue about what it can be.

(*) every format i see has some multidimensional arrays format, if you want to do a good work.

acicovic commented 5 years ago

Awesome. I personally have no other questions or comments and I'm sure we'll have a great localization system for ATK. @abbadon1334 thanks for all the time you've been putting into this.

abbadon1334 commented 5 years ago

@acicovic is a pleasure, i think solution 3 and your description of execution is equal, i just wait for others to discuss and after that we can start deciding what to do.

I think there is much more work to find all strings in code than implement this.

DarkSide666 commented 5 years ago

Does localization also include things like local date formats, number (dot or comma, thousand separator) formats, first day of week (Monday or Sunday) etc. ?

abbadon1334 commented 5 years ago

Does localization also include things like local date formats, number (dot or comma, thousand separator) formats, first day of week (Monday or Sunday) etc. ?

yes, but i think that localization variables will be stored directly in persistence_ui i think there is already a start there, i think that must take the right format from there for normalization/cast.

This idea of localization and translations, pops up because we speak with Sean in the last meeting about calendar date format.

acicovic commented 5 years ago

Dates, currencies and date formats is where things can get tricky.

For example, I might want to have an environment in French but the currency to be dollars and the dates formatted in a yyyy-mm-dd format. So we should be able to customize this through properties or easily create our own custom localization files.

abbadon1334 commented 5 years ago

Dates, currencies and date formats is where things can get tricky.

For example, I might want to have an environment in French but the currency to be dollars and the dates formatted in a yyyy-mm-dd format. So we should be able to customize this through properties or easily create our own custom localization files.

wait..., there is a major refactoring on data cast e field type in atk4/ui and atk4/data, i think number format must be done in data, while in ui we can have field for currency that get a default format with currency sign.

look at this : https://github.com/atk4/data/pull/375

I think is best to focus on translation first, after that probably even other PR will be solved or much ahead in dev and we can work together on a better solution.

BIG NOTE : we need to consider for localization Fomantic/VUE evolution.

romaninsh commented 5 years ago

Notes from meeting (ATK Weekly 11)

Solution 1: only translate STRINGS generated or hard-coded in ATK namespace $form->addButton($form->app->_('Save')); // button label is translated

Solution 2: magically translates during init() or getCaption() of model/field/getCaption $form->addButton('Save'); // button will be translated

routed through $app->_();

(which MAY call _() or __() depending on the library you use)

" 25 items"

"item count: 25"

"% item(s)"

$form->addButton('Delete 25 items');

$form->addButton($app->_('Delete % item(s)', [25]));

$form->addButton(['caption'=>_('Delete % items')]);

$obj->__toString()

function App::($arg, ) { if ($arg instanceof TranslatedString) { return $arg; } return new TranslatedString(parent::($arg)); }

ATK way - Solution 2 Romans - Solution 1 Gowrav - Solution 2 Franchesco - Solution 2 Alain - Solution 2

{$id} -> converts to View->id automatically {}Welcome to my add-on{/} {$header}

_('Hello') => 'Привет'

gettext.po Hello=Привет

$model->addField('name', ['caption'=>'Name']);

2a. argument caption = should init() automatically translate? yes: gowrav, romans

2b. caption to hold english version, but getCaption to translate on the fly? yes: franchesco,

in atk4/somefile.php throw new exception([$app->_('something bad happened')]);

Model data..

DarkSide666 commented 5 years ago

Not sure if I understood everything correctly from notes above, but here are my 2 cents.

  1. what to translate? i vote for full translation support meaning not only we should translate ATK hard-coded strings, but also allow user to translate his own texts of course.
  2. when to translate? I think it should happen on render phase (and when throwing exceptions) because only render phase can show something to user (UI).
  3. how to translate? I think we should implement simple TranslatableTrait in atk4/core and apply it to all other classes. This trait should implement _() proxy method which calls $this->app-_() if possible.That way we can simply call $this->_() in any class and should not always do something like $this->app ? $this->app->_('Save') : 'Save' And yes, I vote for using gettext(). I think it's quite ok and developers should be used to it (Wordpress). But if we separate translation engine from App class (use app only as proxy to Translator class) then developer will be able to inject his own Translator class and use something else than gettext.
  4. Need to think if we actually need to translate something in html templates. I think that html templates shouldn't contain any text in any language. If such text is needed somewhere in template, then php class should add it (and allow to configure it and ... of course will be translated then).
  5. Need to think about howto translate texts coming from JavaScript...
pkly commented 5 years ago

I'm using the Agile Toolkit, Data and so on for our company's new CMS, mostly for UI and some operations, we're not english speakers so obviously the first thing I looked into was getting translations working.

Currently, the fork I've made has a few modifications:

So, a file like View, if it had text to translate, it'd look like this:

/atk4/ui/View/{lang_code}.json

{ "Original Text": "Translated Text" }

I've been using this solution for a while, if you're going to implement your own solution I'd ask that it's an interface and you can somehow completely redirect everything.

abbadon1334 commented 5 years ago

@pkly sure!

@DarkSide666 i like the idea of Trait, because it can be used everywhere and is not coupled only to App, and you are right about use of standalone atk4/api or atk4/data, but i think we have a problem

interface TranslatorInterface
{
    /**
     * @param string[] ...$args
     * @return string
     */
    public function translate(...$args): string;
}

class TranslatorNull implements TranslatorInterface {

    public function translate(...$args): string
    {
        return /* return string as it is */;
    }
}

trait TranslatorTrait {

    public $_translator_trait = true;

    protected $translatorClass = TranslatorNull::class; // <-- how to define this? we don't have any singleton, why start now?

    public function translate(...$args): string
    {
        return /* return string translated */;
    }
}

Questions :

  1. how we can pass the TranslatorClass efficently?
  2. TranslatorClass must be only one per Application, if we define $TranslatorClass in application, how we can do this in atk4/data or atk4/api, because if we create a Trait in atk4/core that trait must be capable to be used everywhere, not only in atk4/ui

I start a draft on core/ConfigTrait to implements not only simple configs but even instance and factory referencing them by Interface name and not Class name, like Slim3 do for ContainerDI, i will push something later and we can talk further ( config file will look something like this : https://github.com/abbadon1334/nemesi-php/blob/master/config/logger.php ).

pkly commented 5 years ago

@abbadon1334

Why not make it a class that gets passed as a parameter to /atk4/App (default: null), implements interface from /atk4/core.

If /atk4/App notices the translation class is null, simply generate the default one (sitting in /atk4/core) That way you could evade using a singleton and allow for the App class to control the translations somewhat.

abbadon1334 commented 5 years ago

@abbadon1334

Why not make it a class that gets passed as a parameter to /atk4/App (default: null), implements interface from /atk4/core.

If /atk4/App notices the translation class is null, simply generate the default one (sitting in /atk4/core) That way you could evade using a singleton and allow for the App class to control the translations somewhat.

as i said the problem is not how to pass to an App that use TranslateTrait, like you already said, atk4 will do the magic just in this way :

new App(['TranslationClass' => $TranslationClass]);
// After this all elements will have a reference $element->app and access to relative method @__

I just point my finger on usages of atk4/core/TranslatorTrait other than ui like api | data | reports etc.. etc.. UI implementation is easy because all of us already have done it custom and already know the real cases

abbadon1334 commented 5 years ago

Probably i put too much thinking but i prefer to put on the table every situation and possible use when we are still analyze and probably "lose" time.

DarkSide666 commented 5 years ago

Well, translator trait or class or whatever shouldn't be dependent on App because not always we will have app. But if we implement it as trait then we can add trait to any php class.

Even more, we can have 2 Views in same UI page and each View text could be localized in separate language. That would be awesome.

And not sure if that's possible somehow, but by default if App or View have localization trait set, then maybe it could pass that localization trait/class to any child object you add with $object->add() if that child object don't already have translator linked.

abbadon1334 commented 5 years ago

Well, translator trait or class or whatever shouldn't be dependent on App because not always we will have app. But if we implement it as trait then we can add trait to any php class.

agree

Even more, we can have 2 Views in same UI page and each View text could be localized in separate language. That would be awesome.

this is what i try to figure out, because all framework work as a container and can contains more than 1 app, is stated multiple times in the documentation and i don't want to break this interesting feature

And not sure if that's possible somehow, but by default if App or View have localization trait set, then maybe it could pass that localization trait/class to any child object you add with $object->add() if that child object don't already have translator linked.

I have a little fear that we need to implement something on our own because from what i have seen gettext and all library around it use gettext like a static reference object scoped to php because it is called at SO level, someone can try to find info regarding this?

abbadon1334 commented 5 years ago

PR wip in atk4/core => https://github.com/atk4/core/pull/93 Waiting feedback

pkly commented 5 years ago

@abbadon1334 I looked at the changes you've proposed but I don't really understand why you'd have that last part that loops over arguments to simplify sprintf. I don't believe that using something like

$this->caption = sprintf($this->_("This is some %s thing"), "aaaa");

is really that bothersome, it also helps to keep things simple. You should probably just keep it to string $string, ?int $plural = null

I'm also unsure if having the plural case is absolutely needed, but I don't think it's a problem. The translation system I've made for my project uses a different approach, but similar, it uses the class name to match the string to a file, which allows for a few things:

I know that atk4 doesn't really use singletons, so maybe it's not the best approach.

If you go ahead and implement it as a trait it should be alright, since most of the strings are in the classes and not really in objects from what I've seen, so I can modify it to call my translation system with minimal changes if required.

I could also provide a Polish translation for the whole UI/Code/Data/DSQL set of libraries, since that's what I'm currently depending on in my project.

abbadon1334 commented 5 years ago

1st : some language can have more than 1 plural form 2nd : sometimes you need even zero form, not only singular and plural

we don't want to use singleton, if we use singleton we lose one of the best thing in atk, i mean the possibility of running multiple app as independent "containers of definitions", the code it's just a base to discuss but i think we need to avoid direct calls to sprintf. I think to help user sometimes in atk4 we allow too many ways to accomplish the same result, i try in translation to give only one way to do things to stop disorienting users, i think you notice that i marked final the default class because i want the user to implements the interface and not extend our specific implementation.

i have some work to do, but in a few days i will write some examples of using the translation system, to further discuss about it, the idea is to remain simple, it will be something like this :

but i will be much specific with some examples in a few days

pkly commented 5 years ago

Sure, keeping multiple forms for plurals and zero is fine, but I don't really get the whole extra sprintf usage.

And like I said, I also don't think that putting a singleton into atk4 is a good idea, since it makes no real sense and provides barely any benefits.

abbadon1334 commented 5 years ago

the idea is to make nested translation possible, can be wrong and can be unuseful, i think only when we use it or figure out the usage possibilities we can see if it is need or not.

you show this case :

$this->caption = sprintf($this->_("This is some %s thing"), "aaaa");

why not :

$this->caption = $this->_("This is some %s thing", ["aaaa"]);

and in the method _ it will do something like :

$caption_translated = $this->_("This is some %s thing"); // if not present it return the string as it is
$aaaa_translated = $this->_("aaaa"); // if not present it return the string as it is

return sprintf($caption_translated,$aaaa_translated);

in this way if there is a translation for aaaa it will be translated too.

abbadon1334 commented 5 years ago

i update the PR https://github.com/atk4/core/pull/93 , can you take a look @pkly ? you were right if we start do that sprintf we enter in an hell because we need to include even context of translation :D

abbadon1334 commented 5 years ago

i add tag hangout agenda, because i need to know some opinions, if i have time i will prepare some case usage to discuss it.

pkly commented 5 years ago

@abbadon1334 I looked through it, besides that one exception that I don't like I've also listed a suggested implementation idea in the comments, although I'm unsure if it's something that you'd be for.

The question is, do you want to have translations outside of UI, if not, I believe that my idea is not half bad - that way you only rely on dummy traits in Atk4/core, and when used together with Atk4/ui it'd simply switch to it.

I can write a usage example of my idea, if you'd like.

DarkSide666 commented 5 years ago

Translation mechanism should be usable outside of app and UI.

For example, if I only use atk4\data, but not atk4\ui then I should be able to use translated exception messages coming from atk4\data. Theoretically I can use atk4\data or atk4\dsql alone in some other UI framework.

pkly commented 5 years ago

We've also talked about it on the atk/core pr listed above, which would make all the elements use a trait and a translation class, which should support using core + data (without ui)

romaninsh commented 5 years ago

Closing this. Further discussion is in : https://github.com/atk4/core/pull/93

After merged, we can implement a follow-up PR here to take advantage of new traits.