Support Magento2 as Storage Format

hostep commented 1 year ago

Hi there

Update: the initial question is no longer relevant now that we've changed the title in this issue

Just playing around, we are using csv files for translations (used by Magento2 e-commerce) which this tool doesn't support yet, but out of curiosity I quickly converted them to json to be able to play around a bit with this tool to see how well it works and if it would be useful for us or not.

But I noticed something strange when dots (.) exist in the translations:

Here's an example json file:

{
    "An unknown error occurred.": "Een onbekende fout is opgetreden.",
    "Searching...": "Zoeken..."
}

After running vendor/bin/phpunuhi fix:structure it turns this json into something very strange:

{
    "An unknown error occurred": {
        "": "Een onbekende fout is opgetreden."
    },
    "Searching": {
        "": {
            "": {
                "": "Zoeken..."
            }
        }
    }
}

It looks like it converts a . character into a new key-value each time, which is kind of unexpected.

Or am I missing some configuration setting somehow?

Thanks!

boxblinkracer commented 1 year ago

Hi there

thank you for this

this is because of the key names. JSON is a nested format, therefore it uses "." in the keys for the nesting levels

I never saw key names of translations that look like this: "Searching...". is that really valid in your case?

thank you

hostep commented 1 year ago

Yeah, in Magento we don't use keys for translations, we use the actual English phrases that are used in the application, and those phrases are then translated in different languages via csv files (one csv file per locale, where the english phrase is in the first column and the translated phrase in the second column, here's an example should you be curious).

Anyway, if you believe this is not something your tool should support for json files, then no problem, I'm just doing some experiments at the moment to see if the tool can do what we would need in our specific case. If it turns out to be useful and we want to invest some time into it, maybe I can try to make some PR's to add support for Magento2 csv files. But no promises yet 😄

boxblinkracer commented 1 year ago

ah ok now it makes sense so in that case your assumption was wrong to convert it into JSON

this can totally be done with phpunuhi

what is missing for you is a CSV storage format. so it could either be a magento one, or...maybe better a "csv" format where it would be possible to use either no key column (your case), or even a key column..

also, all translations are in 1 file...so theres the same approach with INI at the moment, where its possible to define the iniSection of a locale. so in your case it would also be the same file, and one needs to provide the csvColumn

you know what i mean? what do you think, should i try it? or do you want to give it a try?

boxblinkracer commented 1 year ago

aa sry, i was wrong...the key is the first one, and then theres only 1 other locale because every locale has a separate file

in that special case...maybe a special magento format :D

hostep commented 1 year ago

It indeed sounds like an idea is shaping here

However, the Magento case is even more complex, because there are a lot of different layers of translations involved where we can have parent and child translations packs where childs can append and overwrite translations from parents. Like:

Magento community language packs or 3rd party language packs as the base
custom language packs we build ourselves to install in all our projects
some modules come with their own translations
custom language packs inside a project
custom translations in one or more of the custom themes or modules

What we would need is that all those different layers are being merged together, with the correct way of overwriting translations, and at the end we would need to compare the result between different languages to be able to find missing ones, and output those missing ones in a specific file (usually the custom language pack inside a project)

So this sounds quite complex and might be a bit out of scope of this package here. But the package here definitely contains already some nice functionalities that we could use after some quick testing!

boxblinkracer commented 1 year ago

i dont think its out of scope, not really

so in shopware 6 for example there are 2 options a) plugin manufacturers provide plugins. in that case the JSON format is used, because your deliverable is this one b) when testing translations for the shop itself, the shopware6 storage is used which directly connects to the database. it also uses an additional dimension for products...meaning translations for a product are grouped into an identifier for that product. then you can even translate content data....

im not really good with magento, but i think its possible (and i need to learn it a bit more, but i know a few people that could also help). i think its important to differ between what is the deliverable to be tested (those language packs) vs. do I want to test my final result in the shop?

i hope that makes sense a bit?

if so, then I would just rename this ticket to "support magento" :D ?

hostep commented 1 year ago

Okay great

Here's a proposal, not really thought through but might be a good start. I would propose we have something like this that we can configure:

...
        <set name="Storefront">
            <locales>
                <format>
                    <magento2>
                        <output name="de_DE">
                            /path/to/the/custom/translation/file/for/the/project/de_DE.csv
                        </output>
                        <output name="en_US">
                            /path/to/the/custom/translation/file/for/the/project/en_US.csv
                        </output>
                    </magento2>
                </format>
                <locale name="de_DE">
                    <file sortOrder="1">
                        /path/to/base/language/pack/de_DE.csv
                    </file>
                    <file sortOrder="2">
                        /path/to/vendor/language/pack/de_DE.csv
                    </file>
                    <file sortOrder="3">
                        /path/to/modules/with/glob/support/**/i18n/de_DE.csv
                    </file>
                    <file sortOrder="4">
                        /path/to/the/custom/translation/file/for/the/project/de_DE.csv
                    </file>
                </locale>
                <locale name="en_US">
                    <file sortOrder="1">
                        /path/to/base/language/pack/en_US.csv
                    </file>
                    <file sortOrder="2">
                        /path/to/vendor/language/pack/en_US.csv
                    </file>
                    <file sortOrder="3">
                        /path/to/modules/with/glob/support/**/i18n/en_US.csv
                    </file>
                    <file sortOrder="4">
                        /path/to/the/custom/translation/file/for/the/project/en_US.csv
                    </file>
                </locale>
            </locales>
        </set>
...

So things that might be important:

some files aren't always under our own control and can be installed via composer, so we can't just write missing translations to those, hence why I suggest we specify one output file per locale where all the missing translations are put into
we need some kind of glob syntax support, because in the course of the lifetime of a project you'll install modules and remove others and if we would need to specify every one of those new paths and remove non existing ones, this would become tedious
the sortOrder attribute could be used to determine in which order the files need to be read so that the correct inheritance level is being followed, so that the same translation from a higher sortorder overwrites one from a lower sortorder

This is just a proposal, no idea how difficult it would be to implement this with what is already here, without having to rewrite a lot of code all over the place.

Still investigating on our side, I need to propose this to some team members and see what they think. If approved, we can try to help out with building this feature request.

boxblinkracer commented 1 year ago

hi there this is great, and indeed a lot to do

dont be worried that i didnt answer yet ill check it out, but please give me a bit time, i guess its a lot :)

hostep commented 1 year ago

Small update from our part, because we've decided to take another approach and a colleague of mine already finished an initial working prototype. Unfortunately it's in private code part of a bigger project we can't open source.

So for now we won't need this functionality anymore from the phpunuhi project. Sorry and thanks for the feedback you've provided here.

You can close the request if you think it won't be helpful to other people.

boxblinkracer commented 1 year ago

Hi

ok thanks for letting me know, but I would still try to add it :) so I'll keep it open

thanks for your input on this

boxblinkracer / phpunuhi

Support Magento2 as Storage Format #14