feat: atlas push pull scripts: FC-55

Amr-Nash commented 2 months ago

Description

This pull request provides the CI scripts and developer tooling for the Atlas Translations Management Design proposal related to OEP-58.

Testing instructions

Testing the `pull_translations`:

make ATLAS_OPTIONS='--repository=Zeit-Labs/openedx-translations --revision=fc_55_sample' pull_translations

When the above command is executed:

1) The None-English translations will be pulled from openedx-translations repo into I18N/ directory. 2) Then each language file will be split and distributed to the modules one by one 3) In the end, the pulled translations will be removed and only the split translations can be found in the modules' directories

If the script is working as intended:

1) The above command should not produce errors when executed. 2) The pulled translations should be correctly distributed among the modules in the expected paths. 3) The I18N directory should not exist after the script has been executed. 4) The app should have all its translations in place and can now be deployed.

Testing the `extract_translations`:

make extract_translations

When the above command is executed:

1) The English sources from all values/strings.xml files in all modules will be extracted 2) The keys of each translation entry will be updated to contain the name of the module followed by a dot e.g. "Module_name.OLD_KEY_of_the_entry". 3) Then all of the updated entries will be put in one file: I18N/src/main/res/values/strings.xml

If this script is working as intended:

1) The above command should not produce errors when executed. 2) Every translatable entry from the English sources of each module should exist in I18N/src/main/res/values/strings.xml with the corresponding comment, if present, placed before it. 3) The keys in I18N should contain the name of the module from which the entry originated, as specified. 4) The values and comments should be identical to the original ones.

Q and A

1. When are source strings extracted? what do the files look like?

An automated action will be developed to extract the English sources if they are changed in the Android repo, then the action will push the sources to the translations repo.

The sources will be in one file: I18N/src/main/res/values/strings.xml and it will look like this:

<resources>
<string name="Module_one.FIRST_ENTRY_KEY_IN_MODULE_ONE">English Translation for the first entry in module one.</string>
<string name="Module_one.SECOND_ENTRY_KEY_IN_MODULE_ONE">English Translation for the second entry in module one.</string>
    .
    .
<string name="Module_two.FIRST_ENTRY_KEY_IN_MODULE_TWO">Translation for the first entry in module two</string>
    .
    .
</resources>

2. Does anything happen to those files before they are added to openedx-translations

The only extracted file is the English sources from the modules in the Android repo as it is the only translation that lives there.

3. When translation files are pulled via atlas what do the files look like?

When the translation files are pulled via atlas the files' structure is going to look like the structure below:

I18N/src/main/res/
----values-ar/
--------strings.xml
----values-uk/
--------strings.xml
----values-es/
--------strings.xml

As an example the values-es/strings.xml would look like: I18N/src/main/res/values-es/strings.xml

<resources>
    <string name="Module_one.FIRST_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la primera entrada del módulo uno.</string>
    <string name="Module_one.SECOND_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la segunda entrada del módulo uno.</string>
        .
        .
    <string name="Module_two.FIRST_ENTRY_KEY_IN_MODULE_TWO">Traducción de la primera entrada del módulo dos.</string>
        .
        .
</resources>

4) What happens to those files after `atlas pull` to make them work in the app build process?

After atlas pull is completed, all language translation files are retrieved, similar to the process described in the previous question. Subsequently, a Python script is executed to process each file. This script iterates through each translation file, splits it, removes the module name from the keys, and organizes each entry into its corresponding module.

For instance, the I18N/src/main/res/values-es/strings.xml file is split into two separate files:

Module_one/src/main/res/values-es/strings.xml
Module_two/src/main/res/values-es/strings.xml

Inside Module_one/src/main/res/values-es/strings.xml, the content would resemble:

<resources>
    <string name="FIRST_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la primera entrada del módulo uno.</string>
    <string name="SECOND_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la segunda entrada del módulo uno.</string>
        .
        .
</resources>

Similarly, inside Module_two/src/main/res/values-es/strings.xml, it would appear as:

<resources>
    <string name="FIRST_ENTRY_KEY_IN_MODULE_TWO">Traducción de la primera entrada del módulo dos.</string>
        .
        .
</resources>

Once the Python script completes its task and the files are split, the make script removes the entire I18N directory and its contents.

5) Why is this needed?

This new process is being introduced to have the best combination of Developer Experience and Translator Experience.

The best experience for Translators requires combining source strings into as few transifex resources as possible. The best experience for Engineers requires splitting translation source files to fit within the modular architecture.

6) What should live in openedx translations?

Translations that were done by the open-edx translators are going to be stored in openedx translations.

7) What should live in atlas?

Atlas is the tool that we are using to pull the translations from openedx translations; therefor, no translations are stored in atlas.

8) What should live in the openedx-app-Android repo?

In the Android app repo only the English sources are going to be stored.

TODO

[x] Refactor the code into a single Python script to improve code reuse
[ ] Add CI scripts for make extract_translations
[ ] Test with the Android team
[ ] Review by Brian Smith

openedx-webhooks commented 2 months ago

Thanks for the pull request, @Amr-Nash! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

Amr-Nash commented 1 month ago

@OmarIthawi could you please review this PR if we are done with the ios PR?

OmarIthawi commented 1 month ago

Tested on both extract and pull (below):

Fresh (before pull)	After pulling Ukrainian translations

OmarIthawi commented 1 month ago

@brian-smith-tcril @volodymyr-chekyrta please take a look and approve the tests to run.

OmarIthawi commented 1 month ago

Thanks @Amr-Nash, please squash the commits to fix the Lint Commit Messages failure https://github.com/openedx/openedx-app-android/actions/runs/9293840766/job/25604477406?pr=317

Amr-Nash commented 1 month ago

Thanks @Amr-Nash, please squash the commits to fix the Lint Commit Messages failure https://github.com/openedx/openedx-app-android/actions/runs/9293840766/job/25604477406?pr=317

Done.

@OmarIthawi

Thanks for addressing the error handling.

My pleasure.

Amr-Nash commented 1 month ago

@OmarIthawi I squashed the commits. Now, it is ready to be merged.

openedx-webhooks commented 3 weeks ago

@Amr-Nash 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

openedx / openedx-app-android