Enable Transifex in Mardown/JSON based content workflow

[jmatsushita]

As described here the process to translating content in the Panic Button app is too complex.

Some of it is the result of separating content for the wizard (in the assets folder) which is generated from a jekyll based site, allowing content editor to use Prose to submit content updates to the github repo which trigger a rebuild of the site, including a json version which used to be consumed directly by the app. It is now generated once and baked into the APK. The advantage of this is that it allows content editors to be able to modify the content of the website and of the wizard from a single, simple interface. The downside is that it is a non standard way to manage content in an Android app, adds an extra step in the generation of the APK, and is currently implemented as a content database which slows down the startup of the app (#75).

This needs work and support!

Related to #43, #50.

[niccs]

@jmatsushita , https://www.transifex.com/otf/panic-button/ -- I think, this above project is for the panicbuttion.io website. So, as per my understnading, for the PB app a) new organization/project should be created in Tansifex. b) Then uploading all the resource files there at a central place. c) There will be no change , how the android app is parsing the json files and storing them in local sqlliteDB (which is best way to use data in the app, having faster access to localDB). d)The latest translated files will be pulled while creating release apk.

Please give inputs.

[jmatsushita]

Hi @niccs.

a) yes this is correct. I'll get in touch with the localisation lab. b) actually it would be good to setup the txgh workflow described here (http://docs.transifex.com/integrations/github/) but in the meantime upload will do. c) so for a start we can upload both the Android XML String files and our custom JSON files. But I'd like us to refactor the JSON and SQLite database. I don't think there is a reason to keep them now. It was useful during development to make the wizard flexible and allow non-tech users to modify the content via the Jekyll site (it used to be synced with this file https://panicbutton.io/api/mobile.json) but since we removed this feature, and given that the UX won't change, and given that adding this complexity can be a source of bugs or security weaknesses I think we should progressively remove it and transform our fragments back into simple Android fragments. From a translation stand point this should mean that we only have Android XML String files at the end. d) With txgh this process should be automated, but until then we can indeed do so.

Let me know if that's clear or if you have more questions! Feel free to break down this issue into several smaller issues.

Cheers,

Jun

[jmatsushita]

Regarding a) I've created our own account on Transifex and we'll migrate it when needed. Can you create an account and I'll make you admin? https://www.transifex.com/panicinitiative/

[jmatsushita]

Regarding c) I can't upload our current JSON files and get a Values must be strings. Key "version" is of type int and has a value of 9. message given that Transifex is expecting all keys and values to be strings. Can you think of a workaround? If not then we might have to start refactoring the fragment sooner than I thought. Let me know your thoughts.

[niccs]

@jmatsushita,

1) I have made changes to accept version number in String format. Merged the changes in dev branch.

2) https://www.transifex.com/panicinitiative/ link is still not active/working. I have my account with the same email id. This is the link to my Transifex profile (linked with Github account):- https://www.transifex.com/user/profile/Nikks/

[niccs]

@jmatsushita, when are we planning to have the code refactoring for removing SqlLiteDB and json? This will be a big change ,covering changes in whole application and so will require thorough testing also. So sooner we target , better it would be.

[jmatsushita]

@niccs I have added your account as an admin of https://www.transifex.com/panicinitiative/

[jmatsushita]

@niccs, Yes let's start doing this in a separate "refactor" branch. I think it makes sense to start now since it's potentially a lot of changes as you said..

[niccs]

@jmatsushita, I will take this on and later try uploading changed files to Transifex.

[jmatsushita]

@niccs did you try to upload the json file with the version number as string format?

[niccs]

@jmatsushita, I tried uploading it, but it is giving other format issues(The format expects key-value .json format,however,in action field, there is no mapping). Do you still want me to look and fix the json file format? or as per our discussion, can I start merging .json files to Strings.xml? Please advise.

[jmatsushita]

Thanks for the report. Please carry on merging into the Strings.xml On 23 Oct 2015 11:18, "Neetika M" notifications@github.com wrote:

@jmatsushita https://github.com/jmatsushita, I tried uploading it, but it is giving other format issues(The format expects key-value .json format,however,in action field, there is no mapping). Do you still want me to look and fix the json file format? or as per our discussion, can I start merging .json files to Strings.xml? Please advise.

— Reply to this email directly or view it on GitHub https://github.com/PanicInitiative/PanicButton/issues/82#issuecomment-150536200 .

[jmatsushita]

From @niccs :

But I'd like us to refactor the JSON and SQLite database. I don't think there is a reason to keep them now. It was useful during development to make the wizard flexible and allow non-tech users to modify the content:

For every common fragment type, there are set of screens. Like e.g, if fragment type is “Simple”, Setupdisguise,Setup Alarm,SetupAlert and many more screens uses a same SimpleFragment code. And UI data for that particular screen is fetched from the json object array.

Benefit of this approach is, if in case we want to add a new UI screen functionality in furure , we just need to add the corresponding new json object entry , and screen UI design with basic functionality will be up. Now Strings.xml (for android applications)stores either simple Strings or String arrays( and we are looking for some hashMap resource.) So again it will be like storing Json data inside a String as follows:

We can put the data in Strings.xml as json values and then inside our screen, we can build the HashMap on the start of screen.

<string name="data">{"id":"setup-disguise","type":"simple","title":"Setup Disguise"}</string>

and everytime a screen is loaded the data is parsed then UI screen is loaded, which is almost same as having .json files.

Also in every UI, action buttons are also dynamic, based on the json file action tag, the number of buttons, the button texts and there action (next UI screen for navigation) is decided.

So in PB app’s UI screen, there are no static set of Strings , which corresponds to a particular UI, it is all dependent on the JSON object array and is fetched at run time.

These JSON files are not the simple resource files, these are serving as the base of the PB app design. Parsed JSON files are mapped to POJO’s , and these POJO are used everywhere in PB app. Same POJO’s are mapped to content DB.

If we are planning to remove the DB layer, then every time a UI screen is loaded, the json parsing will be done which according to me will be slower than fetching it from local content DB which is a one time job at the start. And making each fragment lightweight with Strings.xml doesnot seems feasible with the current PB design. It requires changes in the way app is designed.

So, my view point is:-

For performance improvement, what you suggested in #75 (when we load the app, only OS language specific data should be parsed and put into DB, and others are triggered when language is changed ) is the better approach. Other than performance, for translations we need to follow the .json files. I will check transifex for .json translations meanwhile.

[jmatsushita]

Hi @niccs thanks a lot for the analysis. All of this was the result of decisions to make the system flexible and driven by markdown files (with frontmatter) in github. The markdown files (such as this one) would then be built (using this jekyll template) into a versioned JSON file. Note that we also did the same thing to share the Help section between the Android app and the Website.

I have been thinking also that maybe we don't want to loose the flexibility of the system, but I was also attracted to the possibility of simplifying it, given that it can be a source of bugs and therefore security flaws. Could you do a little bit of research to see if others are using a similar approach or not, so that maybe we could move to using a library instead of doing it ourselves? There is also the possibility to build this into a library... Do you think that would be a possibility?

Please note that storing JSON in the string is not satisfying given that we want to make the translation easier.

So maybe a better approach is to do as follow:

• Keep the flexible UI system and start thinking about it as a library.
• Go back to thinking of the panicbutton.io github repo as being the master for the android content. (this might require some changes to be done upstream to the content).
• Include a step in the build process to download the api/*.json files (maybe remove the versioning and just compare to remove the manual steps needed previously in the Jekyll build).
• Manage the translation at the level of the markdown files (this would mean doing the reverse of what I've been saying and moving things out of the strings.xml and into the markdown). I've tried this but Transifex only manages plain text files or YAML separately, not markdown with a frontmatter (which creates lots of noise for the translator with keys that are relevant and others that shouldn't be translated).

I've submitted a feature request to transifex, but in the meantime I propose that we look into dealing with the markdown/YAML upload problem with a simple text processing step inside our translation workflow. It would need to maintain sync between It could go something like this:

• After updating the english. Or maybe adding a git pre-commit hook to check the api/*.json file for changes or simply use git? (do we need the json remote generation step anymore? Would it be better to do this on the build machine?)
• If there is a change then pull the _posts/mobile and _posts/help folders from the panicbutton.io repo.
• process each .md file into a different folder (locales for example) and for each .md file create 2 files, one .yaml file and one .txt file (maybe a .yaml_ignore file for ignored yaml keys).
• tx push the locales folder.
• After updating the translations on transifex
• `tx pull and recreate the original markdown files from the yaml and txt files and git commit them.

There's probably some steps to look into when adding languages (since right now it needs modification of the code), but that should be manageable.

What do you think?

[jmatsushita]

Work on this is happening on the transifexbranch. https://github.com/PanicInitiative/PanicButton/compare/dev...transifex