support Google Play upload tools: Fastlane and Triple-T Gradle Play Publisher

[eighthave]

There are two major tools for managing the localized files for Google Play: Fastlane Supply and Triple-T Gradle Play Publisher. F-Droid also supports the same file/dir structure. If Weblate would directly support those files, then there would be a clean, simple, automated way to translate Android apps using Weblate. Here's the ideal workflow:

1. Weblate pulls source from git
2. translators work in Weblate
3. translations get pulled into app's source directly where Fastlane or Triple-T expect them
4. F-Droid automatically uses those translations from git
5. those translations are directly published to Google Play via the respective tool (Fastlane Supply or Gradle)

Both Fastlane and Triple-T use plain text for the source and translation file format. title and _shortdescription would be easy to support directly as plain text, changelog and _fulldescription much less so:

• title: 30 char limit, one line, plain text
• _shortdescription: 80 char limit, one line, plain text
• _fulldescription: 4000 char limit, multiple lines, can contain basic HTML
• changelog: 500 char limit, multiple lines, plain text

But perhaps this setup is just too different from standard translation workflows that assume strings are in a single source file. I've also opened issues on those tools to get feedback there:

• https://github.com/Triple-T/gradle-play-publisher/issues/227
• https://github.com/fastlane/fastlane/issues/9520

I really want to find a way to get this workflow working really smoothly. Translations are really important, and something that really should be as automated as possible.

[eighthave]

Here's one hard part of this whole setup: the hard character limits for _fulldescription and changelog. So 4000 and 500 chars are pretty big blocks of text for a single translation string. But I haven't found a way to set a character limit for a group of strings, only for a single string.

[azachar]

Hello, is here any progress?

[nijel]

AFAIK nobody is working on this right now.

[eighthave]

I applied for a small grant to fund work on this, and related issues. Wish me luck :) I'll let you know.

[PanderMusubi]

Any luck with that grant proposal?

[eighthave]

yeah, I got a small grant to work on this, so I should be able to get something done.

[eighthave]

And finally the grant is started! This is one of the tasks for the grant.

Also, I wanted to point out that Triple-T seems to be implementing an XLIFF-based solution, which would be ideal: https://github.com/Triple-T/gradle-play-publisher/issues/227 I think we should help them with that goal where we can, since using XLIFF will provide lots of benefits.

@matrixcoffee

[nijel]

The basic implementation has just landed in our repo, there still are some rough edges and missing features, but the basics should work:

• It parses title, short_description, full_description, changelogs and video files, setting proper length and flags for those
• Every file is presented as single string in Weblate (see below)
• The discovery should find the correct component configuration
• Support for changelogs still needs to be implemented

I originally wanted to have long descriptions split, but in the end it's not there for following reasons:

• Lot of files seems to wrap paragraphs, what would lead to splitting real paragraphs into more strings, what would probably lead to weird translations
• The split strings would lead to question what to write into the file if only some of the strings are translated
• The maximal length check would become tricky (and especially tricky to present it reasonably to user)
• Weblate should be clever enough to detect changes in the source string and present those to user

It needs some more testing before it can be deployed (I will get to that later this week), but any feedback is welcome.

[eighthave]

Sounds great! How can we test it? The git repo for https://hosted.weblate.org/projects/f-droid/f-droid/ already has the changelogs in it. I guess it would need a new component. Same with https://hosted.weblate.org/projects/f-droid/privileged-extension/

[nijel]

Ah, so it's usual to have only changelogs there? In that case I need to do some further tuning to make this case work...

[eighthave]

I don't know how common it is, but it is valid in fdroid to only have one of the elements, e.g. only changelogs, only video link, only summary/short description, etc. It could also be some partial mix of the elements, like the summary and description, but nothing else.

I'm not sure how Fastlane or Triple-T handles incomplete file arrangements.

One thing to also note is that the basedir for both Fastlane and Triple-T can be configured. For example, fastlane supply and fdroid work with both of these:

• metadata/en-US
• fastlane/metadata/android/en-US

[nijel]

I've just adjusted the format to behave correctly in this case. I think I will be able to deploy this feature tomorrow and can enable it for one of the projects for real life testing. So just let me know which one do you prefer to test and I'll configure it tomorrow.

[eighthave]

Excellent! I think f-droid/f-droid is a good one to start with.

[nijel]

Great, it's now alive: https://hosted.weblate.org/projects/f-droid/f-droid-metadata/

Please report any issues with it as it's first multi file backend in Weblate and even though it has been tested, I expect some corner cases to discover bugs.

[eighthave]

Ok, its live! Translators have started working on it! I just pushed some existing source and translation files for description.txt and they were not picked up by Weblate. It is still only the changelogs.

[eighthave]

Nevermind, I had the filenames wrong. fdroid supports alternate names from Fastlane, but we don't need them supported by Weblate.

[nijel]

Indeed the description.txt is not parsed. So far I've only seen short_description.txt and full_description.txt. Is there some specs which files are supported? I can make it load all *.txt files as well, but I thought that might be confusing to pick up files which will not be used by the tools.

[nijel]

Adding additional files is pretty easy:

https://github.com/WeblateOrg/weblate/blob/0a3d373e2d88d73246d9e8f03334e4a3cde2d9a2/weblate/formats/txt.py#L95-L102

And related part in discovery:

https://github.com/WeblateOrg/translation-finder/blob/3b81c1f172605a9072f0443424f31c38be2c9723/translation_finder/discovery/files.py#L166-L175

[eighthave]

The current way makes sense, I think it is good to keep it simple.

The most recent push just triggered a bug. For some reason, there is a commit from the Norwegian translator adding copies of the en-US changelogs but not in the changelogs/ dir:

commit 2050df1382e3f587c39658125fd1dee78df59118

From 2050df1382e3f587c39658125fd1dee78df59118 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Allan=20Nordh=C3=B8y?= <epost@anotheragency.no>
Date: Thu, 14 Feb 2019 19:38:50 +0000
Subject: [PATCH 1/1] Translated using Weblate (English (United States))

Currently translated at 100.0% (15 of 15 strings)

Translation: F-Droid/F-Droid metadata
Translate-URL: https://hosted.weblate.org/projects/f-droid/f-droid-metadata/en_US/
---
 metadata/en-US/1000000.txt |  7 +++++++
 metadata/en-US/1000001.txt |  3 +++
 metadata/en-US/1000002.txt |  7 +++++++
 metadata/en-US/1000003.txt |  3 +++
 metadata/en-US/1000004.txt |  7 +++++++
 metadata/en-US/1000005.txt |  3 +++
 metadata/en-US/1000010.txt | 21 +++++++++++++++++++++
 metadata/en-US/1000011.txt | 21 +++++++++++++++++++++
 metadata/en-US/1001050.txt |  7 +++++++
 metadata/en-US/1002050.txt | 15 +++++++++++++++
 metadata/en-US/1002051.txt | 17 +++++++++++++++++
 metadata/en-US/1003050.txt | 19 +++++++++++++++++++
 metadata/en-US/1003051.txt | 20 ++++++++++++++++++++
 metadata/en-US/1004050.txt | 13 +++++++++++++
 metadata/en-US/1005050.txt | 13 +++++++++++++
 15 files changed, 176 insertions(+)
 create mode 100644 metadata/en-US/1000000.txt
 create mode 100644 metadata/en-US/1000001.txt
 create mode 100644 metadata/en-US/1000002.txt
 create mode 100644 metadata/en-US/1000003.txt
 create mode 100644 metadata/en-US/1000004.txt
 create mode 100644 metadata/en-US/1000005.txt
 create mode 100644 metadata/en-US/1000010.txt
 create mode 100644 metadata/en-US/1000011.txt
 create mode 100644 metadata/en-US/1001050.txt
 create mode 100644 metadata/en-US/1002050.txt
 create mode 100644 metadata/en-US/1002051.txt
 create mode 100644 metadata/en-US/1003050.txt
 create mode 100644 metadata/en-US/1003051.txt
 create mode 100644 metadata/en-US/1004050.txt
 create mode 100644 metadata/en-US/1005050.txt

diff --git a/metadata/en-US/1000000.txt b/metadata/en-US/1000000.txt
new file mode 100644
index 0000000..d9ff1c0
--- /dev/null
+++ b/metadata/en-US/1000000.txt
@@ -0,0 +1,7 @@
+* Support installing media, OTA, ZIP, etc. files
+
+* Fully support APKs signed by multiple signing keys
+
+* New Tibetan translation
+
+* Remove related apps and categories after disabling a repo
diff --git a/metadata/en-US/1000001.txt b/metadata/en-US/1000001.txt
new file mode 100644
index 0000000..32b0e64
--- /dev/null
+++ b/metadata/en-US/1000001.txt
@@ -0,0 +1,3 @@
+* Fix bug removing apps from repos (#568)
+
+* Much faster index updates
diff --git a/metadata/en-US/1000002.txt b/metadata/en-US/1000002.txt
new file mode 100644
index 0000000..330c5e0
--- /dev/null
+++ b/metadata/en-US/1000002.txt
@@ -0,0 +1,7 @@
+* Prevent HTTP ETag from being used as a tracking cookie
+
+* Improved screenshots layout
+
+* Properly clean up temp and cached files
+
+* Dark mode fixes
diff --git a/metadata/en-US/1000003.txt b/metadata/en-US/1000003.txt
new file mode 100644
index 0000000..cd77ae1
--- /dev/null
+++ b/metadata/en-US/1000003.txt
@@ -0,0 +1,3 @@
+* Big UI performance improvements, especially with archive enabled
+
+* Fixed crasher bugs
diff --git a/metadata/en-US/1000004.txt b/metadata/en-US/1000004.txt
new file mode 100644
index 0000000..d0206dd
--- /dev/null
+++ b/metadata/en-US/1000004.txt
@@ -0,0 +1,7 @@
+* added swipe gestures to the Updates tab
+
+* display warnings with actions in Updates tab for KnownVulns
+
+* translation updates
+
+* dark UI fixes
diff --git a/metadata/en-US/1000005.txt b/metadata/en-US/1000005.txt
new file mode 100644
index 0000000..d85beb2
--- /dev/null
+++ b/metadata/en-US/1000005.txt
@@ -0,0 +1,3 @@
+* Fix bug that prevented translations from showing up on Android >= 7.0
+
+* Fix DB upgrade crash from 1.0-alpha3 → 1.0-alpha4 #1181
diff --git a/metadata/en-US/1000010.txt b/metadata/en-US/1000010.txt
new file mode 100644
index 0000000..6eb1286
--- /dev/null
+++ b/metadata/en-US/1000010.txt
@@ -0,0 +1,21 @@
+F-Droid is 1.0!
+
+This release includes lots of big changes:
+
+* Completely overhauled workflow for updating apps
+
+* Fully translatable app summaries and descriptions
+
+* "What's New" section to show changes in current release
+
+* Screenshots and feature graphics
+
+* Support installing media, OTA, ZIP, etc. files
+
+* Improved protection against tracking (HTTP ETag, TLS, etc.)
+
+* Fully background updates with Privileged Extension
+
+* Highlight donations to app developers
+
+* Much faster index updates
diff --git a/metadata/en-US/1000011.txt b/metadata/en-US/1000011.txt
new file mode 100644
index 0000000..58bc6d4
--- /dev/null
+++ b/metadata/en-US/1000011.txt
@@ -0,0 +1,21 @@
+F-Droid is 1.0!
+
+This release includes lots of big changes:
+
+* completely overhauled workflow for updating apps
+
+* fully translatable app summaries and descriptions
+
+* "What's New" section to show changes in current release
+
+* screenshots and feature graphics
+
+* Support installing media, OTA, ZIP, etc files
+
+* improved protection against tracking (HTTP etag, TLS, etc.)
+
+* fully background updates with Privileged Extension
+
+* highlight donations to app developers
+
+* much faster index updates
diff --git a/metadata/en-US/1001050.txt b/metadata/en-US/1001050.txt
new file mode 100644
index 0000000..4c93bf7
--- /dev/null
+++ b/metadata/en-US/1001050.txt
@@ -0,0 +1,7 @@
+* Support for official mirrors for any repo, with auto-selection based on availability
+
+* Lots of UI polish, as well as stability and bug fixes
+
+* New privacy prefs: Turn off screenshots; exit on panic
+
+* New languages: Korean (ko) and Vietnamese (vi)
diff --git a/metadata/en-US/1002050.txt b/metadata/en-US/1002050.txt
new file mode 100644
index 0000000..8dc8301
--- /dev/null
+++ b/metadata/en-US/1002050.txt
@@ -0,0 +1,15 @@
+* Automatically trim or delete cache when storage space is low
+
+* Improved performance on low memory devices
+
+* Make all downloads respect "Only on Wi-Fi" preference
+
+* Reduce memory usage when device is running low
+
+* Add custom mirrors to any repo by clicking links, scanning QR codes, etc.
+
+* Lots of swap/nearby bug fixes and improvements
+
+* Fix one cause of reoccurring update notifications (#1271)
+
+* Make F-Droid recognize F-Droid nightly URLs from GitLab
diff --git a/metadata/en-US/1002051.txt b/metadata/en-US/1002051.txt
new file mode 100644
index 0000000..9232041
--- /dev/null
+++ b/metadata/en-US/1002051.txt
@@ -0,0 +1,17 @@
+* automatically trim or delete cache when storage space is low
+
+* improved performance on low memory devices
+
+* make all downloads respect "Only on Wi-Fi" preference
+
+* reduce memory usage when device is running low
+
+* add custom mirrors to any repo by clicking links, scanning QR codes, etc
+
+* improved automatic mirror selection
+
+* lots of swap/nearby bug fixes and improvements
+
+* fix one cause of reoccuring update notifications (#1271)
+
+* make F-Droid recognize fdroid nightly URLs from GitLab
diff --git a/metadata/en-US/1003050.txt b/metadata/en-US/1003050.txt
new file mode 100644
index 0000000..f2d946b
--- /dev/null
+++ b/metadata/en-US/1003050.txt
@@ -0,0 +1,19 @@
+* More reliable buttons and status updates
+
+* Overhaul of Settings including new data usage controls
+
+* Battery conscious background operation on Android 5.0 and newer
+
+* Share menu item for repos
+
+* Turn off crash reports in Settings
+
+* Install History viewer when logging is turned on
+
+* Clickable Anti-Features list, added NoSourceSince
+
+* Android 4.0 is now the minimum supported
+
+* Huge improvements for custom whitelabel F-Droid versions
+
+* Expert Setting to send debug version+UUID on download
diff --git a/metadata/en-US/1003051.txt b/metadata/en-US/1003051.txt
new file mode 100644
index 0000000..16393e7
--- /dev/null
+++ b/metadata/en-US/1003051.txt
@@ -0,0 +1,20 @@
+v1.3.1
+* Big overhaul of core nearby/swap plumbing
+
+* TLSv1.3 support, when the device supports it
+
+v1.3
+* More reliable buttons and status updates
+
+* Overhaul of Settings including new data usage controls
+
+* Battery conscious background operation on Android 5.0 and newer
+
+* Share menu item for repos
+
+* Turn off crash reports in Settings
+
+* Install History viewer when logging is turned on
+* Clickable Anti-Features list, added NoSourceSince
+
+* Android 4.0 is now the minimum supported
diff --git a/metadata/en-US/1004050.txt b/metadata/en-US/1004050.txt
new file mode 100644
index 0000000..7d75e26
--- /dev/null
+++ b/metadata/en-US/1004050.txt
@@ -0,0 +1,13 @@
+* Huge overhaul of the "Versions" list in the App Details screen, and many other UI improvements, thanks to new contributor @wsdfhjxc
+
+* Fix keyboard/D-pad navigation in many places, thanks to new contributor @doeffinger
+
+* Show "Open" button when media is installed and viewable
+
+* Add "Share" button to "Installed Apps" to export CSV list
+
+* Add clickable list of APKs to the swap HTML index page 
+
+* Retry index downloads from mirrors
+
+* Fix "Send F-Droid via Bluetooth" on recent Android versions
diff --git a/metadata/en-US/1005050.txt b/metadata/en-US/1005050.txt
new file mode 100644
index 0000000..6a30356
--- /dev/null
+++ b/metadata/en-US/1005050.txt
@@ -0,0 +1,13 @@
+* support swapping via SD Cards
+
+* Nearby swap bug fixes and improvements
+
+* more improvements on the "App Details" screen
+
+* update language and translations about Nearby and swap
+
+* improved app icon selection logic
+
+* add repos via built-in additional_repos.xml from ROM, OEM, Vendor.
+
+* display versionCode in expanded Versions list entries in Expert Mode
-- 
2.7.4

[eighthave]

Another idea: it would be great to automatically mark some of the strings with higher priority so translators work on them first:

• title.txt
• _shortdescription.txt
• changelog/*.txt file with the highest number.

The value of translating old changelogs rapidly declines with time, so really the more there, the lower priority they should be, as their number gets smaller. That numbering scheme is the Android Version Code, which is guaranteed to be a 32-bit integer, with the highest value always being the latest release.

[nijel]

Hmm, @comradekingu was editing source strings. Actually it seems that changelogs handling is broken in most cases, I'm looking at it now...

[comradekingu]

@nijel I was, it works, part of the time… What I did initially was middleclick (new tab) open a bunch of edit links from Zen. Then editing one after the other. With varying result, though changes always show in the diff below. Then did just the one string, and that worked.

[nijel]

The files are shown in order defined in the code, so the only thing which does not have priority right now is latest changelog entry.

[nijel]

I've now fixed git history on Weblate side and pushed fix for changelogs handling, so hopefully that part is now okay.

[comradekingu]

Worked for me.

[eighthave]

@janpio @romanr @taquitos @mpirri pinging you here since https://github.com/fastlane/fastlane/issues/9520 was closed and locked. @nijel has implemented direct support in Weblate for the existing Fastlane file layout. A huge win! No plugins or changes to Fastlane needed!

[janpio]

Awesome @eighthave - and thanks @nijel! (I added a link back here to the closed/locked issue over at fastlane.)

[eighthave]

I just realized, I can already predict diffs/merge conflicts based on whether there is a trailing newline on the Fastlane files. Fastlane people: is there a canonical format for title.txt, summary.txt, etc. regarding leading and trailing spaces? Since Weblate is based on git, those changes matter when managing merging.

[janpio]

I don't think fastlane has any preferences here, it just takes whatever is in the files.

[eighthave]

When fastlane writes out title.txt and summary.txt based on the data from Google Play, it would have a format that it would use to write. I think we want to standardize on that format. Maybe fastlane just takes what comes from Google Play and writes it out, or maybe it strips leading/trailing whitespace. F-Droid sends the exact, unchanged data to the client, then the client strips as needed.

[janpio]

Could you create an issue over at fastlane and mention me? Then I can investigate at the fastlane codebase a bit and not spam all the other people here ;) Thanks.

[eighthave]

Two small bug reports after using this for a bit:

• The character counts are wrong for at least some strings. For example, an English changelog file that is 498 bytes on disk and wc --chars says 498 characters is showing up in Weblate as 519 characters. https://hosted.weblate.org/translate/f-droid/f-droid-metadata/ru/?checksum=1ed204482ed782d6#comments

• commit 9e338694a46f248ad259fb4f4dce4c1f0342cb0a in f-droid/f-droid-metadata has come back after I removed it and force-reset Weblate on the upstream git repo. It is deleting most of the source strings. I asked @comradekingu if he knew why, since it shows as his commit, and he wasn't sure.

[nijel]

The 500 chars limit for translation is not coming from string length, but is hardcoded based on your initial comment in this issue. And I have no clue how somebody came to 519 chars...

If that limit is wrong it can be certainly changed. This again opens up same question: is there some specification for all this, or it's just randomly placed text files with random semantics depending on used tools?

[nijel]

And the commit is from Fri Feb 15, so it probably was not removed on Weblate side? Are you sure it was removed from upstream repo when doing reset on Weblate side?

[eighthave]

• I'm sure about those character limits, that's what we implemented in F-Droid after lots of research. I can dig up the references if you need them. I asked the translator where they got the 519 number from.

• About the 9e338694a46f248ad259fb4f4dce4c1f0342cb0a commit, I made a local branch of the commits I got from weblate, then removed the commit that removed the source strings, then merged that into our repo. Then I went to the Weblate Repository Maintenance web interface and clicked on "Reset".

[nijel]

I guess the 519 counts whitespace chars indicated by Weblate. But still the problem is that there is (nearly) 500 chars string which is supposed to be translated into 500 chars, what is not really easy for many languages.

I've reproduced the issue with wiping strings locally, working on a fix.

[nijel]

I've just deployed new version which should not produce such commits.

[comradekingu]

@nijel You get to 519 by copying the source string with the button, that way you get the same number as the source to work with. Some of the strings 1, 2 and 3 are empty. It seems like 1 and 2 weren't meant to be, as they previously had content that isn't found in other strings.

[nijel]

That's caused by bug discussed above, I've just discarded broken commits and forced update, should be fixed now.

[comradekingu]

@nijel Now none of the strings are corrected though. Is it safe to edit them in Weblate?

[nijel]

Yes, it should be safe now (it discarded changelog edits, sorry).

[nijel]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see problem with the fix, please comment on this issue.
• In case you see similar problem, please open separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[cetti]

@eighthave where did the grant come from?

[eighthave]

https://www.iscproject.org/