ApkTrack fails to retrieve version information for some APKs from the net

[Olf0]

ApkTrack always fails to read update information from the Öffi and Guardian Project: "No update information found" But the F-Droid client has no issues reading and parsing the repository data from the Guardian Project and Öffi. Tested with ApkTrack 2.1.1, 2.1.1b and the current test version.

[JusticeRage]

The web page previously used for the Guardian Project was not returning the right information. I will probably integrate support for their F-Droid repository directly into the ApkTrack Proxy as it's easier for me to update it than push an ApkTrack update every time something changes. As for Öffi, I haven't touched this one so I'll have to investigate.

[Olf0]

I investigated the unreliability when querying APK sources for current APK versions a bit further with ApkTrack 2.1.1 (from f-droid):

• Querying the Öffi repository always fails, but there is only a single APK to test available there: Öffi Querying Öffi's repository metadata is working fine in the F-Droid client, after adding https://oeffi.schildbach.de/fdroid/repo to its repository list. I wonder, if you are using "web scraping" for Öffi or if ApkTrack is reading aforementioned repository metadata in f-droid's format. Note, that there has been an issue with the F-Droid client (versions I did not use) reading Öffi's repository metadata (IIRC Öffi's repository metadata uses an old version of f-droid's format), but unfortunately I fail to find it on Gitlab (searching for "öffi" versus "oeffi", and in various of the F-Droid's projects issue trackers there: client, fdroiddata etc.?), anymore.

• Querying the Guardian Project repository always works for Orfox, but fails for other APKs installed from there (tested a few) with "Unable to read version information."!
  Querying Guardian Project's repository metadata is working (mostly) fine in the F-Droid client, after activating it in the repository list there.

• Querying the F-Droid repository works for most APKs, but fails consistently for a few (e.g. VLC) with "No update information found". Querying F-Droid's repository metadata is working fine in the F-Droid client for all APKs available there.

• I have not tested querying the Xposed stable repository with ApkTrack intensively, just tried a single APK, which works.

• Querying PlayStore seems to mostly work technically, but often returns "Varies, depending on device" as version information, which is useless. But it also fails for a few APKs, e.g. Öffi: "Unable to read version information."

• The ApkTrack Proxy seems to have issues on its own: I believe I have seen it successfully querying version information for APKs it now fails to withb"No update information found". But I suppose that there are no issues for ApkTrack to query the ApkTrack Proxy, as both are under your control.

So my impressions are:

• All F-Droid based repositories (F-Droid itself, Guardian Project, Öffi) seem to be somewhat problematic for ApkTrack.

• The Guardian Project repository is not especially problematic, compared to the F-Droid and Öffi repositories.

• Even PlayStore shows a few failures.

• ApkTrack Proxy is a different story, which needs more testing and constitutes a different issue.

[Olf0]

And I retested this with APKtrack_test.apk (2017-09-04 22:51) to see, if commit 70531d6 is already in place: Obviously not, behaviour is unchanged, except for Öffi's version information is now successfully queryable from PlayStore (resulting in "Varies, depending on device"), but still not from Öffi or ApkTrack Proxy.

Edit: Oh, I see, commit 70531d6 was in the beta branch, the latest test version was likely built from the stable branch.

[Olf0]

Retested with ApkTrack 2.1.2:

• Querying the Öffi app now works flawlessly from Öffi, ApkTrack Proxy and Play Store. Nice, thank you.

• Querying Orbot (originally from the Guardian Project, but also available at F-Droid) is also working flawlessly from Guardian Project (!), F-Droid and ApkTrack Proxy. I believe this has been the case with v2.1.1, too, so supposedly no change here.

• Querying LocationPrivacy (also from the Guardian Project, but not available at F-Droid) is working flawlessly from Guardian Project (!) and ApkTrack Proxy, too.

• In contrast to that, querying Orfox (from the Guardian Project, too; also not available at F-Droid) always fails, using the Guardian Project or ApkTrack Proxy.

• Other apps from F-Droid, from which ApkTrack 2.1.2 fails to query update information for, are e.g. CSipSimple and VLC, while querying (literally) hundreds of different apps from F-Droid is constantly working fine.

• Edit: I was unable to detect any failures to query update information from the Play Store, anymore (on first sight, only quickly tested, in contrast to the other sources for app metadata mentioned in this comment).

• Edit 2: The app I2P (which I did not look at before), originally from I2P, but also available at F-Droid, is not detected as originating from the I2P repository!

Hence, @JusticeRage, please reopen this issue, or declare these failures as "won't fix".

[Olf0]

Side note: I wonder, why the release notes for ApkTrack 2.1.2 state, "Removed the Guardian Project, ...", while it is actually working well for most apps from there, which I have tested so far. Anyway, I am glad to see, that querying the Guardian Project is still supported in v2.1.2

Edit: After taking a (my first) look at main/assets/sources.json, I have a faint idea, how querying app metadata works in ApkTrack (also realising I was misled by the term "web scraping"). So it should have said, e.g. "Internally moved Guardian Project to ApkTrack Proxy.", but never mind: I don't think anybody else will stumble across this statement.

P.S.: Thank you for reopening this issue.

[JusticeRage]

I apologize for not following up on this earlier. Are you still facing these issues? I would like to solve this one next if it still need to be.

Regarding the Guardian Project, I reluctantly moved the scraping part to my proxy because of changes on their website that made scraping from the app (in particular, with a single query) quite difficult.

[Olf0]

Hey @JusticeRage, no need to apologise, as:

• This is a rather minor issue.
• I have sometimes experienced longer delays, until developers responded, than two months.
• I already failed to respond (as a bug reporter) within months.
• We are all doing this in our spare time.

I will check this issue the next couple of days (hopefully), knowing that you upgraded ApkTrack's proxy bandwidth lately.

[Olf0]

Retested, still with ApkTrack 2.1.2 from F-Droid: Exactly the same results, as before.

I.E. I still cannot successfully query Orfox (installed from the Guardian Project repository, not available on F-Droid), no matter if ApkTrack proxy or Guardian Project is selected as source for its update information. All other Or* apps are fine (as before), as they are available from F-Droid, hence queryable selecting F-Droid. But also as before, a couple of other apps from the Guardian Project repository, which are not available on F-Droid (e.g. Location privacy) are queryable selecting Guardian Project and also ApkTrack Proxy!

Also the same (still negative) result for CSipSimple (via F-Droid and ApkTrack Proxy, installed from F-Droid) and VLC (F-Droid, although ApkTrack Proxy is working for it, but I attribute this to an oversight when testing the last time. Furthermore, the result from ApkTrack Proxy does not conform to my intention, as I use the VLC version built and signed by F-Droid on this device, but its original release is displayed, as available e.g. on Aptoide).

The only real difference I detected when repeating these tests, is that I2P (mentioned in "Edit 2" above) is now successfully queryable from F-Droid (only, but not ApkTrack Proxy and the I2P repository is not offered as a separate source for querying update information for it). But as I2P was a couple of times not available on F-Droid (it was a "on / off" thing for many weeks, also visible in the F-Droid client app), I attribute this (the former failure and having it working now) to that.

While carrying out these tests another time with the same results, I wondered, if you (or anybody else) can successfully query update information for Orfox and / or the other apps mentioned here?

[JusticeRage]

I've just push some code which takes new modifications to the F-Droid website into account. I have also tested the Guardian Project repository right this instant and was able to upgrade an outdated Orfox package successfully.

[Olf0]

Thanks. Did you also create a new ApkTrack test version to try?

Side note(s):

Looking at sources.json, I wonder why sources of APKs, which provide a well defined XML format (e.g. F-Droid) are queried by webpage scraping, not by parsing their XML data? The webpage format is ever changing, resulting in regularly adapting ApkTrack to these changes (which is tedious). Only for I2P the repository data (in F-Droid's XML format) seems to be parsed directly, but is then filtered by RegExes for specific APKs (which appears to be counterintuitive on first sight, not looking for all APKs offered there). Note, while querying the I2P repository with ApkTrack was working well on multiple tests three weeks ago, it is currently not available (in general, i.e. also in the F-Droid client app) once again.

For other repositories offering well defined XML data for APKs hosted there (actually the same F-Droid format), the ApkTrack Proxy is used (e.g. Guardian Project). As the ApkTrack Proxy consumes quite some bandwidth and storage space (for which you pay), I also wonder why the XML data of these sources is not directly parsed by ApkTrack (hence moving the workload from the ApkTrack Proxy to the clients running ApkTrack)?

[Olf0]

Retested with ApkTrack 2.1.3 from F-Droid: Still the same results, as before and originally with 2.1.2 (which was an enhancement over 2.1.1* used at the start of this issue description).

Please note that I did test ApkTrack's ability to retrieve update information for all APKs from specific sources (e.g. F-Droid, Guardian Project), not the download of APKs (which was noted to be enhanced in 2.1.3).

[Olf0]

Rereading this, it does not make much sense to me:

• Your test was O.K.
• My tests show no difference between v2.1.2 and v2.1.3

@JusticeRage, any ideas how I may proceed with testing to make sure we are testing the same, behaviour and code (as usual, I installed per F-Droid client).

[JusticeRage]

I was thinking about this yesterday too. I was able to upgrade an app from F-Droid with no issue. I think the easiest way to go would be to mention for what app you have issues (let me know the current version, and expected version if you can).