search

JusticeRage / ApkTrack

ApkTrack is an Android app which checks if updates for installed APKs are available.
GNU General Public License v3.0
123 stars 27 forks source link

ApkTrack does not properly rescan installed apps #107

Closed Olf0 closed 6 years ago

Olf0 commented 7 years ago

ApkTrack rarely automatically rescans local (i.e.installed) apps properly (i.e. does only rarely detect, when an app was upgraded). Manually, I was only able to trigger updating the information about apps from the net.

Scanning all installed apps works fine on ApkTrack's first run (i.e. after installation or after deleting all app data).

But maybe I just did not find the proper way of triggering a local app rescan.

Tested with ApkTrack 2.1.1 under AOSP 4.1.2

Please let me know, if logs etc. would be helpful.

Olf0 commented 7 years ago

Oh, it sometimes works: After updating a couple of apps, ApkTrack shows the recent version number for two of them and still an old version number (the one they had before being updated) for many others.

JusticeRage commented 7 years ago

That one is quite weird and I have no idea about the cause yet. Is the problem fixed if you close and re-open the app? What if you scroll up and down?

Olf0 commented 7 years ago

Closing and reopening ApkTrack, rebooting Android (both multiple times in the last 24 hours) did not help. But it was the only issue (of those I filed yesterday), which does not always fail (i.e. is merely mostly reproducible). For example, I upgraded ConnectBot from v1.9.1 to v1.9.2-oss yesterday (thanks to ApkTrack showing me an upgrade being available, as I accidentally denied the F-Droid client to update it) and ... (see screenshots below). Side note: For those apps, which ApkTrack noticed as being updated, scrolling a bit seemed to trigger this, when ApkTrack was left open while using an apk-store client (e.g. F-Droid or Aptoide) for updating an app. So this is working fine (when it works).

Note, that I am running Android apps in an AOSP 4.1.2 (API level 16, so quite old) based Android runtime environment (AlienDalvik with an "Android kernel") under SailfishOS 2.1.1 (similar, but technically not related to BlackBerryOS 10.3's Android runtime environment). But OTOH, while this is a bit special, the F-Droid and Aptoide client apps (and many others, e.g. Application Info, AIDA64) have never exhibited such (or similar) issues of not noticing updated app versions (and I used many versions of these apps throughout the past 2 years under SailfishOS / AlienDalvik).

screenshot_20170902_001 screenshot_20170902_002

Olf0 commented 7 years ago

As stated in issue #106, it is likely, that those apps ApkTrack did not detect as being updated, were updated while ApkTrack was not running (neither in the foreground or in the background). If ApkTrack 100% relies on permanently running to capture all app upgrades / installations, I believe a possibility to trigger a rescan manually is needed (in case something ever goes wrong or for users, who prefer to have only absolutely necessary apps running permanently in the background). Or alternatively an automatic rescan at each start-up of ApkTrack (the recent test version was pretty fast at that on its first run, but missed a few apps, as already denoted in issue #106).

JusticeRage commented 7 years ago

I agree on the cause analysis. Providing a "rescan app" button would be my preferred solution, but considering the number of open issues and the fact that most devices will let ApkTrack auto-start, I will classify this as low-priority for now!

Olf0 commented 7 years ago

@JusticeRage, O.K., that's reasonable.

For other readers of this issue: As a (tedious) workaround until a "Rescan installed apps" button is implemented, one can delete ApkTrack's "app data" (hence resetting it to its state before its first run) to trigger a rescan of (all) installed apps, when ApkTrack could not capture an app update or installation; but ApkTrack's settings are also lost this way.

anthologist commented 6 years ago

I have the same problem, this is becoming a pain because it makes the app almost useless.

JusticeRage commented 6 years ago

I understand your frustration, but I can only spend so much time working on the app (in addition to my day job and other duties I have). Pull requests are very welcome.

Olf0 commented 6 years ago

I understand your frustration, but I can only spend so much time working on the app (in addition to my day job and other duties I have).

I understand that well (and hope @anthologist does so, too). Look, documenting flaws I perceive, is not for complaining, but aimed at enhancing useful and unique software, as ApkTrack definitely is. Please do not take any bug report or feature suggestion as an offense.

Pull requests are very welcome.

Unfortunately a working pull request is out of reach for me as a former (15 years ago) acceptable Shell and mediocre C programmer. Otherwise I would have tried already, as it is "just" a new menu entry triggering the already existing "scan for installed APKs" function, which is working fine on ApkTrack's "first run". Well, I know even for an experienced Java programmer it is likely a bit more work than that.

Side note: As for @anthologist, I perceive this as the major flaw ApkTrack currently has. On my rather old, slow and memory limited device (plus Android not running natively, but in a container) I only run the apps in the background for which this is necessary to function properly (i.e. only K9 Mail and Conversations). So every other week I delete ApkTrack's app data, start it anew and go through its settings (which is well feasible, but tedious).

JusticeRage commented 6 years ago

Look, documenting flaws I perceive, is not for complaining, but aimed at enhancing useful and unique software, as ApkTrack definitely is. Please do not take any bug report or feature suggestion as an offense.

I didn't take it that way. I apologize if my answer came across as snappy, it was not what I intended. In any case, I've tried implementing the feature that you requested. I generated a test APK. As I cannot test the feature myself (apps don't hibernate on my device), would you be able to try it out and let me know if it solves your problem?

EDIT: Actually, I've just been able to test this on my device and it seems to work! I also notice that since Oreo, background services seem to be missing a lot of messages. I should look into that ASAP.

Olf0 commented 6 years ago

Well, I detected two slight regressions while testing, but basically this enhancement seems to work fine:

  1. Uninstalled ApkTrack 2.1.2 from F-Droid, as the signatures will conflict. Note, that I did not clear ApkTrack's app data.
  2. Installed ApkTrack test version (2017-10-31).
  3. After starting it, the list of installed apps displays every entry twice, see screenshot 1 (in the appendix below).
  4. Triggered a rescan of the installed apps in ApkTrack's settings ("Refresh application list").
  5. Returned to ApkTrack's main screen, while the rescan was running: I could see a third entry for every app popping up.
  6. When scrolling the list (while the rescan was still running) ApkTrack crashed.
  7. Retried steps 4 and 5 in order to see if this behaviour is reproducible: Yes, now four entries for each app are shown (see screenshot 2, below), and it crashes again, even though I waited a little this time (I think the rescan had finished, but I am not absolutely sure, as I missed to check for that in ApkTrack's settings).

Note that you should have received two crash reports from these tests, each annotated with a small note and my e-mail address. I hope this feature (sending crash reports from within an app, but also having the choice not to) is working, as the average user will invest no time in any reporting, if it is not mostly automated.

As something was obviously wrong right from the start, I cleared ApkTrack's app data and tested again:

  1. Now only a single entry for each app is displayed.
  2. Just to check: Scrolling down and up like mad in ApkTrack's main window is working fine.
  3. Triggering a rescan finishes almost immediately: "Detected 0 updated, 0 new and 0 deleted app(s)". Kudos for the well implement user feedback!
  4. Closed ApkTrack and updated a single app.
  5. Started ApkTrack again: It displays the old version of this app as installed (O.K.).
  6. Rescanning again (still finishes almost immediately): "Detected 1 updated, 0 new and 0 deleted app(s)". Perfect!
  7. Now ApkTrack correctly displays the version of the updated app.

Conclusions (by a non programmer):

Appendix:

JusticeRage commented 6 years ago

Thanks for your feedback. I am quite surprised that you could uninstall ApkTrack without deleting its data, as I didn't know that it was possible. The mismatched key issue is expected, as F-Droid signs releases with their own keys.

I have indeed received your two crash reports. Thanks for those, as they contain all the information I need to locate the issues precisely. In this particular case, the bug will be fixed in the next commit. Long story short, the issue is not with the scrolling, but with the fact that the user is leaving the Settings screen while the scan is running. When the scan ends, the function tries to write the results ("X apps updated", etc.) to an object which doesn't exist anymore.

Regarding the duplicate apps issue, I don't understand why this happened just by upgrading the app. If there was a bug in the redetection function that caused apps to be added multiple times it would make sense... But on upgrade? The code at startup that's responsible with this only operates when the database is empty. This might be related to the fact that you could reinstall the app without removing its data? Can you tell me more about how you did this?

Olf0 commented 6 years ago

First of all: Thanks again for your efforts and patience.

I am quite surprised that you could uninstall ApkTrack without deleting its data, as I didn't know that it was possible.

Oh, I was also quite surprised about this 2,5 years ago, when I started using Android apps in the AlienDalvik runtime environment. But I always assumed the fact, that app settings are retained, when uninstalling and subsequently reinstalling an app (e.g. on a downgrade or when changing to an app version signed with a different key), to be an Android feature, at least in AOSP 4.1.2 (on which the AlienDalvik version I use is based). Actually this is quite convenient, especially when apps have extensive settings (e.g. OSMand). Only very rarely an app is unable to properly use app data written by a newer version of it after a downgrade, then clearing its app data resolves this. But as I seldom downgrade apps (only to check if a newly experienced bug was already present in former versions of it), this is basically a non-issue. Furthermore, going from ApkTrack 2.1.2 from F-Droid to the ApkTrack test version (2017-10-31) was rather a "side-grade", as they both exhibit the version number 2.1.2.

This might be related to the fact that you could reinstall the app without removing its data? Can you tell me more about how you did this?

All at the GUI, no special measures taken:

  1. Trigger a "Uninstall app action", either in the F-Droid client ("Uninstall" button in the app specific page; or the same in the Aptoide client) or by a long-press on the app icon on the desktop ("launcher") followed by tapping on the "X" symbol, which then appears overlaid on the app icon (as this is on the SailfishOS desktop, the details of triggering an uninstall action this way are presumably specific to SailfishOS).
  2. Confirm the "Uninstall Android app" dialogue; I believe this is the standard dialogue from AOSP 4.1.2 (is my belief correct?). See screenshot 3 for an example.
  3. Trigger the installation of the downloaded APK in a file-manager.
  4. Confirm the "Install Android app" dialogue; I believe this is the standard dialogue from AOSP 4.1.2 (is my belief correct?). See screenshot 4 for an example.
  5. After I experienced the duplicated entries in ApkTrack's app list, I cleared its app data at the GUI, which is a special feature of SailfishOS IIRC, as the available actions displayed in screenshot 5b are performed from outside of the Android runtime environment (AlienDalvik). I also added screenshot 5a to show the complete Android app specific settings of SailfishOS (also displaying that Android apps have to be explicitly enabled to start their background services on start-up of the Android runtime environment, regardless if they requested the "run at start-up" right upon their installation).

Well, I was quickly trying to analyse what is going on in ApkTrack's app data, when its test version (2017-10-31) displays duplicate entries, but forgot where the app data is located. IIRC (and very faintly) it was a sqlite database somewhere; if you provide me with the original path to the app data under Android 4.1.2, I am sure that I will find it and take a closer look at it.

Screenshots:

Olf0 commented 6 years ago

BTW, as far as I understand, not granting Android apps the rights they request upon installation has been a feature of CyanogenMod / (now) LineageOS for long and was introduced into stock Android with version 7, enabling the user to deny most (or any) requested rights, not just "run at start-up" (as in SailfishOS with AlienDalvik). This is one of the reasons, why I questioned the assumption, that ApkTrack will always properly run in the background to capture APK upgrades, installations and uninstallations. And you also mentioned, that not all app upgrade, installation and uninstallation intents reach ApkTrack under Android 8, correct?

Olf0 commented 6 years ago

LOL, a similar mishap is exposed (but in the opposite direction), when uninstalling ApkTrack's test version (2017-10-31) and reinstalling ApkTrack 2.1.2 from F-Droid: Only about 1/8 of all installed Android apps were displayed (shock!). Again, clearing its app data resolved this. :)

JusticeRage commented 6 years ago

Thanks for the update. I'll try to look into this "sidegrade" database issue but I'm not sure I will be able to reproduce this on my device. BTW, depending on your device, ApkTrack's data could be located in /data/data/fr.kwiatkowski.ApkTrack/.

Regarding the permission issue, Android Oreo sets new restrictions on background processing and the information apps can receive anyway, so I'll have to rewrite that auto-detection part anyway.

Olf0 commented 6 years ago

[...] ApkTrack's data could be located in /data/data/fr.kwiatkowski.ApkTrack/.

Thanks for confirming the place, where I looked first. Side note: This was a case of "Know the crappy software you use well", as my preferred SailfishOS-native file-manager "FileCase" displayed nothing there and explicitly stated "Empty directory". Thanks to your confirmation, I did this at the shell (why else do I a hold a real GNU/Linux machine in my hands?): "Access denied" was the proper answer, one needs to be root!

So this is how it looks like under SFOS: [root@sailfish fr.kwiatkowski.ApkTrack]# pwd /data/data/fr.kwiatkowski.ApkTrack [root@sailfish fr.kwiatkowski.ApkTrack]# ls -l databases/ shared_prefs/ databases/: total 1352 -rw-rw---- 1 10173 10173 1359872 2017-11-03 00:16 apktrack.db -rw------- 1 10173 10173 21032 2017-11-03 00:16 apktrack.db-journal ` shared_prefs/: total 4 -rw-rw---- 1 10173 10173 248 2017-11-02 04:37 fr.kwiatkowski.ApkTrack_preferences.xml [root@sailfish fr.kwiatkowski.ApkTrack]#` This should be identical to a native Android 4.1.2 installation, right?

I see that the apktrack.db is in SQLite 3 format and I have sqlite3 installed. Should I try to dump its content when duplicate entries are present and diff that with a db-dump when everything is O.K. (i.e. after clearing app data and starting ApkTrack anew)? Or would you prefer to capture catlog output, when rescanning installed apps while ApkTrack is already confused (i.e. while it inserts another duplicate entry for each installed APK)?

JusticeRage commented 6 years ago

Ideally, here is the information I would like to look at:

  1. A "clean" copy of apktrack.db before the sidegrade.
  2. A copy of the same file after the sidegrade, when all the apps have been duplicated.
  3. The logcat output taken just after the sidegrade, after the "new" ApkTrack is launched the second time and has duplicated all the apps.

If that's not possible, I may find what I need with only item 2.

Olf0 commented 6 years ago

1. & 2. is easy, so I will do that tonight: Can you please send me a private email (I submitted my email address to you in the two crashlogs), so I can mail the database files directly to you.

I will try to do 3., too, if I am not running out of time today (otherwise: this weekend).

Olf0 commented 6 years ago

I have indeed received your two crash reports. Thanks for those, as they contain all the information I need to locate the issues precisely. In this particular case, the bug will be fixed in the next commit. Long story short, the issue is not with the scrolling, but with the fact that the user is leaving the Settings screen while the scan is running. When the scan ends, the function tries to write the results ("X apps updated", etc.) to an object which doesn't exist anymore.

Well IMO it would be nice, if this object could persist as long as ApkTrack is running: This would not only resolve the crashes I experienced (if I understand correctly), but also increase usability, as I perceived it as surprising that the "Detected X updated, Y new and Z deleted app(s)" line was gone after leaving and reentering the settings. IMHO this line should be empty after starting ApkTrack and emptied (i.e. immediately deleted) when a rescan of the installed applications is triggered, but otherwise persist.

JusticeRage commented 6 years ago

The bug has been fixed, but I don't think the result should persist as they are only meaningful when they are displayed (the situation may have changed since). Leaving it may leave users with the impression that continuous scanning is taking place.

From: Olf notifications@github.com Sent: Nov 4, 2017 10:45 To: JusticeRage/ApkTrack Cc: Ivan Kwiatkowski; Mention Subject: Re: [JusticeRage/ApkTrack] ApkTrack does not properly rescan installed apps (#107)

I have indeed received your two crash reports. Thanks for those, as they contain all the information I need to locate the issues precisely. In this particular case, the bug will be fixed in the next commit. Long story short, the issue is not with the scrolling, but with the fact that the user is leaving the Settings screen while the scan is running. When the scan ends, the function tries to write the results ("X apps updated", etc.) to an object which doesn't exist anymore.

Well its would be nice, if this object could persist as long as ApkTrack is running: This would not only resolve the crashes I experienced (if I understand correctly), but also increase usability IMHO, as I perceived it as surprising that the "Detected X updated, Y new and Z deleted app(s)" line was gone after leaving and reentering the settings. IMHO this line should be empty after start-up and being emptied right after a rescan of the installed applications is triggered, but otherwise persist.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Olf0 commented 6 years ago

The bug has been fixed,

Sorry, I did not know it was already finished (as there was not any corresponding commit, when I looked at the master and beta branches): Fine (you did fix this impressively quick).

but I don't think the result should persist as they are only meaningful when they are displayed (the situation may have changed since).

Fair enough, as there may be updates of apps happening in the background, while ApkTrack is opened.

Leaving it may leave users with the impression that continuous scanning is taking place.

Thanks for this reply, as it made me see my own point more clearly: Currently (in v2.1.2-test) there is no indication that the scan is running. Usually it is quite quick, but when not, one can easily become impatient and return to the main window (i.e. the application list) to see, if something changed there (as I did, triggering the original bug, which is already fixed); returning to the settings later in order to look for the scan result then shows nothing.

So outputting "Refreshing ..." where the scan result is displayed lateron (the line "Detected X updated, Y new and Z deleted app(s)") while the rescan of installed applications is running may easily resolve the false impression that nothing is happening after pressing "Refresh application list".

JusticeRage commented 6 years ago

Sorry, I did not know it was already finished (as there was not any corresponding commit, when I looked at the master and beta branches): Fine (you did fix this impressively quick).

I haven't pushed it yet, but I'm quite sure it solves the issue. I will do that on Monday.

Thanks for this reply, as it made me see my own point more clearly: Currently (in v2.1.2-test) there is no indication that the scan is running.

While the search is going on, the button's text should display "scanning device..." or something like this. It may be something I implemented right after generating the test APK. I'll generate a new one on Monday as well, so you can confirm.

JusticeRage commented 6 years ago

Okay, here is a new test APK. Could you give it another go and see if the previous crash is resolved on your device? You should also be able to see a message while ApkTrack is scanning for changes. I've also made changes to make the search quicker - hopefully no new bugs were introduced.

EDIT: Wait, the APK worked fine on the emulator, but it doesn't start on my device. This is why I hate Android so much. I'll keep you updated. EDIT 2: Okay, the APK works correctly now! EDIT 3; Jeez, I do complain a lot, don't I?

Olf0 commented 6 years ago

This feature (rescanning installed apps) now works fine, in a quick test: No crash when hitting "Refresh installed apps" and correctly detecting a freshly updated app (tested only a single updated one via the F-Droid client). Thanks a lot!

As far as I understand, it makes no sense trying to reproduce the "duplicated apps listed" issue (which is tedious, as it happened 2 out of 4 tries for me and needs a treinstallation of ApkTrack 2.1.2-fdroid followed by an update to the recent test version each time) anymore, as you are sure that it is fixed, right?

So closing this one.

Cheers!