Using too many voice recordings

[Serega007RU]

The application does not cope well (it takes a long time to load) if more than 500 audio recordings are stored in the call directory. On my old smartphone, it takes about 30 seconds.

I suggest adding a loading animation meaning that the application loads information about calls. At the beginning, without this animation, I was confused and I thought that the application was not working at all, but after waiting more than 30 seconds, I realized that it had been loading all this time. Also, I suggest making an optimization that loads this information, I think it's not worth reading all existing audio recordings at once.

[nicorac]

Well, I honestly haven't tested such a huge number of recordings 😉. I need to generate a bunch of test files to test it on my side...

Anyway the optimization of recordings directory access already was on my TODO list. The main issue is that BCR-GUI needs access to a custom directory, chosen by the user in BCR. This directory is outside the direct control of the app, because it's outside its private filesystem. It could also be on the external SD!

Starting from Android 10, the only way to let an app access any directory (after the user selects it) is by using Android SAF (Storage Access Framework), which is well-known for its "inefficiency" (at least on some implementations).

Actually the app reads all the filenames in a single shot then, for each of them, it looks for the optional .json file and reads/parses its content. This is done each time the list is shown and, again, is done in a single shot. That's why the application froze...

These are the optimizations I'd like to add:

• make loading asynchronous: the application can immediately work, just after the first records have been loaded; other records could be loaded in background
• consider file contents as immutable: once a file has been parsed (I mean the '.json` metadata files, audio files are read only at play-time), its content could be cached because it won't change in the future; there's no need to re-read it again
• at startup the app must get only filenames (I suppose this should be fast enough), then update the cache when needed
• add a loading spinner to let user wait for load to complete

The latest point is not so hard to add, so I'll add it ASAP 👍

PS: what you mean with 500 recordings? 250 audio + 250 json? or 500 audio + 500 json? or some recordings were created by old BCR versions, so they don't have json files?

[Serega007RU]

All 500+ audio recordings I have accumulated since October 2022 (that is, almost 1 year), of course (except for a couple of recent audio recordings) all of them are without a json file, the function with metadata file appeared quite recently, so I have practically no audio recordings with this file. I want to note that the application did not freeze, I could control it (open menus, settings, etc.)

[nicorac]

All 500+ audio recordings I have accumulated since October 2022 (that is, almost 1 year), of course (except for a couple of recent audio recordings) all of them are without a json file

Ok, this is what I've supposed. And I also suppose you've not changed the default filename format, so BCR-GUI is able to extract some information from it. I'll need to add another feature to allow parsing custom filename formats too...

I want to note that the application did not freeze, I could control it (open menus, settings, etc.)

Yes, you're right, filenames loading it's already asynchronous, so the app does not freeze. But since it misses any kind of load feedback, the user could think something is wrong 👍

[nicorac]

Just released v0.0.4 with deep speed improvements on recordings database.

It's blazing fast on my side. Please check it out.

[Serega007RU]

when updating the list, I get this error

[nicorac]

Please try this test version (still 0.0.4, so maybe you need to uninstall current then install this one)

bcr-gui-0.0.4.1-release.zip

[Serega007RU]

everything works fine, there is no more error

[nicorac]

Just released v0.0.5 with the fix, closing this.

[kunago]

I hope you don't mind commenting on a closed issue, but I suffer from the same issue. I know 500 might sound like a huge number but I managed to gather nearly 2000 such recordings and the app no longer seems to handle this.

Whenever I refresh, it is able to find new recordings, however the refresh as such takes like 10 seconds and any further work with the app is sluggish. I noticed the json files were generated for some older recordings but they don't seem to be created any further.

Is there any way to help you debug this?

[nicorac]

I hope you don't mind commenting on a closed issue

No problem, it's the same issue actually... NOTE: please confirm which version are you using now.

and the app no longer seems to handle this

What you mean with no longer? Was a previous version (which one?) able to handle 2000+ records? Or this same version worked ok with less than 2000, then it became slow when the count grown?

v0.0.13 switched from an "Internal storage DB" to an "Android SAF" based DB, and this could be slower; but DB content is actually kept in memory and saved only on changes (i.e. after a refresh). Sadly Android SAF implementations are really bad, and they could impact both audio files scanning and DB read/write.

This was done to let DB be "accessible" to the user and avoid Android delete it in case of application uninstall/reinstall/data cleanup. Next versions will have features to tag recordings and add notes to them; these data are completelu "user custom", so there's no way to get them back (i.e. from filename or JSON file) if the DB is deleted.

any further work with the app is sluggish

This is something I've already seen, and I'm going to switch from a "monolithic" recordings list to a "virtual" one, where only the visible items are rendered and the ones "out of screen" are loaded on demand. This should deeply improve list rendering performance in case of huge databases.

Another thing I'm going to add is refresh improvements: will try to ask AndroidSAF to only list files created AFTER a given date, to avoid refreshing the same files over and over.

I noticed the json files were generated for some older recordings but they don't seem to be created any further.

Could you please elaborate on this? BCR-GUI only reads JSON files (if existing)...

Is there any way to help you debug this?

Of course, stay tuned... 😉

[kunago]

@nicorac : I tried to take a short video showing it. Any service you prefer for sharing? There are phone numbers seen in the video so I prefer to share in private.

[kunago]

I created an obfuscated version of the video where you can hopefully see the issue.

https://github.com/nicorac/bcr-gui/assets/460971/492f9a44-7bee-4eb2-86bc-79434a3eff0e

[nicorac]

Thanks for the video, I was able to reproduce the bug on my side:

• deleted the database file (to start from scratch)
• created a set of 2500 audio+JSON files
• copied them to BCR folder

app first startup time

First startup of the app requires a huge lot of time, pointing out the first optimization possible: for each file, it searches if it's already present in the DB and, if not, it parses the JSON and adds a record. This operation has an O(n) complexity because increasing records will also increase the search time. It can be deeply improved by filtering the files and keeping only the ones created after the last scan date. This way there's no need to test them for being already present in the DB. This will work only if AndroidSAF can filter (and only return) filenames by their creation date. Should check it...

list refresh once app is started

Refreshing the (huge) list is also slow, because the app actually re-apply the same logic as first startup. It can be optimized the same way as above, by reading (and adding) only the files created after last scan.

DOM optimization

BCR-GUI is built with web technologies (Ionic, Angular), and runs in a "browser" (WebView). As all the other web apps, it became slow when its DOM is crowded by a lot of nodes. This is what happens when the list grows up to 500 items. It can be fixed using a "virtual-scroll" technique: show only a few visible elements and create/delete them when they go in/out of viewport.

so what?

I'm working on all of these optiimizations and I hope I can came up with a fix for the bug... please stay tuned 😉

[nicorac]

Well, it was a huge job but finally there's an update.

SAF improvements

Can't simply get files added after last scan, because deleted files must also be removed from the DB. So the whole list of filenames must be retrieved from the filesystem (through SAF 😔). I've optimized it by only retrieving filenames (no other details) and also optimized Capacitor/Ionic data exchange by sending raw JSON. It's now much faster and less memory hog.

Database improvements

Database is still in-memory and its update algorithm has been improved. Now searching for existing/removed items is much faster, so the update task completes in a shot.

List improvements

Recording list is now "virtual"; it can contain even thousands of records (beware of memory 😉) but only a visible few are rendered. It's actually much faster and responsive.

Finally...

Unzip and test the attached version (still 0.0.14...) and report its bugs here (please do not open new issues for this test version) bcr-gui-0.0.14-virtual-list.apk.zip

[nicorac]

Just released v0.0.15 with improvements to handle big recordings sets (2000+)