Major problems with “loading data into memory” – long delays, strange data styling effects and even crashes

[jajajaneeneenee]

Vespucci Version

20.1.0 Beta3

Download source

Google Play Store

Device (Manufacturer and Model)

Samsung A33

Android Version

14

Behaviour/Symptoms

I have been experiencing a very annoying issue with very long delays with the message "Loading data into memory" after switching apps or going to and from the home screen for a while (already under Android 13 and Vespucci v19.x, and now under 20.0.x and 20.1.0 Beta3). My first screenshot with the problem is from 2023-08-09. It frequently makes mapping difficult or even impossible when you are on the move. I also tried to use the recent apps list only to go back to Vespucci, but I think (not 100% sure) that this does not help to solve the problem.

Describing and reproducing the problem is not easy, as it does not occur in every case, but very frequently and practically always during a mapping tour - and often it does not stop or I do not know how to stop it (I have tried everything possible). And often it becomes a kind of loop and you can no longer carry on normally. And it always costs a lot of time and energy.

I was hoping that the new fixes in v21.1.0 regarding app launch might resolve this, but unfortunately not. That's why I'm writing this (a little longer …) issue now. Perhaps it's also a special Samsung or a general Android problem? Don't know.

There are several effects of varying severity that, in my opinion, are all related to one another. I will try to describe it (from less severe to worse effects):

• After switching apps and returning to Vespucci (e.g. after taking a photo or checking something in Firefox), it typically takes over 20 seconds until Vespucci is usable again (I had some cases of more than 60 seconds!). And this happens with almost EVERY app switch (e.g. to the photo app to take a photo or to Firefox to look something up on the web) - but not always, which makes it difficult to reproduce. Anyway, this is really a time eater and bad enough when you just want to quickly map something ...
• But this is even worse: if it takes too long for Vespucci to become active again, an Android error message "Vespucci is not responding" appears with two buttons "Close app / Wait" – the typical bad Application Not Responding (ANR) dialog. And if you press "Wait", this message appears again - after exactly 10 seconds each time, I think. You simply can't get rid of this message. The only way: close Vespucci, possibly via the recent app list, then restart it and hope that it doesn't take too long - otherwise the system message appears again immediately (and that happens quite often – I am often in a long loop until Vespucci finally starts again without a problem). If you press "Close app", it is possible that current editing/input is lost (understandable).
• In some cases (particularly if the startup process takes a long time - or I don't know under what additional circumstances) the data styling is then switched to “Built-in (minimal)” afterwards. And after switching to the previous data styling (I use custom data styling) everything is displayed correctly, but no label text is shown except for house numbers - an app restart is necessary so that everything is OK with the display of labels again.
• If the loop with the ANR dialog starte , this dialog will appear even if you exit Vespucci normally (press the back button twice, then confirm the "Exit Vespucci" dialog)! Very strange for me …
• In some cases (rather rarely) the “Loading data into memory” process does not end at all and Vespucci crashes.
• In particularly bad cases I get into an endless loop of crashes because even after restarting the app the whole thing happens again. Not even restarting the device helps then. At some point (when the network reception improves perhaps?) it will work again.

Example sequence 1 (without “worst case”, only a long delay up to 20 s, no ANR dialog):

• App switching and back to Vespucci
• “Loading data into memory” – approx. 10 s.
• Black screen – approx. 5 s.
• “Loading presets” – approx. 5 s.

Example sequence 2 (with ANR dialog):

• App switching and back to Vespucci or first start of Vespucci
• “Loading data into memory” / Black screen / “Loading presets” takes more than approx. 20 seconds
• Android ANR dialog appears – and no chance to get rid of it

Example sequence 3 (with data styling reset and delay > 60 s):

• A node (amenity=fast_food) called up in the editor for editing
• Tags entered (contact:phone + contact:website), then the URL marked and copied
• Website opened in Firefox to check something
• Switch back to Vespucci
• Black screen approx. 7–8 s., then “Loading data into memory” — doesn’t stop!
• Terminated after more than 60 seconds by tapping next to it (not always possible I think)
• It's still in the property editor, but previously entered values ​​for contact:phone and contact:website are gone (this problem seems to be gone with the new experimental setting Use "new task" mode for property editor – I have not been able to check it extensively yet, in connection with this problem).
• Both tags re-entered and the editing process completed with OK.
• Map is now displayed with DataStyling "Built-in (minimal)"! See screenshots below.
• After switching to the previous (custom) DataStyling, lines/fills are displayed correctly, but no label texts on nodes, e.g. name for amenity=fast_food node.
• You have to exit Vespucci and restart it so that the label texts are displayed correctly again.

Just some suspicions/thoughts:

• It seems as if an Android time limit is exceeded, I often had the impression that it becomes critical after more than 20 seconds. Or can this "Loading data into memory" process (plus "Loading presets") be shortened somehow? Or at least avoid an ANR state during that process?
• Maybe it would help if Android received an active "I am alive" message from Vespucci after certain processes (is this possible?), and especially after starting and loading the presets etc. – to get rid of the ANR dialog, maybe also after it appears the first time. (I'm reminded of Monty Python and "The Life of Brian" - "I am not dead! I am getting better! I don't want to go on the cart! I feel fine!" See https://www.youtube.com/watch?v=Jdf5EXo6I68)
• The problem seems to get worse when there is poor mobile reception (e.g. when I am out in dense forest). Then sometimes I can no longer work with Vespucci after this problem has started. Or I have to press the "Wait" button of the ANR dialog every 10 seconds … a real challenge …
• However, it also happens with the best reception, e.g. WiFi and fast DSL at home (100 MBit/s), but rather rarely and actually never in the worst forms (DataStyling reset or crash).
• It seems to be particularly problematic when something is open for editing when switching apps and the process has not yet been completed with OK (check mark symbol). Perhaps this is better with the new experimental setting Use "new task" mode for property editor – not sure.
• I had the feeling that it helps a little to first switch to the home screen, then to another app and then back to Vespucci (via recent app list), but I'm not sure. I also tried to do the switching only with the recent app list …

For information:

• For me, a normal start of Vespucci without any problems takes at least 12–15 seconds (approx. 7-8 s "Loading presets", then approx. 4-5 s a black screen, then often briefly "Loading data into memory", e.g. 1-3 s).
• I have already made all the good Android settings regarding energy saving and Vespucci.
• Flight mode doesn't help either once the problems have started.
• I have never tried to clear the app cache or to delete the app data (but I thought about it) – for fear of losing my to-dos (not sure if export/import does work without problems), etc. But I have already tried "Clear Data" and "Flush all tiles caches" etc. in Vespucci – without success.
• It doesn't seem to make much difference how large the map section is that you have downloaded.
• The problem may increase if a large number of changes have accumulated after a long time without uploading (but I'm not entirely sure about that either). Sometimes I forget to upload in small chunks and only do it after a few hours of mapping.
• I use some custom preset files and a custom data styling that is quite a bit larger than the normal one (I've already tried to save as much code as possible and can't really reduce it any further). However, if Vespucci starts normally, I have no problems with it. My preset ZIP files are currently 13 MB (unpacked xml 5,3 MB), 4 MB (unpacked xml 1,9 MB) and 100 KB, my datastyling ZIP is 4,4 MB (with a lot of icons) – the unpacked xml has 8,6 MB.

I have these questions too (among others), maybe you could shed some light on them?

• When the message "Loading data into memory" appears, what is actually being loaded? Only the last saved state and all of the active presets, data styling (= local files)? Or is another data transfer over the network required like tiles or OpenStreetMap Notes or something similar (since, presumably, a data transfer seems to take place before Vespucci can be used once more; at the very least, transfer arrows are visible on the icon in the Android header bar).

• And of course: can these issues be solved? Would really be great!!!

How to recreate

See notes above. I haven't found a reliable way to reproduce it yet, all I can say is that it happens to me pretty much every day when I'm mapping.

Some screenshots

Black screen at startup (for some seconds):

Minimal data styling (in rare cases):

Some debug infos:

Crash dump submitted (no or yes + date)

Yes, several times (I don't remember the dates; I have one crash screenshot from 2024-03-26). I remember adding a text like "Problems after loading data into memory" at least once.

[simonpoole]

Literally everything you list as symptom points to you running out of memory (and we've discussed this before). There is no reason to believe that the changes in 20.1 would improve this, actually pre-loading the NSI will likely make it worse.

And as your last screenshot shows, it isn't the amount of OSM data that is causing the issue (but you are using 60% of the total available memory there with no obvious cause).

Things I would suggest trying:

• disabling the NSI
• checking the effect of any GPX tracks loaded
• checking the effect of any additionally loaded presets

further check for any settings/functions on your device that could remove vespucci from memory that are available, in general it should be -difficult- to get the app to completely restart (loading the state in to memory essentially only happens then) and should really only be the case if you manually completely exited it.

Check that you don't have the "Don't keep activities" option turned on in the Android developer options.

I would further note that both state files seem to be excessively large in that last screenshot, resetting both the OSM data and tasks now and then after editing and uploading might help with that.

PS: the 0.5 GB maximum memory is the same on all modern Android devices, so there is nothing special about the memory budget for your device, aka mine is exactly the same.

[simonpoole]

Is this crash dump from ~2 weeks ago from you? Its an A33, and the crash is happening on parsing the preset after changing the configuration.

Kommentar
I changed the datastyling (to colour round nodes) and the activated presets (from my custom presets to the Simonpoole beautified preset) in the preferences, then a long black screen, then the crash. I often have problems especially when changing presets, I often have to restart Vespucci after that.
Entschleierter Stacktrace
java.lang.OutOfMemoryError: Failed to allocate a 24 byte allocation with 5011664 free bytes and 4894KB until OOM, target footprint 536870912, growth limit 536870912; giving up on allocation because <1% of heap free after GC.
    at libcore.util.NativeAllocationRegistry.registerNativeAllocation(NativeAllocationRegistry.java:270)
    at com.android.icu.util.regex.MatcherNative.<init>(MatcherNative.java:51)
    at com.android.icu.util.regex.MatcherNative.create(MatcherNative.java:45)
    at java.util.regex.Matcher.usePattern(Matcher.java:341)
    at java.util.regex.Matcher.<init>(Matcher.java:209)
    at java.util.regex.Pattern.matcher(Pattern.java:989)
    at de.blau.android.util.SearchIndexUtils.removeDiacriticalMarks(SearchIndexUtils.java:144)
    at de.blau.android.util.SearchIndexUtils.deAccent(SearchIndexUtils.java:133)
    at de.blau.android.util.SearchIndexUtils.normalize(SearchIndexUtils.java:59)
    at de.blau.android.presets.Preset.addToSearchIndex(Preset.java:491)
    at de.blau.android.presets.PresetItem.addValueAndDescriptionToSearchIndex(PresetItem.java:279)
    at de.blau.android.presets.PresetItem.buildSearchIndex(PresetItem.java:219)
    at de.blau.android.presets.PresetParser$1.endElement(PresetParser.java:828)
.....

[jajajaneeneenee]

Is this crash dump from ~2 weeks ago from you? Its an A33, and the crash is happening on parsing the preset after changing the configuration.

Yes, it's mine. I only had such a crash like this one time. I think there was no syntax error in my presets or datastyling when it happened (as far as I can remember). And as I wrote, I deactivated all my files to check something. I don't remember any further details. Except that I was able to restart Vespucci after the crash and there were no particular problems (as far as I can remember). Otherwise I would probably have written a GitHub issue.

[simonpoole]

The point is not that it crashed as a result of the particular action, but that it crashed as the result of an out of memory error. Once you are (so) low on memory such an error can happen essentially anywhere but it underlines that the likely cause of your issues is not enough memory/something using it up very fast.

[simonpoole]

Pro memoria: the NSI seems to currently require ~50MB of memory.

[jajajaneeneenee]

Literally everything you list as symptom points to you running out of memory (and we've discussed this before). There is no reason to believe that the changes in 20.1 would improve this, actually pre-loading the NSI will likely make it worse.

And as your last screenshot shows, it isn't the amount of OSM data that is causing the issue (but you are using 60% of the total available memory there with no obvious cause).

Things I would suggest trying:

* disabling the NSI

* checking the effect of any GPX tracks loaded

* checking the effect of any additionally loaded presets

further check for any settings/functions on your device that could remove vespucci from memory that are available, in general it should be -difficult- to get the app to completely restart (loading the state in to memory essentially only happens then) and should really only be the case if you manually completely exited it.

Check that you don't have the "Don't keep activities" option turned on in the Android developer options.

I would further note that both state files seem to be excessively large in that last screenshot, resetting both the OSM data and tasks now and then after editing and uploading might help with that.

PS: the 0.5 GB maximum memory is the same on all modern Android devices, so there is nothing special about the memory budget for your device, aka mine is exactly the same.

Thank you for all the hints ...

NSI:

To understand it correctly: When "Loading data into memory" appears, is it loading or updating the NSI from the network? That would explain why it is so bad when the network reception is poor (e.g. in dense forest). Is it loading/updating anything else from the network (e.g. OSM notes or Osmose warnings)?

I had "Enable name suggestions" activated + now deactivated (I assume that is what the NSI is used for). "Enable name suggestion prese.." was already deactivated. Will this stop pre-loading the NSI on startup?

GPX tracks:

I have one GPX layer with a self-made GPX file which only contains 24 way points – it has 32 KB. (The debug info says: "GPX 6 GPX true content://com.android.providers.downloads.document/msf%3A1000011300" – but the file name of the GPX file is different). I could delete that layer if it is helpful.

In rare cases I used the GPX recording for a short time, but after that I always used "Delete GPX track and its waypoints". Now, there is no track.

Effect of any additionally loaded presets

I can try to check next time, if deactivating all my (3) custom presets and only use your current "Simonpoole last release" preset (or better only the build in preset?). Or what do you have in mind?

I assume it only loads the activated presets during startup, is that correct?

Normally my 3 presets are:

1. Something like a fork of your “Simonpoole Beautified Preset” (with custom changes and additions and especially many “shortcut” items with special default values ​​and also for overriding existing tags – I use Javascript in these items). And I changed most of the icons from PNG to SVG to have a nicer rendering in Vespucci ... (and added some custom icons). This works great by the way (and I hope that rendering the SVGs doesn't eat up a high amount of memory, the icon files are of course greater than PNGs in 48x48 px).
2. A special "lifecycle preset" with many Javascript items to apply lifecycle prefixes with custom changes and also a lot of "Map Icon Display" items to display (custom) lifecycle icons for objects with lifecycle tags (a long object_keys list is required to let them match – matching with REGEX because of the prefixes would drastically reduce this ...). I know, I could also have used datastyling ... but it would also cause a lot of rules in the XML. And with the presets I also have a suitable matching in the properties editor...
3. A special preset for German traffic signs with all German traffic signs as icons (here as PNGs!) and some complex Javascript elements to generate multiple traffic_sign values ​​one by one (with validation rules etc.). I've thought about releasing this a few times, but it's very Vespucci-focused (with greatly reduced functionality for JOSM as well), and there's no real platform for Vespucci presets yet (like JOSM). That's what's kept me from doing it so far.

Settings/functions on your device that could remove Vespucci from memory

As I wrote, I had already checked everything that came to mind, but this is really a bad thing with Android or the manufacturer's own functions for energy saving and so on ... Some Android settings I use:

• Power saving: off (deactivated)
• Put unused apps to sleep: off (deactivated)
• List of never auto sleeping app: this is strange, because I cannot add Vespucci to this list (it does not appear in the list for selecting the apps – do you know why? Because it's a Java app?)
• Vespucci app settings: "Pause app activity if unused": off (deactivated); all permissions (location, notifications, photo/video); no permissions denied; Mobile data: "allow background data usage": ON; "allow data usage while Data saver is on:" ON; battery settings: Unrestricted; storage used: internal shared storage (but it also shows "External Storage": 88 items – what's that?)

Android developer options

I always had "Don't keep activities" option turned off.

But I have to admit that I completely forgot to check the Developer Options. Very stupid ... I had turned them on some time ago (when the problems described were already there) to see if anything could help improve things. I don't remember exactly when I activated it

And stupidly, I had this still activated:

• Always show crash dialog: ON (now off)
• Show background ANRs: ON (now off) => that could of course explain the nasty and very persistent ANR dialog ... I'll check it out ... (although ANR conditions really do seem to occur)
• Strict mode: ENABLED (ON) – now off
• Disable child process restrictions: ON (now off)

I have now completely turned off the Developer Options for further testing.

(Small side note: although I studied computer science in the late 80s/early 90s, I never worked as a developer or programmer as my main job ... but as a project manager, among other things.)

Memory and state file

Running out of memory: I looked at the debug info several times when the problems occurred and I always saw that "Total memory used" was well below the maximum. I have 6 screenshots with the debug info and the highest value is 322817288. 60% means, that 40% are free – can this really be an OOM? I don't understand it completely why this is a low memory state. I think we already wrote about this, I know ...

What would a "normal" amount of used memory?

And what are the potential candidates for using up the rest of the memory in a fast way (if I don't download new OSM data and all presets/datastyling is already loaded)? Could it have anything to do with Javascript items (I use a lot of them, but most of them only with small and simple Javascript code for setting or removing tags – only my traffic_sign Javascript items are larger and more complex; see infos about presets above)? But could they be responsible for memory not being released properly, for example if they are used frequently?

State file: Is this the file which keeps the downloaded OSM data (and the changes)? In one of the 3 screenshots it's 2584077 (2,5 MB?). Did you mean this with "excessively large"?

Bug state file: What's is stored in that file? And 622480 is also "excessively large"?

Sometimes, I perhaps had downloaded a larger area of OSM data (I try to avoid that), but of course I always use "Clear and download current view" after an upload and if starting with a new area. Or should I also use "Clear data" from time to time (I mean does it clear more data)?

Resetting tasks: I am still a little confused because of the wording and the commands in Vespucci.

• Tasks = OSM notes? Or OSM notes plus Osmose? Or OSM notes plus Osmose plus todos? (In the task layer I have OSM Notes, Osmose and Todos activated, MapRoulette deactivated).
• Should I use the menu command "Tasks" - "Clear" – and will this also delete all my (Vespucci) todos? (I mean do I have to save them before the CLEAR command and load them afterwards?). Perhaps these menu items and the wording could be made clearer and more self-explanatory?). I don't dare tap the "Clear" command because I don't know if a dialog box will appear (asking me what to delete or give some information what will be deleted) or whether it happens immediately and without further information ...

Maybe I need a few more days of testing, especially with developer options disabled (and NSI disabled)... And maybe just use your or the internal presets if there are any problems - I don't think there's much I can change/improve to reduce memory usage for example. Then I'll see what happens.

I will write then again. I'm not quite sure if the weather in the next few days will be good enough for a longer mapping tour.

But it would be helpful if you could answer some (or even all?) of my questions in the meantime. Thank you very much!

[jajajaneeneenee]

The point is not that it crashed as a result of the particular action, but that it crashed as the result of an out of memory error. Once you are (so) low on memory such an error can happen essentially anywhere but it underlines that the likely cause of your issues is not enough memory/something using it up very fast.

Ok, I understand this connection.

It certainly fits the situation – it is very likely that I also changed the presets/data styling when there was a problem, to see if it helped. But I can't remember if I had restarted Vespucci BEFOREHAND, for example, and if a lot of OSM data had been loaded and changes had not yet been uploaded, etc. But I'm pretty sure I was busy mapping...

[simonpoole]

Is this crash dump from ~2 weeks ago from you? Its an A33, and the crash is happening on parsing the preset after changing the configuration.

Yes, it's mine. I only had such a crash like this one time. I think there was no syntax error in my presets or datastyling when it happened (as far as I can remember). And as I wrote, I deactivated all my files to check something. I don't remember any further details. Except that I was able to restart Vespucci after the crash and there were no particular problems (as far as I can remember). Otherwise I would probably have written a GitHub issue.

The problem is not just the sum of available memory but if there are blocks of the required size (so very fragmented memory may still lead to an OOM even if you are asking for less than total free memory). But I don't think you are doing anything that "in general" can't work with the memory you have available just that you are triggering something in a way that leads to an issue.

The default preset uses around 20MB and the NSI around 50MB as an indication. Taken as is, that shouldn't be an issue, however presets contain images that need to be converted to Android bitmaps for display (that will increase their size) and I suspect that this might be one of the causes for the issue.

Tasks include all Variants, that is Notes, Maproulette Tasks, OSMOSE, and Todos. Resetting will definitely delete anything held in memory. However if you simply manually run "prune" the Todos will not be affected.

The state files contain a serialized version of the on the one hand OSM data plus undo information and on the other hand the same for tasks. There are further files holding editing state and per layer information, but they are typically not critical. And yes "loading data into memory" is displayed while the state files are being read.

[jajajaneeneenee]

Hello ... thank you. Some more light in the dark.

Just for info:

• If I unzip your vespucci_v5.zip of the „Simonpoole Beautified presets", the folder has approx. 53 MB, mine (with SVGs) has approx. 70 MB. Your vespucci_preset_v5.xml is 1,1 MB, mine is 5,2 MB.

First experiences yesterday (with Developer Options turned off completely + without NSI in Vespucci); I also choosed "prune" in the task layer and tried to use the recent apps list if possible to return to Vespucci when switching apps.

• First everything OK (I was in the city; good mobile connection; 4G)
• After switching to the photo app and back to Vespucci via recent apps list no „Loading presets“ at all, but 16 seconds black screen and 18 seconds „Loading data into memory“ (memory used 252 MB, State file 2,0 MB – with a small area of OSM data loaded). So it's probably already an app restart, I assume. This is really bad ... Android, what are you doing?
• Then everything OK
• I made a screenshot of the Debug Infos.

Then a strange thing (never had this before):

• While the Debug Info window was still shown, I switched to my messenger (Threema) to read a message and checked something in Firefox, then back to Vespucci via recent apps list.
• The Debug Info window was still shown, but when I pressed the arrow left button in the header bar, Vespucci was minimized to the home screen (I have the app there).
• Tried it again with the recent apps list – same effect. Going to Vespucci with the recent apps and leaving the Debug Info window wasn't possible! (I have the experimental option Use "new task" mode for property editor switched on – not sure if this also affects the debug info window).
• Therefore starting Vespucci with the home screen app icon – 10 seconds „Loading presets“, 14 seconds black screen, 1 second „Loading data into memory“ – everything looks OK at first sight
• But then after 2 seconds: the ANR dialog of the system appeared again for the first time!
• I pressed "wait", then quit Vespucci normally, then pressed "Close app" in the ANR dialog when it popped up again. Then also removed Vespucci from the recent apps list.

Then I did a device restart to be sure ... Then also "Close all apps" in the recent apps list.

• First start of Vespucci after that: 12 seconds „Loading presets“, 16 seconds black screen, 1 second „Loading data into memory“ – and at once the ANR dialog again!
• I pressed "Wait"
• I noted this in Google Notes ...
• Going back to Vespucci via the recent apps list – 10 seconds black screen, 2 seconds „Loading data into memory“ – and no ANR dialog anymore!
• Made a screenshot of the Debug Infos ... Memory used 252 MB, State File still 2,0 MB, Bug State File 200 K
• Switched to Google Notes ... back to Vespucci – no ANR dialog
• Then I continued mapping – for approx 3 hours (with pauses) and some uploads, clear data and downloading new (small) areas ...
• And then until the end absolutely no more problems.
• I even went back to Vespucci with tapping on the app icon on the home screen - no problems, only sometimes black screen for approx. 15 seconds and 2 seconds "Loading data into memory" (app is restarted?)
• Debug Infos at the end: Memory used 250 MB, State file 1,2 MB, Bug State File 200 K

Today (at home, Wifi connection):

• No mapping so far ...
• After using "Clear data" and "prune" in the Task layer and OSM data layer + closing Vespucci and restarting it: approx. 7 seconds "Loading presets", approx. 8 seconds black screen and 1 second "Loading data into memory". No ANR dialog.
• Memory used (non-native) is still 292 MB, but the State file is only 7182 (smallest size I ever saw), Bug state file is 48018.

What I basically don't understand: When I am experiencing the problems, in one case the system seems to trigger an app restart and in another case not (doing the same actions and app switching) – perhaps a bad Android behaviour, and sometimes I waited very long until "Loading data into memory" does end, and short after that (no upload, no new download of OSM data, not very much editing – not much can have changed with the State files) it ends very quickly ... So what was it doing during for example 30 seconds of "Loading data into memory" (It must be gigabytes that are loaded ...) or was it only because of a bad internet connection?

(And why did it happen also if I used the flight mode without any connection – didn't Vespucci notice this?)

And a question about the suspected OOM problems: I understood the situation with fragmented memory. What I don't know is what do I have to do to get Vespucci started with non-fragmented / clean memory? (E.g.: can a device restart be necessary? Can I keep the downloaded OSM data or do I have to clean it? – Would be bad if in dense forest – could I save it and reload it afterwards? Would this make a difference for the fragmented memory?) I'm just looking for a way to continue mapping in such a situation and what I could try ... (If I could not upload OSM data or download a smaller area because of a lack of internet connection = worst case).

Last questions:

1. After disabling NSI now, is there any network connection needed which can cause delays if it's bad or missing – when Vespucci is restarted (e.g. when triggered by the system somehow)? I mean when I have already downloaded all the OSM data I need and also the Bing tiles are all there and I would like to do some mapping like in an offline mode, e.g. in dense forest.
2. Does the value of "Native memory usage" matter in any way? (In one screenshot it's 1063516416 – more than 1 GB?)

I'll be watching what happens more closely over the next few days. If there's anything else I should look out for, please let me know.

[simonpoole]

The 1GB native heap is obviously very large, particularly considering that the app doesn't actually explicitly use it, so those allocations are either made by Android or a library we use, but I'm not aware of any that we use that does that. My suspicion is that that might be image/bitmap related. Wild guess: SVG icons?

The other thing that comes to mind wrt general memory usage is data styling, I can't really see how this could amount to any noticeable amount, but all available data styles are parsed on start up and held in memory.

While behaviour of the app is naturally impacted by bad network connectivity, it shouldn't have any bearing on the issues you are seeing.

PS: as a tendency I would increase the on-device tile cache size from the 100MB you currently have configured if you are in areas with bad connectivity.

[jajajaneeneenee]

Hello ... I haven't done much mapping in the last 3 days, so I haven't been able to test much, just a little in a day.

But first I did some more in-depth research into possible Android and Samsung system settings that could play a role.

And I've known for a while that Samsung is known for having a pretty aggressive battery management system, which includes terminating foreground and background processes. It is quite well documented on https://dontkillmyapp.com/samsung, perhaps you know this site (and the app "DontKillMyApp").

I researched all of this a few months ago, but it's really hard for an end user to decide which settings really help with a problem and what they actually do. Anything but nice...

But what I've now found out and changed, which you might also be interested to know (some if this is Samsung specific):

• I had selected "Battery use: "Unrestricted" (instead of "Optimised") for Vespucci in the app settings. Info text for "Unrestricted": "Allow this app to use battery in background without restrictions. This may reduce your battery life." Sounds good, however: As long as "Unrestricted" was set, you could not add Vespucci to the "Never autosleeping apps" list - a Samsung setting - (Battery setting, sub-item "Background usage limits") - because that is probably the case with the "unrestricted" setting anyway. In my first post I asked why you cannot add Vespucci to this list ("I cannot add Vespucci to this list"). I have now changed it to "optimized" anyway and added Vespucci to this list to see if that makes any difference.
• Then I took a closer look at Android's memory settings + came across "RAM plus" (= virtual memory). I didn't even know that Android does something like that and that it is probably switched on by default (mine was set to 4 GB). I have switched it off, even though it should reduce the number of apps open at the same time, I know. But I try to keep them as low as possible when I use Vespucci anyway (I usually only need: Photo app, Google Notes, Weather app, Firefox - certainly the biggest RAM hog -, Locus Mapp Classic – also needs some more RAM – and a plant identification app). I always think that they should fit in the memory at the same time without being stopped/paused or restarted.
• Then I added Vespucci to the "Excluded Apps" list of the memory settings (info text: "Memory usage for these apps won't be checked") - it's hard to find an exact explanation for this, but it's probably mainly supposed to ensure that Vespucci is excluded when you press "Clean now" (button), which stops background apps.
• By the way, the memory settings show at least 1.5 GB free memory for me by default (out of 6 GB) and I now try to use "Clean Memory" more often in the hope of reducing fragmented memory (and of course freeing up a bit more RAM, but that's usually not much more).
• In general, I've read that many problems with excessive app killing on Samsung have increased significantly since Android 11 (maybe that could explain why I didn't have that problem before) and nothing has fundamentally changed until Android 13, just a few settings. On the website dontkillmyapp.com/samsung, for example, it says: "Android 11 Samsung has introduced a new severe (default ON) restriction. Apps can no longer hold wake lock in foreground services. (...) Even disabling the system battery restrictions does not save the app from being killed."

Maybe interesting for you, or you already know that (?):

• In 2023, the problem (of some bad experiences for end users etc.) is said to have been recognized and addressed with Android 14, even in special cooperation between Google and Samsung. I didn't find much information about it, but here for example: https://9to5google.com/2023/05/05/samsung-background-android-14/
• And I have Android 14 + Samsung UI 6.1.
• However, developers have to follow the "new foreground service API policy" for this. Can you say anything about whether this is already the case with Vespucci?

And one of my main problems with everything is that Vespucci seems to be restarting over and over again and this start-up process takes so long or too long, sometimes leading to ANR conditions. If that didn't happen anymore, it would be a big improvement (or even solve all problems?).

First experiences with the changed settings (especially on one day, about 3 hours):

• While I was out and about, there was no ANR status
• Everything worked very well (free RAM was 1.5 GB)
• I only had a timeout during the upload once (very rare...), but 3 minutes later and 50 m further on, it worked. Maybe a bad connection.
• But then, when I was back home (= good WiFi connection), I was only briefly in the weather app and set a timer in the clock, and then switched to Vespucci because I still had to upload a few small edits.
• Vespucci restarted again - with 13 seconds of “Loading Presets”, 13 seconds of black screen + 1 second of “Loading data into memory” - and immediately the ANR dialog was back!
• Then I tried whether “Clean Memory” would help with the RAM settings - unfortunately not.
• The ANR dialog comes again and again; I have to close Vespucci and press "Close app" in the dialog.
• Later in the evening I went outside again, test started Vespucci: everything OK, no ANR dialog (only 6 seconds “Loading Presets” and 6 seconds black screen, 1 second “Loading data into memory” – I don’t understand why, for example, “Loading Presets” sometimes takes 6 and sometimes 15–20 seconds; maybe because the system is still busy with memory management?)
• Next day at home with a bad WiFi connection: zero seconds “Loading Presets”, 15 seconds black screen, 28 (!) seconds “Loading data into memory” – but no ANR! Strange: the state file was > 3 MB, although I had only loaded a very small map section and no edits. After CLEAR DATA I had to exit and restart Vespucci so that the state file was small again. Is it normal that Vespucci has to be restarted for this?

Here are some screenshots when the ANR dialog appeared:

Here is a final screenshot with the state file > 3 MB:

Otherwise, I have now increased the tile cache to 300 MB (OK?) and set "Max. number of download threads" from 1 to 2 (what is the default value?).

And regarding the data styling and SVG icons (which I use in data styling as well as in my presets): I could possibly replace them with PNGs with 48 x 48 px as a test, even if that would be a bit of work.

And the PNGs really do look much worse ... once you have seen the difference! It would be great if you could use JPGs/PNGs in higher (e.g. double) resolution for high-resolution screens, similar to what is possible in web design. And if scaling the pixel graphics would then be necessary: ​​perhaps it is less memory/processing intensive than SVG processing? Although I actually think that the graphics routines for SVG rendering should be very efficient, but you never know.

Off topic: I also really miss an option to specify the icon size for data styling (as is possible for JOSM) or the proportional rendering of SVG graphics - currently non-square graphics are distorted to a square size...

So much for today ...

[simonpoole]

And regarding the data styling and SVG icons (which I use in data styling as well as in my presets): I could possibly replace them with PNGs with 48 x 48 px as a test, even if that would be a bit of work.

And the PNGs really do look much worse ... once you have seen the difference! It would be great if you could use JPGs/PNGs in higher (e.g. double) resolution for high-resolution screens, similar to what is possible in web design. And if scaling the pixel graphics would then be necessary: ​​perhaps it is less memory/processing intensive than SVG processing? Although I actually think that the graphics routines for SVG rendering should be very efficient, but you never know.

Android does not have any native SVG support, so the only thing we do is render the SVG to a bitmap on device. The way things are done now we allocate a bitmap the size of the SVG render to that and then scale that for actual display. This could very well use a lot of memory if the SVG dimension are large. I'm testing a modification that allocates a smaller bitmap that might help with this potential issue.

[jajajaneeneenee]

For your infomation: I haven’t been able to do any real “stress tests” with longer mapping tours in the last few days. That's why I haven't written anything else.

However, I once had the ANR dialog in a loop again (3 days ago, I was out and about). "Clear data" in Vespucci, restarting Vespucci (small State file after that) and (before) "Clear memory" in the Android settings didn't help either.

Around 10 minutes later (at the same location, only around 50 m away) it started working again. And then for the next 3 hours there was no ANR dialog - despite using other apps, a larger map section was loaded etc. I just can't see any regularity as to when it occurs and when it doesn't.

Once in the evening (at home) I had the phenomenon again with the data styling reset to minimal (black lines etc.), but no ANR dialog! But a fairly large map section was loaded (state file 2.9 MB) ... And I had switched back and forth a lot between different apps, so maybe the Android memory management?

And another piece of information: To test whether it has anything to do with the SVGs, I have now replaced the SVGs with PNGs in all my custom presets and the data styling (which took a day's work) ... However, as I said, I have not been able to test it thoroughly yet, but I will report back as soon as I can say more about it.

So far I haven't had any ANR dialog, and the "Loading presets" process seems to be 1-2 seconds faster (usually about 6 seconds). Of course the preset and datastyling (zip) files are also a bit smaller ...

[simonpoole]

So far I haven't had any ANR dialog, and the "Loading presets" process seems to be 1-2 seconds faster (usually about 6 seconds). Of course the preset and datastyling (zip) files are also a bit smaller ...

The rendering of all icons is on demand, so isn't happening while the preset is being read, but for example while the first map is being displayed.

[jajajaneeneenee]

I have some new information (and some new insights).

First because of replacing all SVGs with PNGs (in my custom data styling and the presets):

• This has improved behavior significantly. I no longer had an ANR state when I was on the move, even after switching apps. Everything felt more stable and it was more relaxed...
• However, I did have the ANR dialog at home twice - once the Debug Info window was open in Vespucci before switching apps, and ANR then came up when I switched back to Vespucci. The other time an Element Info window was open and I had clicked on the ID link and was therefore in Firefox. ANR after returning to Vespucci. And only a tiny section of the map was loaded. But neither of these cases is reproducible... However, this happened after I imported another custom data styling version to test something (under a new profile name!), i.e. a new entry in the data style selection list. What might be crucial – for more information, see below...
• The fact that Vespucci restarts very often (Android/Samsung issue?) has of course not changed, i.e. very regular waiting times of at least 20 seconds - which doesn't sound like much, but when it happens all the time it becomes a real test of patience. In this context, I would be interested in an answer to my question in this comment regarding the "new foreground service API policy" (see also https://9to5google.com/2023/05/05/samsung-background-android-14/). So whether Vespucci already fulfills this or whether it would be possible? I have no idea how difficult, clear or unclear this API policy is. It would also be helpful to know (if you know anything from other users) if this is perhaps a Samsung issue in particular... this is my first Samsung so maybe I would try again another manufacturer next... But this can also change with each system/GUI version, which is very unpleasant. But Vespucci is also the only app with which I have such problems or where the usability is significantly worsened.

New insights (and assumptions) regarding multiple custom data stylings:

This might even be/have been the main problem besides the memory requirements of the rendered SVGs (which by the way are all created in the size 580 x580 px, a few maybe in 600 x 600 px, but not larger).

What happened:

• I had created 2 new versions of my custom data styling 2 days ago - one with only PNGs, the other with SVGs (and some PNGs) – for future testing, etc. And I have to say that I have 2 variants of my data styling with almost identical content and the same size, one of which is a "Lit" styling variant with a small additional block, nodes and lines with lit=yes|no|... to display in a clear way (similar to a JOSM styling that is available).
• That means there were a total of 2 x 2 = 4 new data stylings with 4 new profile names (= entries in the data style selection list).
• I wanted to be able to switch back and forth to test what influence the SVGs have on the problems (in a real life situation).
• After that, Vespucci was practically unusable. Total memory used was close to the maximum (511 MB), etc. See screenshot below.
• Constant ANR and crashes...
• And nothing helped (app restarted, device restarted, app cache deleted and so on) – in the end I had to delete all app data (Android app settings), i.e. completely reset Vespucci and set it up again...
• Luckily I was still able to export my TO DOs before + then re-import. Which now works including the comment texts, although I only use one list (Default). I can't remember reading anywhere in the change information for a new version that the issue with missing comment texts in the export files had been fixed - see my comment in #1890 - there was no answer/info about a fix here either. Since then I've always been afraid of losing my TO DOs if I ever have to completely reset Vespucci. Good that this is working now ...

Here is a debug info screenshot (after the import of 4 new custom data stylings):

What I then discovered when reconfiguring Vespucci (v20.1Beta4):

• Total memory used (non-native) on first start: 110 MB
• Total memory used (non-native) directly after importing my first data styling (ZIP, PNGs only, unzipped 12 MB): 154 MB, so + 44 MB
• Total memory used (non-native) directly after adding my first custom preset file (ZIP, PNG only, unzipped 10 MB): 176 MB, so + 22 MB
• Total memory used (non-native) directly after importing my 2nd data styling – LIT styling variant (ZIP, PNGs only, unzipped 12 MB): 211 MB, so + 35 MB
• Total memory used (non-native) after adding 2 more (smaller) custom preset files (PNG only; unzipped 14 MB): 288 MB, so + 77 MB
• Total Memory used (non-native) 8 minutes later (nothing changed): 260 MB
• Total Memory used (non-native) 2 minutes later (nothing changed, but there was a Vespucci restart): 317 MB (!)
• Total Memory used (non-native) 2 minutes later (nothing changed): 269 MB
• Total Memory used (non-native) 14 hours (after some mapping and so on): 208 MB
• What happens to the memory here is a bit of a mystery to me (results of garbage collection / memory management?).

Since then, no major problems (apart from the frequent app restarts). So far, no ANR state, no crash, no minimal data styling. But no very large "stress tests" either...

Main question/assumption after all this:

I now suspect that ALL imported data stylings are loaded into memory when the app starts and are kept there (or when a new one is imported) – at least the styling rules (If the PNGs used in data styling by iconPath="XY" are perhaps also only loaded when needed for the map, like you explained for the display of the icons in the presets?). And not just the currently activated one. I wasn't really aware of that until now, but it would (perhaps) explain a lot.

So the most important question is: is that the case?

It would explain why I had such high memory consumption the whole time and Vespucci became unusable after adding 4 new data stylings, as described above. (Although it is not 100% clear to me why a data styling that is 12 MB unzipped - and only with PNGs - might lead to a memory requirement of 35 or 44 MB (see above), and whether there are opportunities for optimization. But maybe that is not the most important thing.

The big problem is that you can add custom data stylings with "Import data style", but you cannot remove what you have added (there is no "Remove custom data styling" as for presets). I had already noted this in 2022 in #1874 (that a similar interface for data stylings as for Manage Presets would be very desirable), but nothing has changed since then. I will try to create a new issue for it soon, because I think that would be very important. But maybe I am the only Vespucci user who uses custom data styling (or such extensive ones) and who then encounters this problem?

And because that's the case and I hadn't completely reset Vespucci in a long time (reinstall or clear app data), I had a total of 5 custom data stylings for a long time (most with smaller XMLs than my current version, but it would perhaps explain the high memory consumption and OOM situations if the styling rules of all XMLs are kept in memory and these also require significant memory. Plus the SVG problem...)

As long as there is no "Remove custom data styling" function, I will make sure not to import more than my 2 styling variants, i.e. not to assign new profile names. I would of course also like to test with SVGs again as soon as your SVG scaling/rendering change is published (20.1Beta5?), and assume that I can then import an SVG data styling version with the same profile name as the PNG only version. And that I can then change it back again (PNG only) without this resulting in additional RAM usage (perhaps space usage in the internal memory, if SVGs once imported remain there? That wouldn't be a big problem... I don't know if the DataStyling is unzipped when imported and how the handling of graphic files that are no longer used looks like).

Basically, it would be very helpful for me if you could publish a short information about what exactly happens when the app starts and in what order (and what is permanently stored in the RAM or removed again by various commands and functions in the app).

At the moment I can only guess - e.g. because there is no significant delay when you change the data styling (= all stored in memory?), but a clearly noticeable delay when you have changed the preset selection and exit the settings (I suspect that the selected presets are then loaded into memory and deactivated presets are removed from memory, so it doesn't matter whether the list of presets contains many - deactivated - presets, unlike with data styling).

Off topic: By the way, I have just created another issue – #2632 – regarding preset activation/deactivation (or loading a new version), because it would be necessary for the validator to be reinitialized every time the presets are changed. If there are changes in the presets (e.g. if you have made validator check keys optional in preset items), the validator engine will not immediately recognize this; you must first change something in the validator check keys to trigger a re-initialization.

[simonpoole]

Beta 4 contained the changes to SVG rendering https://github.com/MarcusWolschon/osmeditor4android/releases/tag/20.1.0.4 (a 580px X 580 px image corresponds roughly to a 1.3MB bitmap if not scaled down to the target icon size).

In general it isn't possible to definitely document when things are loaded and not, as nearly all of it happens asynchronously to the main UI thread running. More so, garbage collection of objects that have no active references any more is controlled by the JVM, there is no guarantee that memory used by objects that are no longer referenced is immediate (which is likely what you are seeing).

All available data styles are currently held in memory that that shouldn't really be an issue, as the classes should be small, but potentially there might be some references retained in the icon caches when you are switching back and forward which potentially could be improved.

[simonpoole]

Can this issue be closed?

[jajajaneeneenee]

Can this issue be closed?

A good question ... difficult to answer. There are still some things I would like to understand better. And some questions are unanswered.

First of all: the problem has been significantly reduced for me since I stopped using SVGs (instead, PNGs in 48x48px) and only used 2 custom data styling files. I can now map much more relaxed again...

Since then, practically no more ANR states or these "minimal data styling" effects, possibly caused by OOM states.

Problems/open questions that remain include:

1. Are app startup times between 10 and 20 seconds normal? And why are they sometimes only 10 seconds (sometimes a little less), and then 20 seconds again (sometimes more), with the same amount of data downloaded, etc. It is still a test of patience when you are on the move while mapping. I also wonder WHY it takes so long in the first place (adding a new data styling or updating a preset file is much quicker, for example). Also adding all the used ones again and downloading the OSM data would probably be faster. Do I have to buy a high-end cell phone?
2. Could this loading time be reduced somehow? For example, by not loading ALL data styling into memory, but only the selected one? Or by other measures?
3. Do we still have to live with the very frequent app restarts caused by Android? Do you have no influence on this? (My question about the "new foreground service API policy" remains unanswered - see my comment above). Can you say anything about this? For example: This has no meaning in this case. / This is already implemented for Vespucci. / This cannot be implemented for Vespucci. / This would need to be investigated further.
4. You wrote that the data styling rules should not require a lot of memory (I think so too!), but how can it be that after adding 4 more data stylings, the memory used increases from around 200 MB to over 500 MB, and Vespucci is practically no longer usable? (See my comment and screenshot above).
5. I would like to test this again, and also see how it behaves with SVGs, but the effort of having to completely reset Vespucci and reconfigure everything is very high. If my feature request #2631 would be implemented, it would be much easier and would help to close this issue (and maybe provide further insights).
6. At what resolution are the SVGs rendered after your change (also only 48x48px)? I'm asking because I'm interested to know whether SVGs will continue to be displayed better than PNGs.
7. Would it be interesting for you if I did another test with the SVG variants (of data styling and presets)? Even if that could mean resetting the app again. (A backup/restore function for all settings would be great ...)
8. Would it be interesting for you if I sent you my custom data styling (via private email or similar) including SVGs and PNGs? And possibly also my custom presets? I don’t want to publish this (yet).

[simonpoole]

1. Are app startup times between 10 and 20 seconds normal? And why are they sometimes only 10 seconds (sometimes a little less), and then 20 seconds again (sometimes more), with the same amount of data downloaded, etc.  ...

Except for "cold starts" aka the app was previously terminated and removed from memory, switching to the app should be nearly instantaneous. Reading the saved data state is typically only an issue if the system needs to garbage collect while doing so. There is a specific aspect that likely mainly effects you, that the data styles are currently not read asynchronously on start up, so if you have a lot of them that might be an issue (I'm refactoring the relevant code so there might be some improvements there, however the styles need to be available before the data is rendered so that will always a bit of a limit on things).

2. Could this loading time be reduced somehow? For example, by not loading ALL data styling into memory, but only the selected one? Or by other measures?

3. Do we still have to live with the very frequent app restarts caused by Android? Do you have no influence on this? (My question about the "new foreground service API policy" remains unanswered - see [my comment above](https://github.com/MarcusWolschon/osmeditor4android/issues/2614#issuecomment-2231002175)). Can you say anything about this? For example: _This has no meaning in this case. / This is already implemented for Vespucci. / This cannot be implemented for Vespucci. / This would need to be investigated further._

In general the app never restarts for me (same Samsung OS Android version), aka I have to explicitly exit completely to make that happen. Things you can check are for example the background battery usage setting for the app.

The app has always fulfilled any google set policy restrictions etc for services/background use and this is no different, the new policy requires apps to explicitly require a subject matter specific permission for foreground services (instead of an unspecific one) but that is just about being able to run the service in the first place (which just affects GPX recording BTW).

20.2 has an option to enable "hardware" rendering which allows bitmaps etc to be stored in memory outside of the Java VMs heap which may or may not help with your issue, but at least in principle makes more memory available to the app.

4. You wrote that the data styling rules should not require a lot of memory (I think so too!), but how can it be that after adding 4 more data stylings, the memory used increases from around 200 MB to over 500 MB, and Vespucci is practically no longer usable? (See [my comment and screenshot above](https://github.com/MarcusWolschon/osmeditor4android/issues/2614#issuecomment-2253533347)).

Simply adding additional data styles should not use any significant amount of memory (I would suspect perhaps 1kB per feature entry so, ~0.5-1MB per style), switching between them if they have many custom icons will cause more use because of caching of icons etc, till the system re-runs garbage collection.

5. I would like to test this again, and also see how it behaves with SVGs, but the effort of having to completely reset Vespucci and reconfigure everything is very high. If my feature request [Add a function to remove custom data stylings  #2631](https://github.com/MarcusWolschon/osmeditor4android/issues/2631) would be implemented, it would be much easier and would help to close this issue (and maybe provide further insights).

See my comment on the relevant issue.

6. At what resolution are the SVGs rendered after your change (also only 48x48px)? I'm asking because I'm interested to know whether SVGs will continue to be displayed better than PNGs.

The map icons are 20x20 dip iirc. PNG are scaled down to that size, so likely SVG rendering to that size will still be better.

7. Would it be interesting for you if I did another test with the SVG variants (of data styling and presets)? Even if that could mean resetting the app again. (A backup/restore function for all settings would be great ...)

See Add a function to remove custom data stylings #2631

8. Would it be interesting for you if I sent you my custom data styling (via private email or similar) including SVGs and PNGs? And possibly also my custom presets? I don’t want to publish this (yet).

Yes.