Android Navigate SDK: MapDownloader

[romandanylyk]

Hey, I'm trying to utilize MapDownloader for my app offline mode. During my testing I found some notes that would be worth mentioning, hopefully, you will be able to help me resolve those.

1. Downloading regions.

Most of the time our app will use the whole US area downloaded for offline use - this is quite a lot of data ~8.5 Gb. Downloading it to internal storage takes around 35-40 mins - and this is quite fast, no issues with it.

But once I start using the external storage, like an SD card the download speed drops a lot. I notices that while downloading 51 US states to an SD card, the first 10-15 states downloads with a similar speed as they would on internal storage. But the more states are downloaded to the SD card, the slower download speed gets. I was trying to find out a reason and logged all the download progress.

At some point, once there are multiple states downloaded, and you keep downloading the DownloadRegionsStatusListener might stop producing onProgress callback. With normal conditions onProgress callback is called several times per second, to indicate the progress change. But it might randomly stop for a few minutes and then continues producing the updates. It happens for random states at a random percentage and the more states you have downloaded, the bigger pause you will get. I even had download pauses for 10 minutes and then it just resumes.

Interesting tho, that while onProgress pauses progressing, I don't receive any other callbacks, like onPause​(MapLoaderError error)or onResume() - the connection is stable and the download speed is over 300Mb. It just resumes the download on its own after a few minutes. I thought it might be because of a slow or broken SD card, but I tried it with different ones - the issue is the same.

This issue increases the download speed on external storage drastically. It takes around 2 hours to download the US area, while as I said on internal storage it only takes around 35 mins. Do you have any idea why it is happening with external storage and if there is a way to fix it?

2. Deleting regions.

The other issue happens to me as well on external storage only. Once you have the whole US downloaded and tries to delete some states, say California or Alaska, the DeletedRegionsCallback just won't be called or at least it takes super long to perform the delete. Even while you have only California downloaded and try to delete it, it takes 2-3 minutes to complete - it is more time than to download it.

There should be a better and more optimized way to perform file delete. Deleting all offline maps via clearPersistentMapStorage is working fine tho. It is much faster.

Looking forward to your response and recommendations.

SDK version: heresdk-navigate-android-4.12.2.0.9350

[romandanylyk]

I find the same issue is true for the HereWeGo app, its download/delete offline maps speed is way slower with external storage.

[HERE-SDK-Support-Team]

Thanks, we have created a ticket on our end (#14289) to investigate this further. Are you sure that you are using the same network conditions for the comparison?

[romandanylyk]

Yeah, pretty sure it is not a network issue. As I said I was only experiencing this issue while using external storage, once I switch to internal storage the download speed is back to normal and has no weird pauses. I've tried downloading maps on both the OfflineMaps sample project and my own fresh project to make sure it reproduces there as well - and it does.

Please let me know if you need any more details - I would be happy to help. If you don't mind, please keep me posted on how the fix is going, or at least if you were able to reproduce it on your end - we are really counting on the fix any time soon!

[romandanylyk]

The issue reported here is listed in HERE mobile SDK 4.12.4 as a known issue - hopefully, the fix is coming soon!

[romandanylyk]

Hey @datasun, do we have any progress on resolving this issue?

[HERE-SDK-Support-Team]

@romandanylyk, it's still an ongoing investigation.

[romandanylyk]

HERE support responded that issue #1 Downloading region is addressed and going to be released in 4.12.7 - which is really good!

But we are also quite interested in the fix of the following issue. @datasun do you have any idea when it could be fixed?

[HERE-SDK-Support-Team]

14606 and #14607 have been triaged, but are not done yet. We cannot provide a clear due date yet.

[HERE-SDK-Support-Team]

We have improved the behavior for 4.12.9.0, especially on Samsung devices download on external SD cards should work now as expected.

[romandanylyk]

Awesome news @datasun, looking forward to trying it out, I will let you know how it goes.

[romandanylyk]

@datasun I've been trying 4.12.9.0 and didn't notice any changes related to offline maps downloading / updating. The download speed on SD card is still slow (similar to previous versions). Checking the release notes of 4.12.9.0 the speed issue is still listed in the known issues.

What's also weird is that under Depreciation changes it specified that MapUpdater performMapUpdate and checkMapUpdate are depricated, and we should now use the updateCatalog and retrieveCatalogsUpdateInfo - but there are no such functions in 4.12.9.0 Do you have any ideas related to it? Probably the fix is in the following release version?

[HERE-SDK-Support-Team]

Yes, it seems we fixed only a part of the problem. The related bug tickets are still open on our side and will be worked on.

Regarding the deprecation changes, it seems something went wrong for the last release, but today we released 4.12.10.0 and the example app shows how to use the new methods.

[romandanylyk]

Thanks for letting me know, will check the example app.

[HERE-SDK-Support-Team]

The remaining issue was marked as resolved for 4.13.1.0 release, ticket for reference: #14607.

[romandanylyk]

Thanks for letting me know @datasun. Will check it out once it is released.

[romandanylyk]

@HERE-SDK-Support-Team This issue was reported to be fixed many times now (4.13.0, 4.13.1, 4.13.2), but still wasn't resolved. Can you please elaborate on it and provide a realistic date for the fix release?

[HERE-SDK-Support-Team]

You are referring to this resolved issue in our release notes:

"Calling sdkNativeEngine.dispose() now always shuts down ongoing MapDownloader requests.". Indeed it was falsely reported as fixed in 4.13.0 - however, it was fixed in 4.13.1.0 - it was not reported again in 4.13.2.

The ticket #14607 is also closed.

If you can still reproduce this bug, please leave a reply. Thank you.

[romandanylyk]

No, I was referring to slow download / update and delete regions speed on external storage like SD cards. This issue is still listed as a known issue and is not fixed yet in the latest release notes of Navigate SDK 4.13.2

This ticket wasn't referring to a sdkNativeEngine.dispose() at all. I think you are confused with the other ticket I creates a while ago. It was referring to the disposal issue. It was fixed and I haven't seen issues with it now. https://github.com/heremaps/here-sdk-examples/issues/170

But what this one is all about - is a really slow speed of downloading, updating, or deleting offline map packages on SD cards. This ticket should be reopened since I can still reproduce it.

[HERE-SDK-Support-Team]

Thanks, @romandanylyk. Indeed the comment from Dec 15, 2022 was mentioning a wrong ticket number.

The ticket for the external SD card issue is #14289, but it was never reported as fixed in our release notes. And it is still open on our side.

The issue was blocked by another issue for a longer time. Unfortunately, we can still give no due date, but we have contacted the team working on the task. If we get more feedback on the details, we'll paste it here. Hope this helps.

[romandanylyk]

@HERE-SDK-Support-Team please let me know how it goes, at least you guys started working on it. My team and I were waiting for it to be fixed like forever now. Any information, even minor might be helpful.

[HERE-SDK-Support-Team]

@romandanylyk Yes, we will provide an update as soon as we see progress on the ticket. Thanks for being patient and sorry for the inconvenience. This seems to be an issue that has many device-specifics, ie. Samsung vs. other vendors, so it seems to be not that easy to find a solution.

[romandanylyk]

Good, let me know if you need any help testing it even on alpha builds. Our team has multiple different tablets to run it against. I found this issue reproducing even without an SD card, but just setting an offline maps path to the public directories, like "Documents" or "Downloads". So the speed is affected as soon as the cache path is outside of the app directory.

[HERE-SDK-Support-Team]

@romandanylyk We would like to narrow down the issues a bit more. At first, can you list the details about the used SD cars? Especially, on newer SD cards with good write/read speed we would not expect these issues. If possible, maybe you can give a measure on the speeds under labor situations, ie. WiFi, would be also helpful, then we can try to compare this to our measures.

And lastly, when you use internal storage, what path have you set and what countries have you downloaded with which measures?

Sorry, that we cannot get a better analysis at this point of time. At least, there have been some internal improvements for the latest releases, but it seems they did not had the desired effect.

[HERE-SDK-Support-Team]

PS: Since this issue was already moved to our customer service team and we can connect directly with you, we can close the issue here. Tickets are in place and this issue does not seem to relate directly to example apps. So, closing for now. We'll share updates directly with you. Hope this helps. If there are any concerns, please leave a reply. Thanks!

[romandanylyk]

Hi, I feel like we might need to consolidate together to resolve the issue. Do you think we might communicate further in slack (heredev)?

[HERE-SDK-Support-Team]

It depends on which channel our customer service prefers. Meanwhile, regarding "reproducing even without an SD card" - can you share the details including Android version and device, measures and any other details? Thanks.

[romandanylyk]

Sure, later today I will try it on my end once again, share with you the sample project and all the details.

[romandanylyk]

Here are some more details about the issue. Also shared with Ramirez, Manuel via email.

Title: Offline Maps - slow download/update speed Platform: Android SDK version: Navigate 4.13.3.0 Tested devices: Samsung Tab Active 3 (Android 12), Google Pixel 6 Pro (Android 13) Download region: United States - ID 21000001 - 9.2 GB Network speed: Stable, download speed 400-500 Mbps Description:

Downloading speed for Region United States (ID 21000001) is reasonably slower while using SDKOptions persistentMapStoragePath outside of the app default path.

The speed is quite good once we don’t specify persistentMapStoragePath and it is stored in the default location - /data/user/0/<APP-PACKAGE-NAME>/ Download speed: Google Pixel6Pro - 15min Samsung TabActive3 - 25min

But once we set the path to the public directory “Documents” /storage/emulated/0/Documents/here-maps-cache the download speed is slowed down much. Download speed: Google Pixel6Pro - 50min Samsung TabActive3 - 1hour 30min

While using external storage (SD card) path to the public directory “Documents” /storage/9D6F-12F8/Documents/here-maps-cache the drop-down in speed is even more obvious. 9D6F-12F8 - is the SD card name. Download speed: Google Pixel6Pro - does not support SD cards Samsung TabActive3 - 4hours

More technical details:

Tested on HERE Github sample project OfflineMaps, slightly modified it to download the United States instead of Switz. The speed drop could be viewed in the app logs: CachePrefetchJob - Region prefetch progress: job_id=0, region_id=21013672, status=[460/755]

Once the download with nondefault persistentMapStoragePath is started, it is running all right for the first couple of minutes, until it reached 20-30%. Then, while observing the logs, you will start noticing that the download process might pause for some time and resumes on its own (Region prefetch progress wasn’t showing any new data, while normally it is firing roughly once per second).

So the download process is pausing on its own for a few minutes and then resumes on its own as well. And the more total download progress you have, the more often pauses are happening and the more time it needs to resume the download. If it hadn't these weird pauses, I think that the total download time might be quite similar to the download time with default persistentMapStoragePath.

Based on my testing, pauses are happening on both internal (public directory) and external (SD card). But on the external storage pauses are happening even more often and it takes more time to resume. Since the issue is reproduced in the same way on both internal and external storage I think they are related (or might even be the same).

Downloading offline maps is not the only issue that has weird pauses. Once I tried updating the maps, it had the same issue - my guess is because the update process is making the same downloading under the hood, but with replacing the old maps once the download is done. So I think once we fix the regular download speed, it will also speed up the update process.

Note:

• be sure that the internet connection is stable and it is not the cause of download pauses

• The SD card I was using is San Disk A1 and San Disk A2 U3 V30 - both of them didn’t have any difference in the download speed, and both of them had download pauses described above.

• download pauses are reproducing on all our devices, so it is not a local issue that is only affecting specific devices

• we want the persistent offline maps to be stored outside of the app directory, so it is not removed by the system once the app is deleted

Attaching the sample project to this message - again, the project is based on HERE examples OfflineMaps. If you have any other questions, please let me know, we will be happy to help and provide any details which might help determine and resolve the issue.

https://drive.google.com/file/d/1SKF2wCg7WdTJyIfnMHU7nCu-ZADa9dKy/view?usp=sharing

[HERE-SDK-Support-Team]

@romandanylyk Thanks for the informative details. We noticed that the internal path "/storage/emulated/0/Documents/here-maps-cache" has "emulated" directory in it and we came across a few articles that mentioned that the path "/storage/emulated/0" is treated as an external storage, so that it is compatible with older apps on devices that don't have a true external storage.

It would be really helpful if you can also share the logs so that we can understand the root cause of the slow downloads.

We have an assumption that the slowness may be caused due to processing of data on our side, but we can only verify when seeing the full logs. It would be great if you can share these either here or directly through customer service channel. Thanks!

[romandanylyk]

@HERE-SDK-Support-Team Are you looking for some specific logs or do you want me to take all the logs when the slow download is happening? Have you been able to reproduce it on your side?

[HERE-SDK-Support-Team]

@romandanylyk The complete logs, via adb logs, and of course, when you have any other logs that you find useful. We could reproduce some previous issues, which have been addressed already, but not yet the new findings. If possible, please run this with Wifi, ie. in-house. And if possible, try also another device, instead of TabActive3.

To summarize: We have wo issues

1) Unexpected pausing / resuming 2) Insanely high download times for device TabActive3 when e.g. public directory path is used (public directory “Documents” /storage/9D6F-12F8/Documents/here-maps-cache).

Thanks a lot for your help!

[romandanylyk]

Adding the logs for TabActive3 using /storage/9D6F-12F8/Documents/here-maps-cache The logs showing the download process for the whole US area (ID 21000001 - 9.2 GB) till it reached ~40% Running with in-house stable Wi-Fi connection (400-500 Mbps)

Look closely at logs: CachePrefetchJob - Region prefetch progress: date and time. Scrolling them you will notice a few minutes pauses, which are basically causing the slow download speed. Down below I will add a few dates and times, when the pauses were happening.

4min pause

6min pause

6min pause

Archive.zip contains logcat txt files:

• adb-logcat-raw.txt (all logs) adb logcat
• adb-logcat-filter.txt (filtered logs by package id) adb logcat --pid=$(adb shell pidof -s com.here.offlinemaps)
• AS-logcat-copy.txt (filtered logs by package:mine in Android Studio)

Archive.zip

What might be worth attention is this error: SDKNativeEngine - Failed to create file to lock cache folder

TabActive3 is the only device which supports the SD cards that I currently have. But since the issue is also reproduced using the /storage/emulated/0/Documents/here-maps-cache - I think you should be able to reproduce it on other devices as well.

Let me know if this helps or you need more details.

[HERE-SDK-Support-Team]

Thanks a lot. We think this will help and we'll get back to you as soon as possible. Thanks!

[HERE-SDK-Support-Team]

PS: Just a naive question, when the device takes 4 hours, does it put itself into screen lock? Or is the screen kept alive (via settings)?

[romandanylyk]

All the test runs were made with the app in the foreground and the screen always on (via settings)

[HERE-SDK-Support-Team]

Perfect, thanks for confirming.

[romandanylyk]

Hey @HERE-SDK-Support-Team, is there any progress on this issue?

[HERE-SDK-Support-Team]

This is still being evaluated. But it's best to contact customer service directly as they may have more up-to-date information on the ticket. Thanks.

[romandanylyk]

Hey, would like to share some of my new findings, which might be useful for you guys while investigating this issue. Recently SDK 4.13.4 comes out, so I tried it out to check if it has some changes related to increased download speed on external storages - unfortunately not.

But besides that, I discovered an item, which I didn’t mention before: Once another download pause happened to me, I tried to close the app and relaunch it once again to start downloading packages again. In Activity onDestroy I call SDKNativeEngine.getSharedInstance()?.dispose() to make sure every ongoing request are canceled and SDKNativeEngine could be recreated again on the next app launch. But the thing is, if the download process was paused (on its own, as described in this issue), it won’t get canceled even after SDKNativeEngine.dispose()

That will result in an inability to create SDKNativeEngine at the next app launch until the request is unpaused after a few minutes. So, the bottom line is, the downloading process might pause on its own, and nothing can cancel it until it is unpaused. I wonder if those pauses are related to file processing which is being downloaded by this request.

Hopefully, these details will help you to discover this issue.

[HERE-SDK-Support-Team]

@romandanylyk We have one minor update to share. Using "Documents" or "Downloads" directories is discouraged. We will try to make this more clear in the documentation.

These folders are not really suited for storing HERE SDK files. Typically, the "documents" directory is a centralized location for storing user-generated files, making it easy for users to find and access their documents (like PDFs, spreadsheets, etc.). SDK files are only meant to be used by the SDK itself, not by the user, and especially not by other apps.

Secondly, in order to have access to write and read files in a non-app-specific directory, the app needs to have MANAGE_EXTERNAL_STORAGE permission, otherwise, the HERE SDK initialization might fail (when using Documents on SD card for cache path). Not all the apps are allowed to have this permission according to google policy - https://developer.android.com/training/data-storage/manage-all-files#all-files-access-google-play

Therefore, the only desired location is app-specific (https://developer.android.com/training/data-storage/app-specific) dir.

The investigation for the other issues is still ongoing.

[romandanylyk]

@HERE-SDK-Support-Team thanks for letting me know, what are your recommendations on storing offline maps if we would like to keep the data even if the app is uninstalled?

[HERE-SDK-Support-Team]

Keeping offline maps independent of an installed app is not a typical use case. We have no official guidelines on this. However, we have the following statement in our dev guide:

• "Is it possible to pre-install offline maps on a device? Upon request, it is possible to manually create such packages. However, there is no API available for this and you need to contact us to clarify the details."

This sounds close to your use case, but either way, it would require you to get in direct touch with us.

[romandanylyk]

Ok, looks like this is a kinda separate story we will get back to later. The main issue here is a slow download/update speed on external storage even if the offline map path is set to a private app path. The main goal here is to resolve this issue, and then we might get back and discuss how we can save the offline maps if the app gets uninstalled.

[romandanylyk]

@HERE-SDK-Support-Team Hey, I would like to check out how the fix is going. I heard you guys could reproduce it on your side - so that's good. But do you come up with some solution there, so maybe have some timeline on when it might be released? Any updates here would be helpful. We are relying on your fix there. Thanks in advance.

[HERE-SDK-Support-Team]

Hi @romandanylyk, ticket #14289 is still in progress and the problem is on the data side and not easy to fix.

As a workaround for now, we propose to use the external app-specific directory: https://developer.android.com/training/data-storage/app-specific

Do not use the Documents directory. This will reduce reproducibility or even eliminate the issue completely.

[romandanylyk]

@HERE-SDK-Support-Team thanks for letting me know the fix is in progress. Sadly, the app-specific directory on external storage does not make a difference to me. I was trying it before and checked once again on the latest SDK version 4.14.0. So I guess the only option here is just to wait for the fix. Hopefully, it won't get you too long to make the fix.

I appreciate the recommendation to switch to the app-specific storage - once the fix is ready, migrating our existing offline maps to the app-specific storage should not be an issue.

[peterramsing]

Any estimate on when the fix will be? Weeks, months, longer? Thanks!

[HERE-SDK-Support-Team]

We made progress and narrowed down the issue. A possible fix will work only for new app installations, but not for upgrades. Our customer support can provide you with an evaluation build which contains an alpha release end of June 2023. However, this cannot be used for production. The plan for this is roughly end of June 2023. Hope this helps.

[romandanylyk]

@HERE-SDK-Support-Team just to clarify: your plan (rough) is to release an alpha at the end of June 2023 or the alpha is ready now and we can try it on?

[HERE-SDK-Support-Team]

@romandanylyk, the evaluation build is not ready yet, and our rough estimation is end of June 2023.