Android Navigate SDK: MapDownloader

[romandanylyk]

Hey, I'm trying to utilize MapDownloader for my app offline mode. During my testing I found some notes that would be worth mentioning, hopefully, you will be able to help me resolve those.

1. Downloading regions.

Most of the time our app will use the whole US area downloaded for offline use - this is quite a lot of data ~8.5 Gb. Downloading it to internal storage takes around 35-40 mins - and this is quite fast, no issues with it.

But once I start using the external storage, like an SD card the download speed drops a lot. I notices that while downloading 51 US states to an SD card, the first 10-15 states downloads with a similar speed as they would on internal storage. But the more states are downloaded to the SD card, the slower download speed gets. I was trying to find out a reason and logged all the download progress.

At some point, once there are multiple states downloaded, and you keep downloading the DownloadRegionsStatusListener might stop producing onProgress callback. With normal conditions onProgress callback is called several times per second, to indicate the progress change. But it might randomly stop for a few minutes and then continues producing the updates. It happens for random states at a random percentage and the more states you have downloaded, the bigger pause you will get. I even had download pauses for 10 minutes and then it just resumes.

Interesting tho, that while onProgress pauses progressing, I don't receive any other callbacks, like onPause​(MapLoaderError error)or onResume() - the connection is stable and the download speed is over 300Mb. It just resumes the download on its own after a few minutes. I thought it might be because of a slow or broken SD card, but I tried it with different ones - the issue is the same.

This issue increases the download speed on external storage drastically. It takes around 2 hours to download the US area, while as I said on internal storage it only takes around 35 mins. Do you have any idea why it is happening with external storage and if there is a way to fix it?

2. Deleting regions.

The other issue happens to me as well on external storage only. Once you have the whole US downloaded and tries to delete some states, say California or Alaska, the DeletedRegionsCallback just won't be called or at least it takes super long to perform the delete. Even while you have only California downloaded and try to delete it, it takes 2-3 minutes to complete - it is more time than to download it.

There should be a better and more optimized way to perform file delete. Deleting all offline maps via clearPersistentMapStorage is working fine tho. It is much faster.

Looking forward to your response and recommendations.

SDK version: heresdk-navigate-android-4.12.2.0.9350

[romandanylyk]

Ah, ok then. Thanks for letting us know about the narrowing of the issue. I'm really looking forward to trying the alpha build.

[peterramsing]

Thanks for the movement on this. End of June's going to be great!

[peterramsing]

@HERE-SDK-Support-Team, any update on the Alpha coming? Thanks.

[HERE-SDK-Support-Team]

We'll check this and get back to you. The ticket is still in progress.

[HERE-SDK-Support-Team]

Ok, so here is the update. We have several major changes currently being deployed. Because these changes are quite major, we plan to do heavily internal testing before we are confident to hand it out. The test phase starts in the next week on the alpha version. In best case you can also get this information from your direct sales contact. This would be the best way to check when and how we can share the build with you. Hope this helps.

Our internal tickets for reference: #14289 and #17619.

[romandanylyk]

@HERE-SDK-Support-Team Thanks for the update, we would love to try it out on the alpha, so we will contact the sales for an early build.

[romandanylyk]

Hi @HERE-SDK-Support-Team , We’ve been testing our app (and offline maps example) with the new nightly HERE SDK and here are some of the results:

Download Speed. First and foremost the download speed. This was the major issue we were reporting and it looks like the issue is gone now. Downloading offline maps does not have any weird pauses anymore, thus taking considerably less time to download than we had before. Here are some comparison numbers before and after the SDK update:

Downloading the whole US area (9.2 Gb) Internal Storage 30 min -> 20 min Internal storage was working fine even before, but still I tested it on the new SDK and it looks like it still works pretty well. 👍🏼

External Storage 7 hours -> 40 min This is a significant improvement. Finally, there are no pauses, so the download process continues as it should until all the states are downloaded. This helped reduce the total download time to only 40 min, which is still more than internal storage, but it is fine since external storage has its own read/write limitations. Pretty satisfied with the result 👍🏼

Updating the whole US area (9.2 Gb) offline maps version 90 -> 93

Per my understanding (correct me if I’m wrong), updating maps is similar to downloading maps, but has an additional phase - removing the old maps once the new ones are downloaded. So every map update starts with downloading maps, once it is complete, it removes the old one.

Internal Storage 25 min Internal Storage is working pretty well here as well. It downloaded new maps super fast and then spent a few minutes (1-2) removing old maps. 👍🏼

External Storage 2h 30 min External Storage was significantly slower this time. Downloading the version of the new map took about 40 min (about the same as it would while doing a regular download) and pretty much the rest of the time it was removing the old files. So downloading the updated maps had a decent speed, but for some reason, it was struggling with removing old files. It took almost 2 hours (!) to just remove old files.

This is something that has to be improved for sure, I suppose it might have a similar issue with pauses as it had while downloading maps.

Offline maps compatibility. We also tried to check the offline map compatibility between the old and new SDK.

So I launched the app with SDK version 4.14.4 downloaded a few US states, updated the SDK to nightly-4.15.2, and launched the app again. Unfortunately, I wasn’t able to download or update offline maps afterward. In fact, while trying to get the downloadable regions, I received a MapLoaderError.MAP_DATA_ERROR or MapLoaderError.RESOURCE_NOT_FOUND So it basically means that offline maps were not compatible and the only way I was able to get downloadable regions with the new SDK is to remove the old ones.

I remember you were mentioning that to fix the download speed you had to make some big changes to the file structure, so I assume it caused the maps incompatibility between the old and new SDK versions. Since we received and nightly build with no release notes, I wasn’t aware of any behavior changes, so maybe you can provide us with some more details on it.

It would be great if you guys can work on increasing the performance of removing files while doing an update, since on the internal storage it is happening super fast.

Ultimately updating the maps should happen with patch updates, so we don’t have to redownload the whole maps over again, and this will eliminate the issue of removing the old files altogether. If you can provide us with some plans for making patch updates that would be highly appreciated.

Thanks for sharing the early build and we are looking forward to providing you with any feedback you might require.

[HERE-SDK-Support-Team]

Hi @romandanylyk, thanks a lot for trying the test build and the detailed feedback! Regarding map updates / patch updates: We already work on a feature that will enable customers to update only the parts of a downloaded region that had been changed. Internally, we call this "incremental updates" and it will be ready hopefully soon.

[HERE-SDK-Support-Team]

PS: Since this is an alpha build, the issues you found regarding offline maps incompatibility are known and will be addressed when we make this change public. We plan to have this ready in Q3 and/or Q4.

[romandanylyk]

Got it, thanks for the answers. Please, keep us posted on how it goes, and let us know if you need any help testing it on our end.

[romandanylyk]

Hey @HERE-SDK-Support-Team, just want to check out how the work on the offline maps is going. Have you been able to address the issues we were encountering and (or) if there is a new alpha build we can check it against?

[HERE-SDK-Support-Team]

@romandanylyk The major bits of this should be included in our next release 4.15.5.0 - if all goes well. It would be great if you can provide feedback on this release. It is scheduled for next week. Thanks.

[romandanylyk]

Thanks, looking forward to the 4.15.5.0. Will definitely let you know how it goes.

[HERE-SDK-Support-Team]

Small correction, the change will come in a later release: Scheduled for end of November.

[peterramsing]

Thanks for the update

[romandanylyk]

Hi, @HERE-SDK-Support-Team just want to check out how the new release of HERE SDK is going. Are you guys still on track with releasing the above improvements to the download/updating offline maps in November?

[HERE-SDK-Support-Team]

Hi @romandanylyk, thanks for the patience. The rough plan is to have this shipped with HERE SDK 4.17.1.0. A few bugs have been identified and so far only one is left (#27417). As of today, the next releases will be 4.16.4.0 next week. Then followed by 4.17.0 and 4.17.1, so December sounds now more realistic. Hope this helps.

[romandanylyk]

Yes, thanks for the update.

[romandanylyk]

@HERE-SDK-Support-Team, Happy New Year! This issue was originally reported on Aug 12, 2022. It's 2024 now, how things are going? As far as I can see the previous estimated release of 4.17.1 hasn't happened. Can you please check it out once again and provide a release date of 4.17.1 if it is still planned to be fixed in this version?

[HERE-SDK-Support-Team]

@romandanylyk Happy New Year! The fix is supposed to be included and the remaining ticket has been already resolved. The scheduled release date for 4.17.1 is January 18th. Hope this helps.

[romandanylyk]

Hey @HERE-SDK-Support-Team, how the release of 4.17.1 is going? Looks like it was moved once again. 😔

[HERE-SDK-Support-Team]

Hi @romandanylyk, we are in the last stage, roll-out is planned for tomorrow!