Memory leak when using external tilesets.

[ZackOfAllTrades]

While working with the Google dataset I've been running out of memory. Looking at the call stack It is clear it has to do with external tilesets and indeed if the content is external it is not unloaded as part of the _unloadCachedTiles update.

In practice even when I set maximumCachedBytes to 0 the _loadedTiles will continue to grow indefinitely

I don't have enough context on how external tilesets are used so I could be way off but it seems like there needs to be some kind of tree-trimming functionality to unload parts of the explicit json tile tree that are no longer in use.

[csciguy8]

Can you provide a particular location you were viewing? Also, how long of a session were you in before it crashed?

Ex. Flying the camera around Manhattan for 10 mins straight.

Just trying to establish an easy scenario, so it can be reproduced quickly.

[ZackOfAllTrades]

Hi thanks for the response, the location does not matter and on my pc I too can almost endlessly zoom around but the memory does increase. I've only crashed on hololens2 where I have very little working memory.

I'm sure it has to do with never evicting the explicit external tileset json files, these can be quite large.

[csciguy8]

Ok great, thanks for the clarification.

On Windows 10, I was also able to reproduce what appears to be a memory leak, but no crash. I watched the Task Manager memory usage for UnrealEditor.exe and could see it grow ~4GB in my tests. It went up and down during this time, but it definitely didn't stay within the Maximum Cached Bytes setting of 268,435,456

This was in cesium-unreal-samples, 12_CesiumGoogleMapsTiles.umap, flying from the Googleplex to NYC and around the dense buildings in Manhattan.

[kring]

Indeed the external tileset data isn't unloaded. Generally that's not a major problem, but in the case of Google Photorealistic 3D Tiles it adds up pretty quickly. A couple of minutes flying around can easily end up with a gigabyte of Tile instances in memory.

It's not easy to unload this data, because we need to make sure it's not being used when it's unloaded. For example, one of the Tile instances in the external tileset could be in the process of being loaded in a worker thread at the moment we decide the root Tile of the external tileset can be unloaded. If we actually went through with the unloading, we'd end up deleting a Tile in one thread while it's in use in another.

But this is certainly solvable, and it's worth solving.

[electrum-bowie]

But this is certainly solvable, and it's worth solving.

if it's worth solving, which i already am fully aware of, please make this bugfix top-priority in your to-do list.

[kring]

Worth solving is not the same as top priority, I'm afraid, but we do plan to do this as soon as we can. If you need it sooner, remember that cesium-native is open source and we welcome pull requests.

[electrum-bowie]

👍 appreciate it, just pointing out how important it is, nothing else.

[xavier-ateo]

We’re the development team behind Spatial Earth for Apple Vision Pro.

This memory leak has been affecting Spatial Earth since the beginning. The app loads Google 3D tilesets through the Cesium Unity Plugin, which we’ve modified to work on Apple Vision Pro.

The impact on us is serious: the app usually crashes after about five minutes of use, making it hard to use and hurting our store ratings, which in turn affects our sales. It also prevents us from adding new features, like a Google Street View integration, because loading large panoramas into memory can cause the app to run out of memory (due to tiles not being deallocated properly).

Since no one on our team is familiar with C++, we can’t fix this ourselves. This has stopped development on the app a few months ago, as we currently don’t have a way to make it stable enough to move forward.

We would greatly appreciate it if you could prioritize this issue, as we’re eager to get back to working on the app.

[electrum-bowie]

Are you actually selling an app that breaks after 5 minutes ?

Because you've made it sound too bad of an issue to be sold to consumers..

[xavier-ateo]

Everyone who sells software using cesium-native in combination with Google 3D tiles is selling software that has the potential to run out memory and crash, due to this issue. It's just a matter of RAM size, operating system constraints and how many tiles a user loads.

Our sofware works well enough for the user to have a peak around the globe, which is all the released version currently offers. If it crashes during that, people just relaunch and keep going. And if it bothers them, they have the option for a refund through the App Store.

However I don't consider this to be a good user experience, and would love to have a stable app, which I can build upon.

[meta-meta]

@xavier-ateo Are you using Unity for the app? I've had some luck mitigating the issue by updating to latest LTS, reducing Cesium's MaxCacheSize and making an effort to be better about memory management in the rest of the app with guidance from Unity's memory profiler. This is on the Meta Quest platform.

[xavier-ateo]

@meta-meta Thank you for your advice! We are using the latest version of Unity LTS (2022.3.46f1), along with the latest versions of the Unity PolySpatial Plugin and Xcode. Our MaxCacheSize is set to 0, but this value doesn't seem to have any measurable impact on our app. We've spent a lot of time analyzing memory characteristics and trying to optimize our part of the code.

As you can see in the screenshot, the allocated memory keeps increasing while the app is running. The spikes occur when "flying" from one location on the globe to another, which involves loading many new tiles.

Since we're duplicating all textures (a necessary workaround to get Cesium running on Apple Vision Pro), you can see that memory momentarily decreases after a peak. This decrease happens when the duplicate textures are destroyed, releasing their allocated memory. However, the memory allocated by cesium-native seemingly never gets released, as seen by the total memory usage on the screenshot, which never decreases.

Like iOS and iPadOS, visionOS sets memory limits for each app. For visionOS, I believe the limit is around 5 GB. Our app crashes once it reaches this threshold. (The screenshot shows the app running in the visionOS Simulator, which displays the memory of my computer, which is much higher than the limit on the Vision Pro.)

[kring]

@xavier-ateo it's true, as described in this issue, that cesium-native doesn't unload external tilesets. External tilesets are quite a small part of the overall memory usage of a Cesium for Unity app, though, so I'm surprised to hear this would run your device out of memory after 5 minutes.

More likely, I think, is that there's some other memory leak. The trouble is, you're using a custom build of Cesium for Unity, and you haven't contributed those customizations back to us. That makes it nearly impossible for us to determine whether its the customizations that are causing the problem, or something else in Cesium for Unity, or if it is in fact the external tilesets building up.

We do plan to resolve this external tileset issue soon, in any case.

[xavier-ateo]

@kring Thank for your comments, that's actually very insightful!

I wrongfully assumed that the "external tileset data" included references to meshes and textures. Based on your comment that's not the case and that data gets unloaded. That also means that the memory leak must be happening elsewhere.

I'll dedicate some time next week, or the week after to create pull request which includes our modififactions to the Cesium Unity plugin. Our changes to your code are mostly lightweight, like adding scripting symbols and changing the accessibility level of certain parts of the C# code. The rest is newly created code which interacts with the plugin and a build post processing script, which modifies the Xcode project.

[electrum-bowie]

External tilesets are quite a small part of the overall memory usage of a Cesium for Unity app, though, so I'm surprised to hear this would run your device out of memory after 5 minutes.

Don't be, on standalone even a few extra megabytes over the usual memory consumption could cause instability, due to other present background tasks or applications running.

[electrum-bowie]

In my case it's not 5 minutes, it can take hours for an Out of Memory crash to occur in my app.

[electrum-bowie]

But it does certainly feel like a tiny memory loss incrementally happens over longer periods of time ( 10+ minutes ) based on debug memory availability.

[electrum-bowie]

In my case I'm only using Google's Photorealistic 3D tiles.