Unity freezes when generating datasets with segmentation enabled and large number of objects/image

[calingc]

Hi all, hope you're doing well. We're using unity perception to generate synthetic datasets for retail use cases and we encountered the following issue:

An average image we render has about 1500-2000 objects/image (I'll attach a sample render). Problem is that when we enable segmentations, Unity goes to about 100-150 images after which it freezes. We tested on multiple platforms (osx, linux, windows) and the issue is present on all of them: it renders without any problem, but after 100 or so iterations it completely freezes, and it doesn't even free up the RAM (goes about ~10gb of RAM used)

Not sure which technical details you would need from my side, happy to share as much as I can - The code is closed source though, I'm afraid. We would really like to have segmentations as it has been explicitly requested by our clients.

We're using Unity 2020.3.23f1 with HDRP 10.7.0 and Unity Perception 0.9.0-preview.2.

Cheers!

[shounakgithub]

Hi,

Seems like you're facing some issues wrt scaling your datagen step. We'll need a little more info from you to be able to suggest steps. Please feel free to connect with me at shounak.mitra@unity3d.com and we'll take it from there.

-- Shounak

[mkamalza]

One thing that comes to my mind based on your description of the issue is the number of objects you are instantiating in each frame and your method of memory management. The sample foreground randomizer in the Perception package uses an object cache to avoid creating and destroying gameobjects in each frame. So the first step is to make sure you use some sort of object cache.

Besides implementing an object cache, another quick test would be to try and free memory every few frames to make sure destroyed objects are leaving the memory. Call Resources.UnloadUnusedAssets to do this.

If you already have an object cache and are still facing this issue, it might be that your cache is growing too large (can happen if you have a large variety of objects and a large quantity per frame). To resolve this, try and purge objects from cache using an algorithm like FIFO (first in first out) or LFU (least frequently used)

[calingc]

Hi, Thanks for getting back so quickly to me. After reading your suggestions and playing a bit with the build, I realised the root cause is not the segmentation being enabled (I turned it on/off and the build would still crash), but the memory management itself. Calling Resources.UnloadUnusedAssets every few frames seems to solve the problem, thanks a lot for this tip, I had no idea it existed!

I had a look at the how we do caching and it still has a lot of room for improvement:

• there is an object cache where we load one instance of each object we have in a scene. I realise now that this might not be enough since we have a lot of duplicates for the same assets in every frame
• every iteration we do the following:
  • instantiate(clone) the number of assets we need from the object cache
  • destroy previous assets

Now I suspect this process of instantiating/destroying ~2k game objects every iteration might still get us into trouble. Are there any battle tested patterns unity provides for cache management? Otherwise, the FIFO/LFU purge could works as well.

Many thanks, I guess this issue can be closed (there was no unity perception issue in the first place anyway, just overall poor memory management).