Render Multiple Cameras at a given Time

[Lunar2kPS]

Hey Unity team! :)

I'd like to render the virtual/game world from multiple camera perspectives at every given time. Using 1 PerceptionCamera works (as the system was designed for), basically by:

rendering 1 camera, then time advances, render 1 camera, then time advances.

What I'd like is... render X cameras, then time advances, render X cameras, then time advances.

However, using multiple PerceptionCameras in the scene causes file sharing violations (overwriting due to async/parallelism), and even when modifying PerceptionCameras to use unique file paths based on the Camera's Unity InstanceID, the saved images are often duplicated and skip over PerceptionCameras.

Support for this would allow my team to capture multiple perspectives of the exact same point in time, and would be extremely helpful for capturing more data about rare states in the game world from multiple camera angles. I don't believe this is supported, but please correct me if I'm wrong! I'd really love this to be implemented, and wondered if you had any thoughts (or potentially a time-frame) on implementng this?

[Lunar2kPS]

NOTE: Setting up multiple RenderTexture assets in the project and assigning unique RenderTextures to each Camera component seems to solve our issues.

Otherwise, it seems they might use the same underlying frame buffer and overwrite each other.

[JonathanHUnity]

Hi there, I'm glad you found a workaround. Much of the Perception package has been designed with multiple cameras in mind, but we still have a few issues to work out. Your approach of using the Camera's instance ID is a good one.

And yes, you will need to assign RenderTextures to each camera, at least making sure that no more than one camera is rendering to the same Display.

No guarantees, but we will try to get official support in for the next release, which should be around the first week of September.

[Lunar2kPS]

Ooh, this is amazing news, thanks so much for the fast & detailed response!

Goodluck with implementing it :).

[Lunar2kPS]

Ahh silly me, just an extra note that you're probably aware of:

When my PerceptionCameras used semantic segmentation, support would need to include a fix for this error:

Hope this helps!

[jamesheatonrdm]

I am also having the same problem. Could you tell me what you edited in PerceptionCamera using the InstanceID?

[Lunar2kPS]

@jamesheatonrdm Ah, I can't remember and no longer have access to the repo where we made the change.

However, if you have the Perception package source, see PerceptionCamera.cs and try to find the file/folder path it does use, and change it there using GetInstanceID().

Apologies for the vague answer!

[jamesheatonrdm]

Sorry I have found it and now include the instance ID in the filename, however when it comes to using the BoundingBoxLabeler, in the JSON files produces the filename does not reflect this change. Was this an issue you also encountered? Thanks for the quick response.

[jamesheatonrdm]

I do now see that the json files include the sensor id with the filename for each camera, so I can use that to determine which camera is producing which images and their bounding boxes.

[Lunar2kPS]

You're welcome! Huh, interesting.. it does sound familiar.

Is it possible to see how it produces this JSON file, to ensure it's referencing the proper file name? 🤔 I unfortunately don't have any specifics though to help. Goodluck though, you're definitely on the right track!

[WeiKin007]

When using using Render Textures to output and write ground truth images from multiple Camera at once, the images produced seem to be very dim compared to outputing to different Displays?

Using render textures

Using displays and no render texture

[SCRktkt]

WeiKin, I'm trying to pin down why this is happening, and am curious about the colorspace you are using. The darkness looks possibly like the difference between Linear and Gamma, however a Render Texture should follow the colorspace of the project. I have a few questions to try to pin this down:

1. What colorspace is your project using? You can find this under Edit > Project Settings > Player > Other Settings
2. How are you creating the Render Textures, in Editor or via code?
3. Do your Render Textures (if you can view them in the Editor) match the colorspace of the project?
4. In creating the Render Texture, have you specified the color format?
5. Do all of your cameras share the same settings for ISO, shutter speed, aperture, etc.?
6. Are you using any Tone Mapping in post processing that would affect a camera using the Display versus a Render Texture?

Sincerely, Adam

[WeiKin007]

• The render texture looks the same as what you have with the color space of the project and texture both being Linear.
• I am creating the Render Textures in the Editor, and only changing the size and nothing else.
• The cameras are all duplicated and have their outputs just changed to fit a new render texture for each camera, so they all have the same settings
• No I don't think I have added any post processing other than whatever is default in the HDRP project settings

[JonathanHUnity]

Thanks everyone for your feedback on this. Multiple simultaneous camera support has been added in the latest 0.9.0 release of the package. Try it out and let us know if you run into more issues.

[harry83017622]

Thanks for your updating new features. It saves my life. However, after import perception library with 0.9.0-preview.2 version. An error says

Library/PackageCache/com.unity.perception@0.9.0-preview.2/Runtime/Randomization/Samplers/ISampler.cs(40,22): error CS0106: The modifier 'public' is not valid for this item

Then, I tried to remove the modifier 'public'. But it recovered after my unity compiled the code. So, I comment all line with modifier in the interface ISampler, then It works. However, I guess this may lose the functionality of the random sampler.

[JonathanHUnity]

Thanks for the report. We are looking into it.

[JonathanHUnity]

The latest package supports only 2020.3.17f1 and beyond. Please update to this version of Unity to fix the compilation error @harry83017622