Switch AWSIM from HDRP to URP

[xmfcx]

AWSIM requires very high end hardware to run. AWSIM uses HDRP.

Potential render pipelines:

• URP (Universal Render Pipeline)
• HDRP (High Definition Render Pipeline)

I've made some investigation and realized that HDRP is way too resource intensive. And URP should be more than enough for AWSIM's use cases.

Comparison: https://www.occasoftware.com/blog/unity-understanding-urp-hdrp-built-in

AWSIM graphics don't look good enough to justify the HDRP usage and same level of graphical fidelity could be achieved in URP as well.

• Is there a specific reason to stay with HDRP?
• Is lidar simulation dependent on HDRP?
• What would be the transition costs for switching from HDRP to URP?

The benefits would be a much lighter and faster AWSIM.

Video Comparison: https://www.youtube.com/watch?v=uLQjjMccbOA

I really don't think we need HDRP in AWSIM, URP looks good enough.

cc. @WJaworskiRobotec

[ShrootBuck]

Any idea what the performance gain might be from switching to URP?

[xmfcx]

@ShrootBuck we have made some tests and shared them on the Autoware Discord, will also share them here too.

If you want to see them right now, you can join the Autoware Discord server.

Autoware Discord Invite Link Autoware Discord HDRP vs URP comparison tests messages link

Single PC (5900X & RTX4080) tests:

URP with VSync Frame limiter removed: Before Autoware: 1000+ FPS 😄 With Autoware: 300-400 FPS

HDRP with VSync Frame limiter removed: Before Autoware: 130-140 FPS With Autoware: 12-18 FPS

[Aysenayilmaz]

URP

I converted the AWSIM project and the Nishishinjuku map to URP. I changed the top lidar of the Lexus vehicle in the "AutowareSimulation" scene to Vls128 and also enabled the right and left VLP16 lidars. I disabled the Random Traffic and pedestrians in the scene. Without making any other changes to the scene, I built it.

https://github.com/tier4/AWSIM/assets/108546951/4235684e-56ff-472f-adc4-f424fe301f33

URP with VSync Frame limiter removed: Before Autoware: 1000+ FPS With Autoware: 300-400 FPS

HDRP

In the HDRP version of the Nishishinjuku map, changed the top lidar of the vehicle to Vls128 and enabled the right and left VLP16 lidars. I disabled the Random Traffic and pedestrians in the scene and then built it.

https://github.com/tier4/AWSIM/assets/108546951/3c34dfda-4aa5-408f-bbd4-33e038779d71

HDRP with VSync Frame limiter removed: Before Autoware: 130-140 FPS With Autoware: 12-18 FPS

Both tests were conducted using a single PC. PC specifications are as follows:

• NVIDIA GeForce RTX 4080
• AMD® Ryzen 9 5900x 12-core processor × 24

I directly converted it to URP without making any adjustments in the Nishishinjuku environment. A better appearance can be achieved by adjusting the lights and material settings.I will make some changes and add the developments here.

[ShrootBuck]

Interesting, thanks for sharing your data

[mackierx111]

Thanks for creating the issue and checking out URP potentials. We expect a significant fps improvement and would first like to create a sample project for URP to see if we can simualtion Autoware as before.

Is there a specific reason to stay with HDRP?

URP should not be a problem if proper simulation can be done for Autoware. I think the rendering quality of AWSIM has the most impact on the camera simulation.

Is lidar simulation dependent on HDRP?

As for RobotecGPULidar, they are probably not that dependent on HDRP. @prybicki @msz-rai Do you know?

What would be the transition costs for switching from HDRP to URP?

Probably 3D asset materials

[Aysenayilmaz]

I couldn't fully fix the tree materials in the Nishishinjuku environment, but after adjusting the other materials and lighting settings, I achieved a closer result visually. I'm attaching two photos, one with trees and one without. The one on the left is URP, and the one on the right is HDRP.

[Aysenayilmaz]

After making a couple of adjustments to the Trafficlight script to suit URP materials, the traffic lights started working.

trafficlight_urp.webm

[Aysenayilmaz]

Traffic light detection test

PC specifications NVIDIA GeForce RTX 4080 AMD® Ryzen 9 5900x 12-core processor × 24

Autoware : origin/awsim-stable

The top lidar of the vehicle is VLS128, and the two lidars on the right and left are VLP16.

I have attached the test videos that I tried in two different locations. I added them as YouTube links because the file sizes are large.

Test 1: https://www.youtube.com/watch?v=r28WIJrgRaA ( I removed the videos due to freezes caused by the video recorder. )

Test 2: https://www.youtube.com/watch?v=-s76PCBnOWE ( I removed the videos due to freezes caused by the video recorder. ) cc : @xmfcx @WJaworskiRobotec

[Aysenayilmaz]

I changed my video recorder and re-uploaded the video.

Only Awsim: 600 - 650 fps~ AWSİM + Autoware: 400 - 450 fps~ Awsim + Autoware + obs (screen recorder): 80 - 110 fps~

Test 1 :

https://www.youtube.com/watch?v=PUmss9Qu3BY

Test 2 :

https://www.youtube.com/watch?v=7zp7RloUmDQ

[xmfcx]

@Aysenayilmaz could you propose the changes to the repository in a PR and link it to this issue?

Also could you share the updated Nishishinjuku unitypackage with us and update the documentation with the link?

(Please put it to somewhere safe in our google drive) (Once the file is finalized and tested well enough, I can move it to AWF S3 Bucket too.)

[sasa42]

Hi. I recommend to make both RP possible. There are a row of benefits for HDRP and URP use-cases.

In special the HDRP RT Volume overrides who make it possible to use Raytraced Reflections , RT AO and Raytraced GI. We use them for real Environment on car and car on car reflections. Here a test video to see real environment on car and car on car reflections in action. https://youtu.be/EyVYKxtS6uE?feature=shared and the real thing.... https://github.com/sasa42/project-424-unity/tree/HDRP_Drive_Test This is still 2021.3.15f1. In 2022.latest and 2023.latest it became much faster.

With some manual setup to raytrace only needed layers against others there are still high framerates possible. Check the volume layer setups in the demo.

I would like to constribute a better HDRP quality setup who allows a good scalability on runtime for different use cases.

[xmfcx]

We have switched to URP in AWSIM-Labs so I will close this issue here.

Hi. I recommend to make both RP possible.

@sasa42 it is not possible, we looked into it and it just is not feasible. See:

• https://www.reddit.com/r/Unity3D/comments/11u4cvf/it_is_a_good_idea_to_install_urp_and_hdrp_in_the/
• https://www.reddit.com/r/Unity3D/comments/rry7al/can_i_use_hdrp_and_urp_in_the_same_project/

In the end, for actual AWSIM users (autoware users) who need real time simulator performance, HDRP is useless.

Because Autoware doesn't use camera too much and planning team developers don't need camera at all.

So, visual fidelity doesn't matter. Also HDRP version is waaaaay too slow (We tested 300FPS vs 12FPS on a very high end machine) Not to mention laptop users with low specs.

Also we were able to get reasonable visual fidelity in URP too.

HDRP is only useful for synthetic data generation and that should be separated into a different project.

With some manual setup to raytrace only needed layers against others there are still high framerates possible.

I would like to see your outputs and benchmarks. Apply them, share them and compare against current HDRP.

I would like to constribute a better HDRP quality setup who allows a good scalability on runtime for different use cases.

As long as HDRP is slow and resource intensive as it is in the main branch, it is unusable for Autoware developers. Currently, AWSIM with HDRP is only slightly faster than rendering with Blender :slightly_smiling_face:

[sasa42]

I added an old 2021 test setup (video and repo) to the post before. Q key for auto drive fastest lap WASG for Drive 1 to 6 for cams When you open the project in Unity 2022 , Unity 2023 and Unity 6 latest you also see a steady improvement of framerates.

I found some reasons for the poor performance on actual AWSIM HDRP repo like....

• no HDRP RP Asset Quality setups
• ego car reflection probe setup
• tree game objects
• .....
• no volume layer setups
• culling and optimisations are deactivated because sensors must see them. But you could activate culling and optimisations behind the lidar or radar radius? ... I think i have to show it on an AWSIM fork.)

What would be the target framerate you like to archieve? I could do some quality setups to reach them.

Low Medium Medium RT High High RT

[xmfcx]

I would like to see the performance on Low for this commit version: https://github.com/tier4/AWSIM/commit/df3845eda295a94a66afdd79bf5d29335ee861de of AWSIM main. (This is where AWSIM-Labs had branched off and where the tests for framerates were made.)

See the tests we made in this post: https://github.com/tier4/AWSIM/issues/261#issuecomment-1941680445

I would like to see the best framerate you can get with AWSIM on that commit with Low settings. We could get 1000+ FPS on URP and 140FPS on HDRP. (Without Autoware).

Specs:

• NVIDIA GeForce RTX 4080
• AMD® Ryzen 9 5900x 12-core processor × 24

[sasa42]

Ok.

Can you give me an min and max target framerate for rendering?

In the demo i sent the physics sim runs on 0.002 (i must check it again) because if car sim speeds are over 300 km/h. For rendering i was happy with stabil 60fps. That were the parameters for HighRT on example above. for better understanding: Which would be target framerate you need for AWSIM ? Which framerate range improves the simulation?

[xmfcx]

We are targeting to also run the Autoware on the same computer.

So simulator should use very little resources and VRAM too.

In the end, we will render with 60FPS but we were testing the limits to see how fast it can render.

Since we got 1000FPS with URP, at least 600FPS would be a good target. (Without Autoware)

I would like to see your results with Autoware as well. (not necessary if you don't want to)

[sasa42]

ok . seems to be a challenge. Low 600fps Medium Medium RT High High RT 60fps (looks like photo)

Do you have simulation benefits eg. 60 vs 300 vs 600fps?

[xmfcx]

Please don't misunderstand, end goal is to cap framerate to 60FPS.

But we want to optimize for render time. So, 60FPS can mean 16.66ms per frame (very bad) or 1.66ms per frame and idling until next frame (very good).

So by removing the vsync limiter, we can see how fast it can go and how little time it takes to render a single frame.

This is why I'm asking for 600FPS, we could render a single frame in 1ms to get 1000FPS with URP (without traffic).

So if you achieve 600FPS which means 1.66ms per frame it is acceptable.

I also don't care about High RT (how fast or how it looks, probably will look better than URP) and only care about the Low setting.

[sasa42]

sorry. i answered before i read your last post....

The performace metrics from nextgen lidars like innoviztwo is 10 or 20fps. Cameras around 60fps.

https://innoviz.tech/innoviztwo

Key Performance Metrics 0.05°x0.05° Maximum Angular Resolution (HxV)

10 or 20 FPS Pre Configurable Frame Rate

0.3m-300m Detection Range

120°x43° Maximum Field of View (HxV)

46x137x132 Dimensions (HxWxD)

ASIL B(D) ISO 26262 Compliant

can you give me some insight why framerates around 600fps are needed? I am more the vis guy who is happy with 60fps, photorealism and physics speed who allow 340km/h simulated drive.

[xmfcx]

I've looked into the tests above again and saw:

In the HDRP version of the Nishishinjuku map, changed the top lidar of the vehicle to Vls128 and enabled the right and left VLP16 lidars. I disabled the Random Traffic and pedestrians in the scene and then built it.

So when you make the tests, please switch the top lidar to VLS128 (from the dropdown menu) and also enable the side lidars (they are disabled game objects in the unity scene, just check 2 checkboxes to enable them).

So if you can render each frame in 1.66ms in all these conditions, it is acceptable.

[sasa42]

i will check Low for this commit version: https://github.com/tier4/AWSIM/commit/df3845eda295a94a66afdd79bf5d29335ee861de

I think. Allowing also photoralistic simulated rendering is also very important for the future of awsim.

Let s proof it )

[sasa42]

one more....

After testing the Unity 6 Resident Drawer and GPU occlusion culling it is worth check to parametrize it with max lidar, radar and res from needed sim cameras and main camera. Everything else behind this combined horizon does not need to be rendered. Or is there something more i forgot?

A the moment awsim and awsim labs are rendering the complete scene stack.

[xmfcx]

Please use the Unity version provided in the commit version, not Unity 6.

https://github.com/tier4/AWSIM/issues/261#issuecomment-1941680445

As long as you reproduce this video setup, that's enough. I would like to see all that you can see in this scene. (1 camera only)

[xmfcx]

Also if you are from TIER IV, main reason we have opened this issue is, because: planning or perception or sensing team members in TIER IV are not using AWSIM for anything.

Maybe they are using it for creating datasets (offline task, fps or speed doesn't even matter).

They complain:

• It's slow (✅ fixed with URP in AWSIM-Labs)
• It's heavy (✅ fixed with URP in AWSIM-Labs)
• Can't run it and Autoware on the same PC, can't run on low spec hardware (✅ fixed with URP in AWSIM-Labs)
• Can't add npc's from rviz (⏳ being fixed in AWSIM-Labs)
• Can't reset vehicle position (✅ fixed in AWSIM-Labs)

So, If the actual target audience is not using AWSIM, who is using this? Why are we creating AWSIM?

Ask @TakaHoribe @miursh @takayuki5168 @drwnz if they are using AWSIM for regular Autoware development. And gather their complaints, this is why we scrapped HDRP.