search

carla-simulator / carla

Open-source simulator for autonomous driving research.
http://carla.org
MIT License
11.22k stars 3.63k forks source link

Post process material for camera #893

Closed eds89 closed 5 years ago

eds89 commented 5 years ago

Hello folks,

I'm new to programming on the Unreal Engine. I want to create a post process material similar to the attached picture. It consists on a black rectangle drawn on the screen in Carla. From what I researched, I have to use some sort of sub_UV function and multiply the resulting pixels by 0. However, I cannot find a function to retrieve a small portion of all UVs. Any input would be very much appreciated. Kinds regards.

black_rectangle .

analog-cbarber commented 5 years ago

Wouldn't it be easier to just apply this to the generated images? Why do you need to do this in Unreal?

eds89 commented 5 years ago

I need to see how disruptive this kind of thing is to trained network models.

On Mon, 22 Oct 2018 at 16:40, Christopher Barber notifications@github.com wrote:

Wouldn't it be easier to just apply this to the generated images? Why do you need to do this in Unreal?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carla-simulator/carla/issues/893#issuecomment-431872594, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUNuSZhQV2rOc7Cm5HzRgSAvgL_vQ5cks5uneb1gaJpZM4XyjWo .

analog-cbarber commented 5 years ago

Sure, but you can still do this to the generated rgb images before passing to your network. There is no need to do it in Unreal.

eds89 commented 5 years ago

I'd like to do it in unreal, as I can train existing networks and later compare how networks trained specifically for this perform.

On Mon, 22 Oct 2018 at 18:17, Christopher Barber notifications@github.com wrote:

Sure, but you can still do this to the generated rgb images before passing to your network. There is no need to do it in Unreal.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carla-simulator/carla/issues/893#issuecomment-431902424, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUNuTT9_IyODckCcJbrobn78bmTgn2Uks5unf2UgaJpZM4XyjWo .

analog-cbarber commented 5 years ago

Once again, there is no reason you cannot do that MUCH more easily outside of Unreal. What would you do if you were working with real video cameras? Would you try to build a special video camera that blocks out part of the image?

I really would not waste your time with making special post-processing materials unless it is something that relies on material properties of the scene or takes significant advantage of the GPU pipeline.

eds89 commented 5 years ago

Ok. I must do it in the complicated way, inside Unreal. If you can't help, that's fine --- I'm hopeful somebody can.

For experienced Unreal users, this should be an easy thing to do. Fingers crossed, someone will reply with useful information. Many thanks.

On Mon, 22 Oct 2018 at 18:50, Christopher Barber notifications@github.com wrote:

Once again, there is no reason you cannot do that MUCH more easily outside of Unreal. What would you do if you were working with real video cameras? Would you try to build a special video camera that blocks out part of the image?

I really would not waste your time with making special post-processing materials unless it is something that relies on material properties of the scene or takes significant advantage of the GPU pipeline.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carla-simulator/carla/issues/893#issuecomment-431913332, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUNufEIRW1zmDwyxjoNTebBtNDuH7Ddks5ungVfgaJpZM4XyjWo .

analog-cbarber commented 5 years ago

I assume you have read the Unreal documentation on Post processing materials: https://docs.unrealengine.com/en-us/Engine/Rendering/PostProcessEffects/PostProcessMaterials

You want to create a post processing material that either passes through the SceneTexture post processing input or mask it out based on the angle between the camera direction and the direction to the pixel being rendered which you can compute from the CameraDirectionVector and the CameraVectorWS nodes.

eds89 commented 5 years ago

Hi. I know about the need for using SceneTexture and, multiplying the range of pixels by 0 (black). However, I cannot come to find a way to select a smaller portion of the whole screen UVs, which would correspond to the rectangle.

I will try the CameraDirectionVector and the CameraVectorWS nodes alternative, though.

On Mon, 22 Oct 2018 at 21:23, Christopher Barber notifications@github.com wrote:

I assume you have read the Unreal documentation on Post processing materials: https://docs.unrealengine.com/en-us/Engine/Rendering/PostProcessEffects/PostProcessMaterials

You want to create a post processing material that either passes through the SceneTexture post processing input or mask it out based on the angle between the camera direction and the direction to the pixel being rendered which you can compute from the CameraDirectionVector and the CameraVectorWS nodes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carla-simulator/carla/issues/893#issuecomment-431969429, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUNuUlInQ8nM_Azu0ZCVElKXllQKmYuks5unilQgaJpZM4XyjWo .

analog-cbarber commented 5 years ago

I think you probably want to use ScreenPosition.

eds89 commented 5 years ago

Using ScreenPosition, to me, needs to break the UV float into its individual components, then compare if these components are within your boundaries. I cannot find a way to do multiple conditional branching in Unreal. For example: If (x >= start_x and x <= end_x and y >= start_y and y <= end_y) then 0, else 1. Zero (black) and 1 (no change) would then be added to a multiply node.

On Tue, 23 Oct 2018 at 00:29, Christopher Barber notifications@github.com wrote:

I think you probably want to use ScreenPosition.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/carla-simulator/carla/issues/893#issuecomment-432027371, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUNuVg8wg940f_sCjpeymGqle7SrTxZks5unlTWgaJpZM4XyjWo .

analog-cbarber commented 5 years ago

Use arithmetic operators in place of logical ones. If you are working with ones and zeros, multiply is equivalent to and.

marcgpuig commented 5 years ago

Also, logical operations are really expensive in shaders, try to avoid them.

eds89 commented 5 years ago

Hi. I want to implement a post process for dirt and dust on the camera. What's the easiest way to do that? Should I add some kind of noise to the trained neural network? I can see the default industry practice is to add gaussian noise when training CNN models. It's assumed this will eventually happen. I however do not know how camera occlusion (e.g. dirt) would be dealt with. There's this paper here: https://arxiv.org/abs/1807.06172. It does not use CARLA though.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.