Dynamic shadows

[romfeo]

I am trying to implement a technique I have seen used in ARkit for UE4 in order to achieve dynamic shadows. With this technique you create an unlit material using the passthrough camera and you appy this material to a plane underneath your camera. Then you enable Cast Modulated Shadows on your scene's Directional Light.

My problem is that I haven't been able to setup this material in a way that works as it should.

Example 1 (attached). The passthrough material is not aligned and its scaled differently than blank the space, the real world

Example 2 (attached). With this material setup the alignmed of the shadow plane and the real work is somewhat better, but the shadowing plane does not match in terms of brightness with the real world.

Since shadows play a very important role in creating believable experiences, It would be great if you could share a working method to achieve good, dynamic shadows with AR Core.

My tests are taking place on a Xiaomi Pocophone F1

[bopangzz]

Great question!

First, that GoogleARCorePassthroughCamera node doesn't quite work as expected. You will need to use the texture returned by UGoogleARCoreFrameFunctionLibrary::GetCameraTexture() and use a texture parameter in the material to pass in this texture every frame. Using the GoogleARCorePassthroughCamera node may not guarantee the texture in your material in sync with the camera background.

Second, you need to map the uv correctly from the screen space to the texture space. Basically here is what you need to do: Set up your material like this:

Pass in the CameraTexture and UVOffset to the dynamic material instance like this:

Note that we will include a material and helper blueprint used for this purpose in our next release. So stay tuned.

[romfeo]

WOW! This is more of an answer than I ever hoped! Thank you so much I cant figure out one thing though... What is the "Full Screen UVs" array in the second image?

[romfeo]

And a second issue... It seems that I don't have the Get Camera Texture node... I am on 4.21 binary version of ue4

[romfeo]

Sorry, I've just noticed that the array's elements are on the right of the screen, And that Get Camera Texture is available with google-ar-unreal repo.

[romfeo]

Your solution worked perfectly. My only issue is that there is a brightness difference...

I have mobile hdr enabled. If i disable it, the color difference is not present, but neither are the shadows.

I've ended up adding a SQRT node in my material, just after the Texture Sample, and it seems to correct the problem, but I am not sure if this is the correct approach.

[bopangzz]

Are you using gles3 or gles2? There is a issue in color space when using gles3, which will be fixed in our next release. You workaround should be fine for now. On Wed, Jan 30, 2019 at 9:01 AM Tassos notifications@github.com wrote:

Your solution worked perfectly. My only issue is that there is a brightness difference...

[image: screenshot_2019-01-30-18-22-48-621_com yourcompany a1] https://user-images.githubusercontent.com/2564349/51998357-2a641180-24c1-11e9-9ab5-b0673d8dfcf9.png

I have mobile hdr enabled. If i disable it, the color difference is not present, but neither are the shadows.

I've ended up adding a SQRT node in my material, just after the Texture Sample, and it seems to correct the problem, but I am not sure if this is the correct approach.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/google-ar/arcore-unreal-sdk/issues/49#issuecomment-459023331, or mute the thread https://github.com/notifications/unsubscribe-auth/ASYXB3ubfVRLafK9WBmuzeDLQBLQ0ORLks5vIc_SgaJpZM4aZGHS .

[romfeo]

Hmm, i am not on gles3. I have it disabled. I am using gles2 and still have the issue. My workaround is not perfect either, as there is still some difference in the brightness, depending on the lighting conditions

[bopangzz]

I think you are right, gles2 also have some issue when hdr is enabled. But this should be fixed in the next release.

[wickerman123]

Thank you so much for sharing this!

What's the purpose of the Passthrough camera node in the material editor? Everywhere I look people are tying to use it like Romfeo has here (myself included) and hit the same issues.

Also is there any solution for doing this in the binary version of Unreal? It's not a big issue if there's not - though it's annoying that support for ARCore is very slow in Unreal and it's still using an old version.

[bopangzz]

We probably should remove that Passthrough camera node since it is misleading. The Passthrough camera node gives you the original ARCore camera texture, but since Unreal uses multithreaded rendering, we have to copy the original ARCore camera texture to a buffer texture to make it in sync with the camera pose on game thread. The buffer texture is what get returned by that UGoogleARCoreFrameFunctionLibrary::GetCameraTexture().

There is no way to solve this in the binary version of Unreal(which contains ARCore plugin 1.4) that doesn't involving changing source code. But in theory, you could copy the ARCore plugin code to your project plugin folder to make it a project plugin, in that case, you probably can use the binary version of Unreal and change the plugin source in your project.

[romfeo]

Thank you for giving so much time to help us! So the million dollar question is... When can we expect the next release? :)

[wickerman123]

Really appreciate the help.

I just tried adding the files to a binary UE4 4.21 project and got met with the following message:

"The following modules are missing or built with a different engine version:

• GoogleARCoreBase
• GoogleARCoreRendering"

Upon rebuilding the project fails to compile.

I'd have thought the most up to date version wouldn't have any issues rebuilding the plugin. Do you have any advice? I'm really keen to be able to use the latest version of ARCore in existing projects.

Here's the log: https://pastebin.com/qBZLN4Fb

It appears to be referencing the engine source so naturally it can't find the correct directory. Is this an easy fix?

[bopangzz]

The newer version of GoogleARCore plugin might rely on some engine source code changes, which might not be in the binary version of UE 4.21. I really recommend you to use the source code version of Unreal Engine if you want to use the latest ARCore plugin.

Is there any feature you want in particular in the newer version? If you only need the UGoogleARCoreFrameFunctionLibrary::GetCameraTexture() function with the binary version of Unreal, yo u best bet would be use the source code of the GoogleARCore plugin in here: https://github.com/EpicGames/UnrealEngine/tree/release/Engine/Plugins/Runtime/GoogleARCore And add the function yourself in that version.

[wickerman123]

Accessing the passthrough camera is integral to my project but no other features are really needed.

I'm an artist and have no experience with programming (outside of blueprints,) could you do a quick breakdown of the process for adding this function?

[bopangzz]

It is probably too complicated if you have no experience with programming. Can you start with using that GoogleARCorePassthroughCamera first and see if you see any issues? If you find the camera texture is not in sync, an easier way to fix it is to change the GoogleARCorePassthrough material to also use that node instead.

[wickerman123]

It is probably too complicated if you have no experience with programming. Can you start with using that GoogleARCorePassthroughCamera first and see if you see any issues? If you find the camera texture is not in sync, an easier way to fix it is to change the GoogleARCorePassthrough material to also use that node instead.

Hi, you can see in this image what I had to do to get the passthrough camera to somewhat work in the binary release of UE4: https://i.imgur.com/7TCMFMa.png The resulting effect can be seen in this video: https://www.youtube.com/watch?v=CwJFGYMb0mY

There's a noticeable switch when that passthrough material starts getting drawn. I tried adjusting the scale values to account for aspect ratio and adjusting the gamma (0.45045 rather than 0.22) but to no avail. Also once you see the incorrect aspect ratio it's hard to unsee it.

The result in this GitHub build works a lot better - if a bit trickier to setup and seemingly more expensive - creating a dynamic instance of the PP material and applying it as a blendable to the possessed camera and updating it on tick: https://i.imgur.com/6r41qUP.png https://i.imgur.com/PIQ26GI.png

From what I've seen of ARKit tutorials, it's setup is similar to how we incorrectly assumed the "GoogleARCoreCameraPassthrough" node worked. Is there any plans to make this process simpler or is there a reason I'm overlooking that justifies the method you shared with us?

Again, thank you very much for all the help!

[wickerman123]

Oh and in regards to the Mobile HDR issue in the current built, I was trying to apply gamma correction like I had before and wasn't able to reach a result, until I added in this node network: https://i.imgur.com/h8njAjJ.png - at which point seemed to flip a switch that made my gamma correction start working as desired - even after deleting all the "3PointLevels" nodes and connecting the texture sample output straight to the power node. See the result here: https://i.imgur.com/BzARPlf.jpg

(In the above image, the inside of the frame is the default passthrough that happens regardless and the outside (left side of the image) of the doorframe is the dynamic instance of my PP material. It's not perfect but it's good enough to be barely noticeable.

The left of this image is what the camera texture looked like (with gamma adjustment) before that "switch" that occurred: https://i.imgur.com/lBpxZiv.jpg

[bopangzz]

The reason the camera image aspect ratio looks weird in your first approach is because your uv isn't quite right. The camera texture is in a different aspect ratio and rotation than the screen of the devices. So your uv has to be correct to get the correct camera image rendered.

You should use the second approach to sample the camera texture, but you should be able to replace the dynamic texture(getting from the blueprint function) with the GoogleARCoreCameraPassthrough. Also you don't need to use post processing material to make this method work. It can be any surface unlit material sampling the GoogleARCoreCameraPassthrough/CameraTexture node.

As for the gamma/HDR issue, all you need to do should be simply pow(color, 2) or sqrt(color) to correct the color issue. In the next release, the gamma/HDR issue should be fixed and we will provide a built in occlusion material as well as blueprint function to update the material parameters so it should be easier to setup those type of things.

If you still need some help, it might be easier to share me a simple project with your setup, to reproduce the issue. I can take a close look and point out any potential issues.

[wickerman123]

My first example is in the binary release of UE4 4.20 so there's no way for me to get the camera texture UVs - as per our previous discussion. Correct me if I'm wrong here.

Passing the UVs into the GoogleARCoreCameraPassthrough does work however the corrections you mentioned do not produce ideal results. On the left of this image is the output run though the power node with an exponent of 2 and the right is the output run through the sqrt node: https://i.imgur.com/MtNhlLF.jpg

The only way to get the desired result I previously shared is to have the output be passed through the power node with an exponent of 0.22: https://i.imgur.com/W5oBeel.png

I'm aware that passthrough materials don't need to be done in a post process but the effect I'm producing does require it to be done in the post process - as I don't know how to write custom shaders at an engine level and thus tell shader fragments only to render when the stencil buffer is at a value of 1. The "portal door" effect is commonly produced in Unity but due to the limitations I just mentioned, the only workaround I've thought up is to use the stencil buffer as a mask in a passthrough camera overlay.

[bopangzz]

You can still calculate the correct uv using the node GetPassthroughCameraImageUV and pass in the result uv in the material to modify the screen space uv to sample the GoogleARCoreCameraPassthrough node correctly.

As for the gamma issue, what is the texture looks like if you don't do any modification after sampling the GoogleARCoreCameraPassthrough node? If it looks correct, you don't need to do anything there.

[wickerman123]

You can still calculate the correct uv using the node GetPassthroughCameraImageUV and pass in the result uv in the material to modify the screen space uv to sample the GoogleARCoreCameraPassthrough node correctly.

Ah yes of course - sorry it's 1:45am for me so I'm running on empty! (Will probably be going to sleep very soon)

Without the 0.22 correction the output looks like this: https://i.imgur.com/6FP4zHT.jpg

Though this only happens with "Mobile HDR" enabled - which has to be enabled for this post process material to work. When I had "Mobile HDR" disabled, the post process material would simply not get applied to the camera.

[zyadmardini]

thanks for your help guys , been going on with this for days now , but i still can find the "get camera texture node" working with 4.21 with and the latest sdk (using arcore's helloAR template)

[romfeo]

Hi there,

Just wanted to let you know that I have updated to the new release and the issue in color space when using gles3 or gles2 still remains...

Are you using gles3 or gles2? There is a issue in color space when using gles3, which will be fixed in our next release. You workaround should be fine for now. … On Wed, Jan 30, 2019 at 9:01 AM Tassos @.***> wrote: Your solution worked perfectly. My only issue is that there is a brightness difference... [image: screenshot_2019-01-30-18-22-48-621_com yourcompany a1] https://user-images.githubusercontent.com/2564349/51998357-2a641180-24c1-11e9-9ab5-b0673d8dfcf9.png I have mobile hdr enabled. If i disable it, the color difference is not present, but neither are the shadows. I've ended up adding a SQRT node in my material, just after the Texture Sample, and it seems to correct the problem, but I am not sure if this is the correct approach. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/ASYXB3ubfVRLafK9WBmuzeDLQBLQ0ORLks5vIc_SgaJpZM4aZGHS .

[bopangzz]

Hi romfeo,

Can you attach a screen shot explain the problem you are seeing in the new release?

[romfeo]

Hi romfeo,

Can you attach a screen shot explain the problem you are seeing in the new release?

Hello again,

Do you notice the color difference on the carpet? The darker part is where I have applied the material you have shared in this thread as a reply to my initial question

[Thoeme77]

Many many thanks to "bopangzz" for the solution and the many in-depth answers!

You made my day!

Luckily enough I don't have the coloring/gamma issues mentioned here by "romfeo". I've used the custom built version (4.20 with AR Core 1.6) generated by you guys .

The 4.21 version with AR Core 1.7 didn't work for me. On both testing phones the screen stayed black after launching the app.

[bopangzz]

Hi romfeo, Can you attach a screen shot explain the problem you are seeing in the new release?

Hello again,

Do you notice the color difference on the carpet? The darker part is where I have applied the material you have shared in this thread as a reply to my initial question

Hi romfeo,

I did the test again and I can't reproduce the issue you are having. Can you try use the ARCoreOcclusionMaterial shipped with Unreal ARCore 1.7 release, you should be able to find that material under GoogleARCore Content in the content browser. If you don't see the folder, make sure you enable "Show Engine Content" and "Show Plugin Content" int the view options.

[bopangzz]

Many many thanks to "bopangzz" for the solution and the many in-depth answers!

You made my day!

Luckily enough I don't have the coloring/gamma issues mentioned here by "romfeo". I've used the custom built version (4.20 with AR Core 1.6) generated by you guys .

The 4.21 version with AR Core 1.7 didn't work for me. On both testing phones the screen stayed black after launching the app.

Can you create a new issue for the black screen issue you saw when using 1.7 release? Can you please attach an Android logcat when the issue happens. Thank you!

[romfeo]

After I've looked in the included occlusion material in the new version of the plugin, I've realized that I was missing the 127grey texture from my material. I was using a different texture. Everything works great now! Thanks for everything

[Thoeme77]

Many many thanks to "bopangzz" for the solution and the many in-depth answers! You made my day! Luckily enough I don't have the coloring/gamma issues mentioned here by "romfeo". I've used the custom built version (4.20 with AR Core 1.6) generated by you guys . The 4.21 version with AR Core 1.7 didn't work for me. On both testing phones the screen stayed black after launching the app.

Can you create a new issue for the black screen issue you saw when using 1.7 release? Can you please attach an Android logcat when the issue happens. Thank you!

Sure, I'll do that next week.

But I have another question which came up today.

The solution works perfectly fine on Android. But on the iPad Pro (iOS 12.1.1, using Metal v.2.0) the texture didn't get updated and stayed green (in my case the default texture). Do you have any clue why that could be the case? Is there a specific reason why this wouldn't work on iOS?

[bopangzz]

That texture node only work on Android. On iOS with ARKit you will need to use the node provided by ARKit. You can check how the passthrough camera material in ARKit plug uses the node to render the background image. It should be in the content folder in your ARKit Plugin.

[Thoeme77]

That texture node only work on Android. On iOS with ARKit you will need to use the node provided by ARKit. You can check how the passthrough camera material in ARKit plug uses the node to render the background image. It should be in the content folder in your ARKit Plugin.

You're the best! That worked as well. In my case to get the right gamma correction, I had to plug in the "YUV_to_sRGBCustom" node directly (in case anyone else wonders), without using the other two. I've tested this with an iPad Pro 12.9''. I wasn't able to test this with an iPhone so far.

Edit: Works with iPhone as well.

[Thoeme77]

Quick question: Is it possible to use this setup for AR Core with a Post Process Material as well? I wanted to use this for a terrain mesh, but there it stopped working.

[bopangzz]

Quick question: Is it possible to use this setup for AR Core with a Post Process Material as well? I wanted to use this for a terrain mesh, but there it stopped working.

You mean sampling the camera texture in a Post process material? You should be able to do that kind of things. What issues you are getting?

[Thoeme77]

Quick question: Is it possible to use this setup for AR Core with a Post Process Material as well? I wanted to use this for a terrain mesh, but there it stopped working.

You mean sampling the camera texture in a Post process material? You should be able to do that kind of things. What issues you are getting?

Hi,

never mind the issues, I was just dumb. I was using the same Material Parameter Collection for two different Actors which didn't work. But I changed the setup to Vector Parameter Values which are getting instanced.

And yes exactly. I did that with the ARKit solution, but there I had no issues with the "passthrough" node (only gamma correction, which I was able to fix). See attached image for AR Kit material.

[MrPositivity]

Good Day, Im On UE4 4.22.2 and I tried all of the thees solutions mentioned in this post but they are not working for Android, When applying material and function to update the material , the shadow plane under object just looks with different aspect ratio, or feels out of place and you can clearly see where that shadow plane is. Can anyone provide a straight forward example for this? also Included material in Engine Content is not like one in Screenshot.

Aivis

[NDmitriy98]

I think it can help you: https://answers.unrealengine.com/questions/988932/render-in-main-pass-ar-android.html