Is the rendered flow a backward flow?

[iimmortall]

Excellent work! Is the rendered flow a backward flow?

[HaiyiMei]

Thanks for your attention. You mean is the rendering process differentiable? If this is the question, then the answer is no. Cos the backends of our project are blender and unreal engine, which are not differentiable.

[iimmortall]

Sorry for the confusion. I’m working on image warping using rendered optical flow, but I’m uncertain whether the rendered optical flow represents the motion from the current frame to the next frame (forward) or from the current frame to the previous frame (backward). Could you clarify?

[HaiyiMei]

Yes, it is both possible for forward and backward flow. But I just discovered there is a bug in backward flow. I'll notify you once it's fixed. Before that, you can refer to this BlenderProc.

[iimmortall]

Thank you, I am looking forward to your good news. Is the optical flow in the current version forward optical flow? And is the rendered optical flow accurate?

[HaiyiMei]

I fixed it in this commit https://github.com/openxrlab/xrfeitoria/commit/8f8f2e041fbc4c84b78ad30a70fd9f3eba2f8248. You can use it by following these steps for now (before a new version is released):

git clone https://github.com/openxrlab/xrfeitoria.git
cd xrfeitoria
git checkout alpha_channel
pip install .
python -c "import xrfeitoria as xf; xf.init_blender(replace_plugin=True, dev_plugin=True)"

Thank you, I am looking forward to your good news. Is the optical flow in the current version forward optical flow? And is the rendered optical flow accurate?

As for your question, yes I think now the forward and backward flow are all accurate.

And as for how to read .exr files, you can refer to our code here.

[iimmortall]

I use UE5 as my dev, I find the render optical flow just rg channel has value, does that mean it is forward optical flow? And is the rendered optical flow accurate at UE5 dev?

[HaiyiMei]

I use UE5 as my dev, I find the render optical flow just rg channel has value, does that mean it is forward optical flow? And is the rendered optical flow accurate at UE5 dev?

Yes, in ue5 rg is the forward optical flow, and you should also export it as exr file when rendering. I think it's pretty accurate. Unfortunately, I'm unsure if it's possible to easily generate backward flow.

[iimmortall]

Thank you! If there are any plans to support generating backward optical flow with UE5, please let me know.

[iimmortall]

Hello, I noticed that the all values of the rendered optical flow (EXR) are quite small, typically less than 1.0. How should these values be properly scaled? For instance, should they be scaled based on the rendered image size, such as flow[:,:,0] width and flow[:,:,1] height?

[HaiyiMei]

Hi, firstly, you can try to use this class to visualize the flow to see if it's weird.

[iimmortall]

The visualized optical flow appears correct, as the flow_vis.flow_to_color() function scales the flow appropriately when converting it to an RGB image.

[HaiyiMei]

I guess it's because the object is too far away from the camera? The optical flow is in camera space, so the value is tiny. Try to place the object near the camera, and see if the value is getting significantly bigger?