Open milmario opened 2 years ago
I have the same question and look forward to answering it.
To save you some time, here's what i found out so far:
To summarize, my visual & numerical tests show the following:
I'm looking forward to your input.
@milmario Thanks for your help, I have tested the assumption of using the simulated optical flow as the forward flow t->t+1 using the backward warping function provide by mmcv (https://github.com/open-mmlab/mmcv/blob/master/mmcv/video/optflow.py#L143).
I first normalize the raw flow map (captured at time t) with the shape of [frame_height, frame_width, 2], to the pixel displacements.
flow_data[:, :, 0] *= frame_width * 0.5
flow_data[:, :, 1] *= frame_height * -0.5
Then use the backward warping function on the second frame (at t+1).
warped_image = flow_warp(rgb2_image, flow_data)
The warped result seems match to the first frame (at t).
flow
sampled flow direction on the first frame
the first frame
the second frame
the warped result
I'm not sure whether this assumption is correct. I am working on test as the backward flow from t to t-1. Besides, I hope we can get further official clarification from the Carla team.
To summarize, my visual & numerical tests show the following: Flow[i] is the optical flow from i->i-1 (so backward optical flow).
I think this assumption are right. I have tested the assumption of using the simulated optical flow as the forward flow (t->t+1) or the backward flow (t->t-1) using the backward warping function provide by mmcv (https://github.com/open-mmlab/mmcv/blob/master/mmcv/video/optflow.py#L143).
For given three rgb image rgb0_image
, rgb1_image
, rgb2_image
(at time t-1, t, t+1, with the shape of [frame_height, frame_width, 3]) and one flow data flow_data
(at time t, with the shape of [frame_height, frame_width, 2]) obtained by carla.
rgb0_image
rgb1_image
rgb2_image
I first normalize the raw flow data to the pixel displacements.
flow_10 = flow_data.copy()
flow_10[:, :, 0] *= frame_width * -0.5
flow_10[:, :, 1] *= frame_height * 0.5
flow_12 = flow_data.copy()
flow_12[:, :, 0] *= frame_width * 0.5
flow_12[:, :, 1] *= frame_height * -0.5
flow_10
sampled backward flow vector on the first frame
flow_12
sampled forward flow vector on the first frame
Then use the backward warping function. For forward flow, warp on the next image. For backward flow, warp on the pervious image.
warped_image0 = flow_warp(rgb0_image, flow_10)
warped_image2 = flow_warp(rgb2_image, flow_12)
warped_image0
warped_image2
The warped result using backward flow seems more matched to the frame rgb1_image
at t.
In addition, the flow at time t and the image of at time t are simulated at the same time. At this time, the image of frame t+1 has not been simulated, and the future motion is uncertain. Therefore, I think it is impossible to obtain the forward optical flow from t to t+1 during real-time simulation, which is also consistent with the simple experimental results above.
I'm not sure whether this conclusion is correct. I hope we can get further official clarification from the Carla team.
Hey, @danqu130 thanks for the great visualizations! I have a question about how you obtained the flow_data
from the raw data. I read in the Carla docs that the raw data is 64-bit but if I use np.frombuffer
and dtype float64 to convert from raw to a numpy array, I then receive only one value per pixel location, instead of the expected two values. Any insight would be appreciated. Thanks in advance!
Hey, @danqu130 thanks for the great visualizations! I have a question about how you obtained the
flow_data
from the raw data. I read in the Carla docs that the raw data is 64-bit but if I usenp.frombuffer
and dtype float64 to convert from raw to a numpy array, I then receive only one value per pixel location, instead of the expected two values. Any insight would be appreciated. Thanks in advance!
The code I use to convert carla.Image data
into numpy array flow_data
is as follows. It may be slow, but it works.
flow_data = np.array([(pixel.x, pixel.y) for pixel in data], dtype=np.float32)
flow_data = flow_data.reshape((data.height, data.width, 2))
I was confused when I observed the simulated optical flow. It has obvious layered optical flow in some scenes, but it seems unreasonable. Let me show an example of a vehicle moving forward.
This is the simulated flow and the corresponding first frame.
And this is the optical flow computed by RAFT (https://github.com/princeton-vl/RAFT-3D) using the first and the sceond frame.
I calculated the EPE error of these two flow, and marked the position where the error is greater than 1 pixel as white.
The layering problem in the CARLA flow does not match the real motion, which is also confirmed by the results of RAFT. I have tried a variety of motion speeds and simulation frame rates, which also have this problem. In addition, I have checked my optical flow visualization steps. Even if I directly use the method provided by Carla (carla.OpticalFlowImage.get_color_coded_flow()
), such layering still exists.
Hey, @milmario @jonasdieker, did you also encounter this? Do you think it's normal?
@danqu130, under which simulated conditions (light, speed, camera intrinsic,...) do you get these artefacts? Can you please share your overall config for the above result? It seems that other carla-flow samples look quite good, so I am wondering where it comes from.
Just to make sure: Is this the (very) first frame you are showing? If this is the case, it would make sense to have such artefacts, since OF is backward.
CARLA version is 0.9.13, 4.26.2-0+++UE4+Release-4.26 I can see this artifact by directly using the official manual_control.py python example, with autopilot mode and change to show the optical flow sensor.
I think it is not normal for such artifacts to appear in this autonomous driving scene, whether forward flow or backward flow. In addition, has the CARLA team confirmed that the simulated optical flow is backward flow?
@danqu130 I've encountered the same issue of this staircase effect. Did anyone solve it?
@danqu130 I've encountered the same issue of this staircase effect. Did anyone solve it?
Unfortunately, I don't have any more updates.
Hello, I'm not sure that Carla Optical Flow is in backward mode, I made some test that give me the impression that there are forward.
Just by recording a car passing through the camera:
Image N:
Image N+1:
OpticalFlow N+1:
You will see that the car on optical flow image (middleburry colormap) is aligned with N+1 image and blue color correspond to a vector moving to the left:
This test make me think that we are on forward mode but a confirmation of CARLA team could be interesting.
I also noticed the quantization phenomenon and the 10 bit explanation from this ticket : https://github.com/carla-simulator/carla/issues/5514 seem interesting but same, some confirmation from CARLA team.
Does anyone find a trick to workaround this effect ?
@Axel1092 do you have any clue ?
Best regards
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
For me, the definition of the optical flow output of CARLA is not clear.