Lane lines cannot be displayed in x86 webcamera environment

[orcgeorge]

Describe the bug

I connected a webcamera to the openpilot, the UI can display the camera's picture, but no lane lines appear when capturing the road video.

By printing the log, I found it is stuck in the receiving loop of modeld.cc, and the value of meta_main.timestamp_sofis always 0.

    // Keep receiving frames until we are at least 1 frame ahead of previous extra frame
    while (meta_main.timestamp_sof < meta_extra.timestamp_sof + 25000000ULL) {
      buf_main = vipc_client_main.recv(&meta_main);
      if (buf_main == nullptr)  break;
    }

I tried assigning a value to meta_main.timestamp_sof in camera_common.cc CameraBuf::acquire() but it still doesn't look like the data is being transferred to the model

cur_frame_data.processing_time = (millis_since_boot() - start_time) / 1000.0;
cur_frame_data.timestamp_sof= int(cur_frame_data.frame_id * 0.05 * 1e9);
cur_frame_data.timestamp_eof= int(cur_frame_data.frame_id * 0.05 * 1e9);
VisionIpcBufExtra extra = {
cur_frame_data.frame_id,
cur_frame_data.timestamp_sof,
cur_frame_data.timestamp_eof,
};
cur_rgb_buf->set_frame_id(cur_frame_data.frame_id);
cur_yuv_buf->set_frame_id(cur_frame_data.frame_id);
vipc_server->send(cur_rgb_buf, &extra);
vipc_server->send(cur_yuv_buf, &extra);

And if I close the loop or make the value of meta_main.timestamp_sof more than 25ms, it keeps printing LOGE("vipc_client_main no frame"); error

The environment：

• Instead of using pandas, put the vehicle in the Onroad state by manually configuring should_start

OS Version

Ubuntu 20.04

openpilot version or commit

No response

Additional info

No response

[pd0wm]

What code are you using to send webcam frames over VisionIPC? Make sure you send the frames over the wide angle stream VISION_STREAM_WIDE_ROAD, and set the WideCameraOnly param to enable 1 camera mode.

[orcgeorge]

What code are you using to send webcam frames over VisionIPC? Make sure you send the frames over the wide angle stream VISION_STREAM_WIDE_ROAD, and set the WideCameraOnly param to enable 1 camera mode.

thank you for your reply!

I check out code from master branch, and I send webcam frames using camera_common.cc CameraBuf::acquire()

  cur_yuv_buf = vipc_server->get_buffer(yuv_type);
  rgb2yuv->queue(q, cur_rgb_buf->buf_cl, cur_yuv_buf->buf_cl);

  cur_frame_data.processing_time = (millis_since_boot() - start_time) / 1000.0;
  VisionIpcBufExtra extra = {
                        cur_frame_data.frame_id,
                        cur_frame_data.timestamp_sof,
                        cur_frame_data.timestamp_eof,
  };
  cur_rgb_buf->set_frame_id(cur_frame_data.frame_id);
  cur_yuv_buf->set_frame_id(cur_frame_data.frame_id);
  vipc_server->send(cur_rgb_buf, &extra);
  vipc_server->send(cur_yuv_buf, &extra);

Code of webcamera init in camera_webcam.cc

void cameras_init(VisionIpcServer *v, MultiCameraState *s, cl_device_id device_id, cl_context ctx) {
  camera_init(v, &s->road_cam, CAMERA_ID_LGC920, 20, device_id, ctx,
              VISION_STREAM_RGB_ROAD, VISION_STREAM_ROAD);
  camera_init(v, &s->driver_cam, CAMERA_ID_LGC615, 10, device_id, ctx,
              VISION_STREAM_RGB_DRIVER, VISION_STREAM_DRIVER);
  s->pm = new PubMaster({"roadCameraState", "driverCameraState", "thumbnail"});
}

Should I change VISION_STREAM_RGB_ROAD, VISION_STREAM_ROAD here to VISION_STREAM_RGB_WIDE_ROAD, VISION_STREAM_WIDE_ROAD ?

By the way, if I want to set the WideCameraOnly param. Is it by the following method?

• Add {"WideCameraOnly", PERSISTENT}, in param.cc .

• Add a new file named WideCameraOnly in /home/.comma/params/d/ and set value 1.

[pd0wm]

Looks like you're not using the latest code. camera_webcam.cc was removed because it no longer worked. That param should already be in params.cc.

The way forward is a webcam camera server written in python using something like openCV to do the heavy lifting, we just haven't gotten around to actually writing it (https://github.com/commaai/openpilot/issues/24590). There are python bindings for VisionIPC that you can use for this purpose.

https://github.com/commaai/openpilot/blob/master/tools/sim/bridge.py has some examples on how to send frames.

[orcgeorge]

Well, I found a version for x86 webcam in @zorrobyte project https://github.com/zorrobyte/openpilot Although currently running his code, there is still no lane line display, but it seems to have the correct direction.

[zorrobyte]

Well, I found a version for x86 webcam in @zorrobyte project https://github.com/zorrobyte/openpilot Although currently running his code, there is still no lane line display, but it seems to have the correct direction.

I stopped work as comma was going to re-write x86 and it's unfinished. Willem is right, everything you need should be in bridge.py. Apart from coming up with transforms, you'll just need to handle threading in a smart way as OpenCV can be a little rough with multiple camera streams.

For experimentation, I'd recommend VidGear as it'll abstract most of the difficulty handling cameras so you can focus on the implementation @orcgeorge https://abhitronix.github.io/vidgear/v0.2.5-stable/

And as a final note, I chose to leave the comma discord. I hang out on the Retropilot one these days and focus on OSS hardware. Cheers!

[SyazwanDev]

Describe the bug

I connected a webcamera to the openpilot, the UI can display the camera's picture, but no lane lines appear when capturing the road video.

By printing the log, I found it is stuck in the receiving loop of modeld.cc, and the value of meta_main.timestamp_sofis always 0.

    // Keep receiving frames until we are at least 1 frame ahead of previous extra frame
    while (meta_main.timestamp_sof < meta_extra.timestamp_sof + 25000000ULL) {
      buf_main = vipc_client_main.recv(&meta_main);
      if (buf_main == nullptr)  break;
    }

I tried assigning a value to meta_main.timestamp_sof in camera_common.cc CameraBuf::acquire() but it still doesn't look like the data is being transferred to the model

  cur_frame_data.processing_time = (millis_since_boot() - start_time) / 1000.0;
  cur_frame_data.timestamp_sof= int(cur_frame_data.frame_id * 0.05 * 1e9);
  cur_frame_data.timestamp_eof= int(cur_frame_data.frame_id * 0.05 * 1e9);
  VisionIpcBufExtra extra = {
                        cur_frame_data.frame_id,
                        cur_frame_data.timestamp_sof,
                        cur_frame_data.timestamp_eof,
  };
  cur_rgb_buf->set_frame_id(cur_frame_data.frame_id);
  cur_yuv_buf->set_frame_id(cur_frame_data.frame_id);
  vipc_server->send(cur_rgb_buf, &extra);
  vipc_server->send(cur_yuv_buf, &extra);

And if I close the loop or make the value of meta_main.timestamp_sof more than 25ms, it keeps printing LOGE("vipc_client_main no frame"); error

The environment：

• Instead of using pandas, put the vehicle in the Onroad state by manually configuring should_start

OS Version

Ubuntu 20.04

openpilot version or commit

No response

Additional info

No response

Hi, do you mind sharing which openpilot version are you using?