search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.49k stars 4.8k forks source link

Color stream cannot work at 1920x1080 resolution, showing "Incomplete video frame detected" #3042

Closed oiramario closed 5 years ago

oiramario commented 5 years ago
Required Info
Camera Model D415
Firmware Version 05.11.01.00
Operating System & Version Ubuntu 18.04
Kernel Version (Linux Only) 4.4.143
Platform nanopi m4 (rockchip RK3399)
SDK Version 2.17.1
Language c++
Segment AR
RK3399
RAM 4G
DISK 16G eMMC
CPU Dual-Core Cortex-A72(up to 2.0GHz) + Quad-Core Cortex-A53(up to 1.5GHz)
GPU Mali-T864 GPU，supports OpenGL ES1.1/2.0/3.0/3.1
USB Type four USB 3.0 Type-A ports
Power supply DC 5V/3A

Issue Description

rs-enumerate-devices

shot

I plug D415 into USB3.0. Using GLESv2 + GBM + DRM to show color stream with Zero-Copy. For the purpose of reducing the transmission bandwidth, I set the format RS2_FORMAT_YUYV to display the color stream.

When set resolution to 1280 x 720 it can running fast with 30fps. However, when the resolution is set to 1920 x 1080, the log shows a lot of Incomplete video frame detected. The problem occurred after the startup program, the entire motherboard and CPU did not overheat.

I have reviewed the description of such a problem in the forum, pointing out that the problem is due to insufficient usb capacity, or the CPU load is too high. However, the official recommendation of intel to run the D400 device is AAEON UP Board (see link below), its ability is equivalent to RK3399, so I don't think it is the reason for hardware capabilities. https://click.intel.com/intelr-realsensetm-depth-camera-d415-aaeonr-up-board.html

Also, what is the native format of the D415 color stream? Is it YUYV or UYVY? If it is because of the lack of computing power due to format conversion, or the problem of USB transmission bandwidth, I can directly use the native format and then quickly convert through GPU. Please give your suggestion.

Code attached below:

rs2::context ctx;
auto devicelist = ctx.query_devices();
if (devicelist.size() > 0)
{
    rs2::log_to_console(RS2_LOG_SEVERITY_DEBUG);
    rs2::device dev = devicelist.front();
    fprintf(stdout, "\nRealsense Device info---\n"
                    "    Name              : %s\n"
            "    Serial Number     : %s\n"
            "    Firmware Version  : %s\n"
            "    USB Type          : %s\n"
            "    Stream Color      : %d, %d, %d\n",
                dev.get_info(RS2_CAMERA_INFO_NAME),
                dev.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER),
                dev.get_info(RS2_CAMERA_INFO_FIRMWARE_VERSION),
                dev.get_info(RS2_CAMERA_INFO_USB_TYPE_DESCRIPTOR),
                RS_WIDTH, RS_HEIGHT, RS_FPS);

    rs2::config cfg;
    cfg.enable_stream(RS2_STREAM_COLOR, 1920, 1080, RS2_FORMAT_YUYV, 30);
    pipe.start(cfg);

    processing_thread = std::thread([&](){
        while (running)
        {
            rs2::frameset fs;
            if (pipe.try_wait_for_frames(&fs, 1000))
            {
                rs2::frame color_frame = fs.get_color_frame();
                update_texture(texture, color_frame.get_data());
            }
        }
    });
}

The log file shows that:

Realsense Device info---
    Name              : Intel RealSense D415
    Serial Number     : 732612060238
    Firmware Version  : 05.11.01.00
    USB Type          : 3.2
    Stream Color      : 1920, 1080, 30

 10/01 12:52:49,118 INFO [548547798704] (backend-hid.cpp:1178) Failed to read busnum/devnum. Device Path: /sys/bus/iio/devices/iio:device0
 10/01 12:52:49,118 DEBUG [548547798704] (ds5-factory.cpp:568) try_fetch_usb_device(...) failed.
 10/01 12:52:49,123 INFO [548547798704] (backend-v4l2.cpp:1468) v4l_uvc_meta_device
 10/01 12:52:49,127 INFO [548547798704] (backend-v4l2.cpp:1468) v4l_uvc_meta_device
 10/01 12:52:49,136 INFO [548547798704] (backend-v4l2.cpp:1468) v4l_uvc_meta_device
 10/01 12:52:49,141 DEBUG [548547798704] (backend-v4l2.cpp:1254) Recognized pixel-format Depth data 16-bit (Z16)
 10/01 12:52:49,147 DEBUG [548547798704] (backend-v4l2.cpp:1254) Recognized pixel-format 8-bit Greyscale
 10/01 12:52:49,148 DEBUG [548547798704] (backend-v4l2.cpp:1254) Recognized pixel-format UYVY 4:2:2
 10/01 12:52:49,148 DEBUG [548547798704] (backend-v4l2.cpp:1254) Recognized pixel-format 8-bit Greyscale
 10/01 12:52:49,154 DEBUG [548547798704] (backend-v4l2.cpp:1254) Recognized pixel-format Greyscale 8 L/R (Y8I)
 10/01 12:52:49,160 DEBUG [548547798704] (backend-v4l2.cpp:1254) Recognized pixel-format Greyscale 12 L/R (Y12I)
 10/01 12:52:49,172 DEBUG [548547798704] (backend-v4l2.cpp:1254) Recognized pixel-format YUYV 4:2:2
 10/01 12:52:49,179 WARNING [548547798704] (backend-v4l2.cpp:1248) Pixel format 36315752-1a66-a242-9065-d01814a likely requires patch for fourcc code RW16!
 10/01 12:52:49,180 WARNING [548547798704] (sensor.cpp:338) Unregistered Media formats : [ RW16 ]; Supported: [ ]
 10/01 12:52:49,206 INFO [548547798704] (backend-v4l2.cpp:1460) Trying to configure fourcc YUYV
 10/01 12:52:49,206 DEBUG [548547798704] (backend-v4l2.cpp:1587) Metadata node was successfully configured to D4XX format
 10/01 12:52:49,237 INFO [548170531248] (backend-v4l2.cpp:996) Empty video frame arrived
 10/01 12:52:49,239 INFO [548170531248] (backend-v4l2.cpp:996) Empty video frame arrived
 10/01 12:52:49,239 INFO [548170531248] (backend-v4l2.cpp:996) Empty video frame arrived
 10/01 12:52:49,239 INFO [548170531248] (backend-v4l2.cpp:996) Empty video frame arrived
 10/01 12:52:49,242 INFO [548547798704] (backend-v4l2.cpp:1473) ~v4l_uvc_meta_device
 10/01 12:52:49,242 INFO [548547798704] (backend-v4l2.cpp:1473) ~v4l_uvc_meta_device
 10/01 12:52:49,247 INFO [548547798704] (backend-v4l2.cpp:1473) ~v4l_uvc_meta_device
 10/01 12:52:49,383 WARNING [548170531248] (backend-v4l2.cpp:1007) FD_ISSET returned false - video node is not signalled (md only)
 10/01 12:52:49,386 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 12:52:49,416 WARNING [548170531248] (backend-v4l2.cpp:1007) FD_ISSET returned false - video node is not signalled (md only)
 10/01 12:52:49,419 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 12:52:49,450 WARNING [548170531248] (backend-v4l2.cpp:1007) FD_ISSET returned false - video node is not signalled (md only)
 10/01 12:52:49,453 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 12:52:49,490 WARNING [548170531248] (backend-v4l2.cpp:1007) FD_ISSET returned false - video node is not signalled (md only)
 10/01 12:52:49,491 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 1098504 out of 4147455 bytes (26%)
 10/01 12:52:49,520 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 12:52:49,554 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 12:52:49,587 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 12:52:49,622 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 12:52:49,655 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 12:52:49,687 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 12:52:49,719 INFO [548170531248] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 12:52:49,752 INFO [548170531248] (types.h:936) Incomplete video frame detected!
dorodnic commented 5 years ago

Hi @oiramario

For the RGB sensor, the native format is YUYV (fourcc 'YUY2').

The D415 also offers lower quality RGB stream from the left infrared camera. This stream is available at up to 720p and suffers from projector interference. On the bright side, it is pixel-aligned with depth and projector pattern can be mostly removed with a special preset. For this stream the format is UYVY.

If I understand the problem correctly, you are getting 0 complete frames at 1080p from V4L2 at all, right? Is this only when depth streaming as well, or also when streaming color alone?

I do believe this capability was functional on the Up-Board. If you'd like, I can double-check, but don't have one next to me right now.

dorodnic commented 5 years ago

Something I think might be relevant - LKML Asynchronous UVC To rule out librealsense, you could test the system using something like yavta In general, the Incomplete video frame detected! warning originates in the uvc_video Kernel Module (it is a flag on the buffer object if I recall correctly). However, if we forgot a thread doing while(true); somewhere in librealsense it would certainly be a problem :) Otherwise, librealsense should add very little overhead when configured as you described. @ev-mp

oiramario commented 5 years ago

Hi @dorodnic

I tried to set RS2_FORMAT_UYVY, but librealsense returned an error: 

terminate called after throwing an instance of 'rs2::error'
  what():  Couldn't resolve requests

I'm getting few frames at 1080p when streaming color alone, depth stream not test yet. I think there should be no problem with the depth of the 720P format Z16.

10/01 14:43:23,244 DEBUG [547323199920] (archive.cpp:287) CallbackFinished,Color,5,DispatchedAt,1547131403244.034912
 10/01 14:43:23,244 DEBUG [547323199920] (archive.cpp:294) Frame Callback [Color#5] overdue. (Duration: 110.381592ms, FPS: 30, Max Duration: 32ms)
 10/01 14:43:23,244 DEBUG [547323199920] (archive.cpp:501) CallbackStarted,Color,6,DispatchedAt,1547131403244.155273
 10/01 14:43:23,244 DEBUG [547323199920] (archive.cpp:287) CallbackFinished,Color,0,DispatchedAt,1547131403244.255859
 10/01 14:43:23,244 DEBUG [547323199920] (archive.cpp:287) CallbackFinished,Color,0,DispatchedAt,1547131403244.338379
 10/01 14:43:23,244 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 967432 out of 4147455 bytes (23%)
 10/01 14:43:23,250 DEBUG [547314807216] (archive.cpp:287) CallbackFinished,Color,0,DispatchedAt,1547131403250.352783
 10/01 14:43:23,250 DEBUG [547314807216] (archive.cpp:294) Frame Callback [Color#0] overdue. (Duration: 1547131403250.352783ms, FPS: 30, Max Duration: 32ms)
 10/01 14:43:23,288 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 2966280 out of 4147455 bytes (71%)
 10/01 14:43:23,302 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,336 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,370 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:23,403 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,436 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,470 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,504 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,537 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,571 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:23,604 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,638 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,672 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:23,713 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 1196808 out of 4147455 bytes (28%)
 10/01 14:43:23,739 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:23,775 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 639752 out of 4147455 bytes (15%)
 10/01 14:43:23,815 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 1426184 out of 4147455 bytes (34%)
 10/01 14:43:23,838 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:23,872 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:23,910 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 770824 out of 4147455 bytes (18%)
 10/01 14:43:23,941 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:23,974 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:24,008 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:24,041 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:24,104 DEBUG [547323199920] (sensor.cpp:450) FrameAccepted,Color,7,0,7,Arrived,1547131404071.235107 1547131404103.760986 diff - 32.525879 ,TS,1547131404103.776611,TS_Domain,System Time last_frame_number 6 last_timestamp 1547131403230.192871
 10/01 14:43:24,107 DEBUG [547323199920] (archive.cpp:501) CallbackStarted,Color,7,DispatchedAt,1547131404107.559082
 10/01 14:43:24,108 DEBUG [547323199920] (sync.cpp:143) DISPATCH (TS: )--> Color 7 1547131404103.776611 

 10/01 14:43:24,108 DEBUG [547323199920] (sync.cpp:551) clean inactive stream in (TS: )Depth Infrared Infrared Color 
 10/01 14:43:24,108 DEBUG [547323199920] (sync.cpp:551) clean inactive stream in (TS: )Depth Infrared Infrared Color 
 10/01 14:43:24,108 DEBUG [547323199920] (sync.cpp:551) clean inactive stream in (TS: )Depth Infrared Infrared Color 
 10/01 14:43:24,108 DEBUG [547323199920] (sync.cpp:551) clean inactive stream in (TS: )Depth Infrared Infrared Color 
 10/01 14:43:24,108 DEBUG [547323199920] (metadata-parser.h:153) Metadata mismatch - actual: 0x0, expected: 0x80000005 (Intel RGB Control)
 10/01 14:43:24,108 DEBUG [547323199920] (sync.cpp:143) DISPATCH (TS: I Depth I Infrared I Infrared I Color )--> Color 7 1547131404103.776611 

 10/01 14:43:24,108 DEBUG [547323199920] (sync.cpp:551) clean inactive stream in (TS: I Depth I Infrared I Infrared I Color )Color 
 10/01 14:43:24,109 DEBUG [547323199920] (metadata-parser.h:153) Metadata mismatch - actual: 0x0, expected: 0x80000005 (Intel RGB Control)
 10/01 14:43:24,109 DEBUG [547323199920] (sync.cpp:99) I Color--> Color 7, 1547131404103.776611

 10/01 14:43:24,109 DEBUG [547323199920] (sync.cpp:258) SYNC (TS: I Depth I Infrared I Infrared I Color )--> Color 7 1547131404103.776611 

 10/01 14:43:24,109 DEBUG [547323199920] (metadata-parser.h:153) Metadata mismatch - actual: 0x0, expected: 0x80000005 (Intel RGB Control)
 10/01 14:43:24,109 DEBUG [547323199920] (sync.cpp:516) fps 1 Color 7 1547131404103.776611 
 10/01 14:43:24,109 DEBUG [547323199920] (sync.cpp:529) (TS: I Depth I Infrared I Infrared I Color )Color 7 1547131404103.776611 fps 1 gap 1000.000000 next_expected: 1547131405103.776611
 10/01 14:43:24,109 DEBUG [547323199920] (sync.cpp:258) SYNC (TS: )--> Color 7 1547131404103.776611 

 10/01 14:43:24,110 DEBUG [547323199920] (metadata-parser.h:153) Metadata mismatch - actual: 0x0, expected: 0x80000005 (Intel RGB Control)
 10/01 14:43:24,110 DEBUG [547323199920] (sync.cpp:516) fps 1 Color 7 1547131404103.776611 
 10/01 14:43:24,110 DEBUG [547323199920] (sync.cpp:529) (TS: )Color 7 1547131404103.776611 fps 1 gap 1000.000000 next_expected: 1547131405103.776611
 10/01 14:43:24,110 DEBUG [547323199920] (archive.cpp:287) CallbackFinished,Color,0,DispatchedAt,1547131404110.459229
 10/01 14:43:24,110 DEBUG [547323199920] (archive.cpp:294) Frame Callback [Color#0] overdue. (Duration: 1547131404110.459229ms, FPS: 30, Max Duration: 32ms)
 10/01 14:43:24,110 DEBUG [547323199920] (syncer-processing-block.cpp:28) SYNCED: Color 7, 1547131404103.776611 
 10/01 14:43:24,110 DEBUG [547323199920] (archive.cpp:501) CallbackStarted,Color,7,DispatchedAt,1547131404110.847168
 10/01 14:43:24,111 DEBUG [547323199920] (archive.cpp:287) CallbackFinished,Color,6,DispatchedAt,1547131404111.008789
 10/01 14:43:24,111 DEBUG [547323199920] (archive.cpp:294) Frame Callback [Color#6] overdue. (Duration: 868.772949ms, FPS: 30, Max Duration: 32ms)
 10/01 14:43:24,111 DEBUG [547323199920] (archive.cpp:501) CallbackStarted,Color,7,DispatchedAt,1547131404111.299072
 10/01 14:43:24,111 DEBUG [547323199920] (archive.cpp:287) CallbackFinished,Color,0,DispatchedAt,1547131404111.460205
 10/01 14:43:24,111 DEBUG [547323199920] (archive.cpp:287) CallbackFinished,Color,0,DispatchedAt,1547131404111.592285
 10/01 14:43:24,111 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:24,117 DEBUG [547314807216] (archive.cpp:287) CallbackFinished,Color,0,DispatchedAt,1547131404117.408447
 10/01 14:43:24,117 DEBUG [547314807216] (archive.cpp:294) Frame Callback [Color#0] overdue. (Duration: 1547131404117.408447ms, FPS: 30, Max Duration: 32ms)
 10/01 14:43:24,142 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:24,176 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,209 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,242 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,278 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,313 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,346 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,380 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:24,413 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,446 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,481 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,514 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,548 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,581 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,615 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,649 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,683 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,713 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:24,749 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 377608 out of 4147455 bytes (9%)
 10/01 14:43:24,785 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 443144 out of 4147455 bytes (10%)
 10/01 14:43:24,823 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 1458952 out of 4147455 bytes (35%)
 10/01 14:43:24,849 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,883 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:24,916 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:24,951 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:24,989 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 1131272 out of 4147455 bytes (27%)
 10/01 14:43:25,015 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:25,048 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:25,082 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,118 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,151 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,185 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,218 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,252 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,286 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,320 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,353 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,387 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,420 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,454 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,487 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,521 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,554 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,588 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,622 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,655 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,689 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,723 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,756 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,789 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,824 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,856 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 410376 out of 4147455 bytes (9%)
 10/01 14:43:25,891 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:25,922 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:25,955 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:25,990 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 377608 out of 4147455 bytes (9%)
 10/01 14:43:26,025 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:26,059 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,098 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 1327880 out of 4147455 bytes (32%)
 10/01 14:43:26,126 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:26,158 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:26,190 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:26,227 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,258 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:26,293 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,325 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 246536 out of 4147455 bytes (5%)
 10/01 14:43:26,361 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:26,395 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,428 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 181000 out of 4147455 bytes (4%)
 10/01 14:43:26,462 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,495 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,529 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,563 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,596 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,630 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,663 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,697 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,730 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,764 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 213768 out of 4147455 bytes (5%)
 10/01 14:43:26,796 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:26,830 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:26,864 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:26,897 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:26,931 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
^Cselect err: Interrupted system call
 10/01 14:43:26,964 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:26,998 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:27,032 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 279304 out of 4147455 bytes (6%)
 10/01 14:43:27,082 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 2409224 out of 4147455 bytes (58%)
 10/01 14:43:27,099 INFO [547323199920] (types.h:936) Incomplete video frame detected!
Size 443144 out of 4147455 bytes (10%)
 10/01 14:43:27,126 INFO [547323199920] (backend-v4l2.cpp:923) Stream finished
 10/01 14:43:27,134 INFO [547636520624] (archive.cpp:354) The user was holding on to 1 frames after stream 0x0x2e6dd2d0 stopped
 10/01 14:43:27,136 INFO [547636520624] (backend-v4l2.cpp:1473) ~v4l_uvc_meta_device
 10/01 14:43:27,136 INFO [547636520624] (backend-v4l2.cpp:1473) ~v4l_uvc_meta_device
 10/01 14:43:27,138 INFO [547636520624] (backend-v4l2.cpp:1473) ~v4l_uvc_meta_device
 10/01 14:43:27,139 DEBUG [547636520624] (archive.cpp:287) CallbackFinished,Color,7,DispatchedAt,1547131407139.421631
 10/01 14:43:27,139 DEBUG [547636520624] (archive.cpp:294) Frame Callback [Color#7] overdue. (Duration: 3031.862549ms, FPS: 30, Max Duration: 32ms)
 10/01 14:43:27,139 INFO [547636520624] (archive.cpp:364) All frames from stream 0x0x2e6dd2d0 are now released by the user

If you can, I recommend that you check it out on Up-Board.

I understand what you mean, the problem should be in the UVC protocol here, after all, ZED supports 4K resolution. I will try to read the references you give, but I think this is really important, it is best done by your development team.

RealSenseCustomerSupport commented 5 years ago

[Please Ignore - RealSense system comment]

RealSenseCustomerSupport commented 5 years ago

@oiramario Did you get chance to try the patch in LKML Asynchronous UVC?

RealSenseCustomerSupport commented 5 years ago

Patch link not synced just now. Here is the link: https://lkml.org/lkml/2018/1/3/699

oiramario commented 5 years ago

Not yet. I'll try the patch next week.

RealSenseCustomerSupport commented 5 years ago

@Oiramario Good. Looking forward to your update!

oiramario commented 5 years ago

Hi guys, in this question, the rockchip linux kernel I used has already hit the patch you mentioned. See these below: FROMLIST: media: uvcvideo: Protect queue internals with helper FROMLIST: media: uvcvideo: queue: Support asynchronous buffer handling

Then what's your suggestions?

RealSenseCustomerSupport commented 5 years ago

@Oiramario Please try to add below two patches.

1. modify_urb_number_and_uvc_packet.patch --- a/drivers/media/usb/uvc/uvcvideo.h +++ b/drivers/media/usb/uvc/uvcvideo.h @@ -172,9 +172,9 @@

define DRIVER_VERSION "1.1.1"

/ Number of isochronous URBs. / -#define UVC_URBS 5 +#define UVC_URBS 32 / Maximum number of packets per URB. / -#define UVC_MAX_PACKETS 32 +#define UVC_MAX_PACKETS 16 / Maximum status buffer size in bytes of interrupt URB. /

define UVC_MAX_STATUS_SIZE 32

define UVC_CTRL_CONTROL_TIMEOUT 300

2. rk3399-irqchip-xhci-irq-142-to-all-cpu.patch

--- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -421,6 +421,12 @@ static void __init gic_dist_init(void) affinity = gic_mpidr_to_affinity(cpu_logical_map(smp_processor_id())); for (i = 32; i < gic_data.irq_nr; i++) gic_write_irouter(affinity, base + GICD_IROUTER + i * 8); +

oiramario commented 5 years ago

I'll check this out.

RealSenseCustomerSupport commented 5 years ago

XiangyuX Sun, Jan 28, 10:04 PM

@oiramario Any good news? Looking forward to your update. Thanks!

RealSenseCustomerSupport commented 5 years ago

@oiramario Can the patch work for you? Looking forward to your result. Thanks!

RealSenseCustomerSupport commented 5 years ago

@oiramario Does it works? Please update. Thanks!

oiramario commented 5 years ago

@RealSenseCustomerSupport I just came back from vacation and I will check this out next week.

RealSenseCustomerSupport commented 5 years ago

@oiramario Any update? Thanks!

oiramario commented 5 years ago

Sorry, the confirmation of the patch is delayed because of some other work. There is a problem with the patch content you posted, and it cannot be used directly after copying it. Referring to the code, I re-created the patch and successfully patched it into the kernel. However, I found that in the kernel v4.4.143 used by rockchip, the macro UVC_MAX_STATUS_SIZE is 16, and 32 of the content you sent, I am not very clear about this part, I hope you can help confirm it. Next week I will confirm the modified kernel on the board of rockchip 3399, thank you.

modify_urb_number_and_uvc_packet.txt rk3399-irqchip-xhci-irq-142-to-all-cpu.txt

oiramario commented 5 years ago

For your reference: https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/usb/uvc/uvcvideo.h

shot

RealSenseCustomerSupport commented 5 years ago

@oiramario The difference I think is due to different kernel version. You can have a try with your setting of UVC_MAX_STATUS_SIZE

RealSenseCustomerSupport commented 5 years ago

@oiramario How's it going? Any update? Thanks!

RealSenseCustomerSupport commented 5 years ago

@oiramario Any good news?

RealSenseCustomerSupport commented 5 years ago

@oiramario Now unroot android solution is included in 2.19 librealsense. No root needed and no patch needed. Please have a try to see if this can fix your issue. Thanks! https://github.com/IntelRealSense/librealsense/tree/master/wrappers/android

landersson commented 5 years ago

@oiramario , sorry to hijack this thread... I'm trying to do what you seem to have succeeded in doing, i.e getting a RealSense device to work on a NanoPI M4. Please let me know if you would be happy to outline the basic steps that are required... I have successfully build the realsense libs/tools and I can grab images from some of the /dev/video* devices created, but when trying the rs-enumerate-devices script I'm just getting "No device detected. Is it plugged in?"... I guess I have to patch and rebuild a custom kernel?

RealSenseCustomerSupport commented 5 years ago

@landersson What's OS you getting the issue, android or linux?

landersson commented 5 years ago

I'm using the the standard Ubuntu 18.04 based "FriendlyDesktop" image for NanoPI M4. Default kernel 4.4.143. Basically the same setup as oiramario initially reported.

RealSenseCustomerSupport commented 5 years ago

@oiramario Sorry, my mistake. I've thought the issue was found on Android. Will check this issue on Ubuntu and update you later.

oiramario commented 5 years ago

@oiramario , sorry to hijack this thread... I'm trying to do what you seem to have succeeded in doing, i.e getting a RealSense device to work on a NanoPI M4. Please let me know if you would be happy to outline the basic steps that are required... I have successfully build the realsense libs/tools and I can grab images from some of the /dev/video* devices created, but when trying the rs-enumerate-devices script I'm just getting "No device detected. Is it plugged in?"... I guess I have to patch and rebuild a custom kernel?

Follow these steps:

1. apt-get install git libssl-dev libusb-1.0-0-dev pkg-config
2. git clone --depth 1 https://github.com/IntelRealSense/librealsense.git
3. cd librealsense
4. sudo cp config/99-realsense-libusb.rules /etc/udev/rules.d/ 
   sudo udevadm control --reload-rules && udevadm trigger
5. mkdir build && cd build
   cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_WITH_TM2=false -DBUILD_GRAPHICAL_EXAMPLES=false -DBUILD_EXAMPLES=false
   make -j$(nproc) && make install 
6. rs-enumerate-devices

More detail: https://github.com/IntelRealSense/librealsense/blob/master/doc/installation.md

Nanopi-m4 uses kernel 4.4, so we don't need to do anything like patch or rebuild custom kernel.

oiramario commented 5 years ago

@oiramario Any good news?

Sorry, I have been stuck in some kernel-related issues recently, but I will take time to verify this issue as soon as possible.

landersson commented 5 years ago

Thanks @oiramario, those are pretty much exactly the steps I took too, but I'm still getting "No device detected..." when running rs-enumerate-devices. Running as sudo doesn't help.

My camera is a 435i, so I guess that must be why we're getting different results. Apart from the camera model, our setups are nearly identical (except you have 4G RAM while I have 2).

landersson commented 5 years ago

@oiramario , did you connect the depth camera via some kind of powered USB hub, or straight to the Nano PI M4?

I'm thinking perhaps the NanoPI can't provide enough power via USB to run the D435i...

oiramario commented 5 years ago

I didn't have any USB hub here, just plugged D415 into NanoPi-M4. Could you post some logs or shots?

landersson commented 5 years ago

Ok, I tried with a powered USB3 hub, but that didn't change anything. I've attached an strace of rs-enumerate-devices, the output of "lsusb -vvv" and the end of dmesg... Let me know if there's anything else that can help you figure out why this doesn't work. Thanks.

lsusb.txt dmesg-tail.txt strace.txt

RealSenseCustomerSupport commented 5 years ago

@oiramario Did you get chance to try again with the kernel patch?

@landersson How many USB ports on your NanoPI? Did you try them all? Seen from the result of lsusb, the camera has been connected and recognized and rs-enumerate should work. Please double check if "sudo cp config/99-realsense-libusb.rules /etc/udev/rules.d/ " executed successfully.

landersson commented 5 years ago

The Nano M4 have 4 USB3 ports.. I've tried them all as well as connecting the D435i using an external, powered USB3 hub.The udev script has been properly installed.

landersson commented 5 years ago

Just took some time to look further into this... With logging enabled, I see this message:

22/03 03:19:33,133 WARNING [547723022352] (ds5-factory.cpp:617) DS5 group_devices is empty.

Looking at src/ds5/ds5-factory.cpp:591, it looks like my D435i is disabled because it's a HID device, but no HID sensors were detected. I assume the HID sensors (gyro and accelerometer) would be detected if with a patched kernel, so I guess I'll just have to go ahead and rebuild the kernel.

@oiramario, if you have any pointers in regards to rebuilding the kernel for the NanoPI M4, that would be great.

RealSenseCustomerSupport commented 5 years ago

@landersson What's the FW version of your D435i? Could you please try to connect it to one PC to see if it's 5.10.13+ and if it can work normally? Please be noted that "D435i requires FW version 5.10.13+ and is currently supported on Linux and Windows 10 only. "

RealSenseCustomerSupport commented 5 years ago

@landersson If need HID related pacth, please refer to https://github.com/IntelRealSense/librealsense/tree/master/scripts

Please select the patch accordingly such as https://github.com/IntelRealSense/librealsense/blob/master/scripts/realsense-hid-ubuntu-bionic-master.patch

landersson commented 5 years ago

realsense-viewer on my Ubuntu 18.04 PC says: Firmware Version: 05.10.13.00

landersson commented 5 years ago

In order to end up with a kernel that is going to work well on the NanoPI M4 SOC, I assume I will have to use a custom kernel that have proper support for this board, such as:

https://github.com/friendlyarm/kernel-rockchip

This is based on Linux 4.4, so I guess I have to apply the realsense patches to this kernel manually.

RealSenseCustomerSupport commented 5 years ago

@landersson Yes, you're right. You need apply the patch manually. Any update please let us know. Thanks!

RealSenseCustomerSupport commented 5 years ago

@landersson Any update? Thanks!

landersson commented 5 years ago

Hi, I just applied the patches to the standard NanoPI M4 RockChip kernel (4.4.154)... I had to fix up a few of the changes manually. After booting with the patched kernel, rs-enumerate-devices finds my D435i and realsense-viewer works pretty well too. I do get a lot of "Incomplete video frame detected" warnings if I try to use the depth stream in all but the lowest resolutions/frame rates. realsense-viewer also seem to be using up a fair bit of CPU even when no streams are enabled.

I haven't had time to investigate and reduce the number of incomplete frames... any suggestions are welcome. Color seem to be able to run as much higher resolutions without getting incomplete frames... Is the color stream being transferred as compressed video, while depth is uncompressed? Or is it more likely that CPU bottlenecks are causing incomplete depth frames?

RealSenseCustomerSupport commented 5 years ago

@landersson Glad to see the progress! What's the resolution and fps of depth when you see the incomplete video frame detected? Any other streaming at the same time when you get incomplete frames?

@oiramario Any progress from your side? Thanks!

RealSenseCustomerSupport commented 5 years ago

@landersson @oiramario Any update? Thanks!

landersson commented 5 years ago

Regardless of whether the color stream is running or not, the depth stream in only running with out incomplete frames in the lowest resolution.

I have run out of time to work more on this for the moment, but we will probably get back to this at a later stage. As far as I'm concerned, please feel free to close this issue. I will open up a new issue related to NanoPI M4 depth stream issues later on if I'm unable to resolve the incomplete frame problems myself.

RealSenseCustomerSupport commented 5 years ago

@oiramario If no other questions, will close the ticket accordingly. Thanks!

oiramario commented 5 years ago

@RealSenseCustomerSupport I'm back! I'll into this next week and feed back to you.