Closed Mad-Thanos closed 4 years ago
matkatz
commented
5 years ago Hi @GucciPrada That seems to be the same issue that was reported in #3612. I'm still not able to reproduce it on my RK3399 setup and I plan to try some other devices. Did you see this issue on earlier versions of librealsense?
Mad-Thanos
commented
5 years ago Hi,@matkatz thanks for your reply. I just read that similar issue you mentioned. I didn't try earlier version of librealsense or firmware. I'll try it out today. And I'm also keep finding other android devices that support usb3.1/3.2 to try. I'll post any news I get at here.
For additional information, 1.I've tried some android phones that only support usb 2.1(e.g. Oppo Find X/Vivo Nex), the 'capture' example are working nicely, successfully and stably delivering RGBD stream of VGA(640x480)@30Hz. That's quite a nice work from you guys! However I wanna a higher resolution RGBD stream(.i.e 720p or 1080p). Have you or other people successfully made the D415+android deliver 720P or 1080P RGBD stream @30fps? which android device do they use? and the cable?
matkatz
commented
5 years ago I tried to reproduce this issue on Pixel 2 (Android 9), Samsung S8 (Android 9) and FireFly RK3399 (Android 8.1) with no success. I will try to get other platforms next week and hopefully reproduce the issue. As for the high resolution streaming, all of the platforms above were capable to stream 720p depth with 1080p color but with significant frame drops, I get about 20 FPS for the depth stream and 15 FPS for the color.
Mad-Thanos
commented
5 years ago I tried to reproduce this issue on Pixel 2 (Android 9), Samsung S8 (Android 9) and FireFly RK3399 (Android 8.1) with no success. I will try to get other platforms next week and hopefully reproduce the issue. As for the high resolution streaming, all of the platforms above were capable to stream 720p depth with 1080p color but with significant frame drops, I get about 20 FPS for the depth stream and 15 FPS for the color.
Thank you! That is quite informative. What about the USB 3.0 cable or adapter you used? any advice or recommendation?
matkatz
commented
5 years ago I'm not sure what type my cable is it but you can probably test your cable with realsense-viewer over Windows/Linux
Mad-Thanos
commented
5 years ago OK, Thank you! I tested my cable with realsense-viewer on Macbook. I'll test it on Windows later.
Mad-Thanos
commented
5 years ago I tried with my new Android Phone, Honor V20 (another brand of Huawei), the device was enumerated as usb2.1 device but actually the Phone is declared to support USB 3.1 Gen1. And this time, the same 'out-of-memory' problem happened with firmware 5.11.4 & librealsense 2.22.0 or 2.21.0. Here is the log while running firmware 5.11.4+librealsense 2.21.0:
And I then tried to use firmware 5.11.4 + librealsense 2.19.2, it can work correctly, delivering 640x480 RGBD@30Hz, although the device was enumverated as usb 2.1 device. Here is the log:
dreifelix
commented
5 years ago Encountered this issue as well, using D435 + Pixel3 and D435 + Samsung S7 (same results). Both devices are not rooted, and are detected as usb 210. Firmware 5.11.04, librealsense 2.22. Tried both camera and capture apps.
For camera app, it starts with black screen, then the Settings page is displayed after a few seconds. I tried selecting different settings, including the lowest frame rate and resolution and only one active stream, but I still get the Out of memory issue.
Next, I tried building the AAR and created my own app. Same result for both devices. Let me know if you want me to try something else.
See sample log below, if it helps:
matkatz
commented
5 years ago Thanks for the inputs! I'm trying to put my finger on the changes that cause this issue. I understand that it does not exists on 2.19.2 and it does on 2.21.0 What about 2.20.0 (uploaded to Bintray today)?
matkatz
commented
5 years ago There is a change on USB transfer method on v2.21.0 that might cause this issue. I created a branch that partly revert this change: android_bulk_transfer Its purpose is only to verify that this is indeed the cause so it might be very buggy. You can also consume it by changing the bintray repo to: https://dl.bintray.com/intel-realsense/librealsense-dev And specify version 2.23.0.372
dreifelix
commented
5 years ago Happy to share that your branch works! I have tested it on S7 + D435, no errors in the log. Will test it on the Pixel3 on Monday when I have the device with me. It worked with all three streams (color, depth, infrared) turned on simultaneously at low resolution and 15 fps each.
Thanks so much for the fix!
Mad-Thanos
commented
5 years ago Thanks! I'll give it a try and report the results at here.
Mad-Thanos
commented
5 years ago I tried librealsense 2.20.0 + firmware 5.11.04 on Honor V20 android phone, the example capture can work normally now.
Here is the log:
however, I am not sure whether the third to last line in the above log is critical or could be ignored.
Mad-Thanos
commented
5 years ago And the example app capture built with librealsense-dev 2.23.0.372 can also work normally.
Here is the log:
Mad-Thanos
commented
5 years ago Hi @matkatz , Do you have any thoughts about the problem of the USB protocol version? That is, librealsense running on a USB 3.1 compliant android phone recognizes the D415 camera as a USB 2.1 device.
I've checked the cable and the phone using both a windows tool named 'ChipGenius' and the MacOS X System Report App on MacBook Pro.
On windows, I plugged my Honor V20 android phone into the USB 3.2 type-c port of the PC laptop, using the same c-to-c cable that I used to connect the android phone with the D415 camera. And the ChipGenius give me a USB detection result of the Honor V20 phone as follow:
Device Description: USB Composite Device (HUAWEI PCT-AL10)
Equipment Type: Supplier Customization
Protocol version: USB 3.10
Current speed: High Speed
Power consumption: 504mA
USB device ID: VID = 12D1 PID = 107E
Equipment serial number: 5EN0219223001846
Equipment supplier: HUAWEI
Equipment Name: PCT-AL10
Equipment revision: 0299
Host/Master Controller model: UnknownAs you can see, the phone + the cable is recognized as USB 3.1 device.
on MacBook Pro (Retina, 13-inch, Early 2015 version), I got almost the same result:
I think the Speed: Up to 5Gb/sec implies that the phone is connected to USB 3.0 port of my MacBook.
So why it's susb specification- 210 when the D415 is connected to the same phone using the same cable? Am I missing something?
Added: However, in the first post of this issue, I ran librealsense 2.22.0+ capture app on Huawei P20(also support USB3.0). The D415 can be recognized as a USB 3.0 device. But at that time the app ran into the OOM problem. I'll try to build the capture app using librealsense-dev 2.23.0.372, run it on Huawei P20 and check if the D415 is recognized as a USB 3.0 device and if the OOM problem is gone.
matkatz
commented
5 years ago Thanks for the feedback!
I'm testing the library on 4 different platforms:
On all of those systems the camera(D415 FW 5.11.4.0) is identified as USB 3.2. It is possible that the camera will be detected as USB 2 in some cases, but unplug and plug back the camera should solve that.
I'm still not capable to reproduce the memory issue on those platforms, currently I'm working on both directions:
Mad-Thanos
commented
5 years ago I tried to build the example capture with librealsense-dev 2.23.0.372, and ran it on Huawei P20.
something similar to issue #4215 happened. See this screencast video:
You could see that the frames were fitfully updated, the screen got stuck for a while and got updated later again, off and on repeatedly. I could see from the logcat a warning and an error message alternately appeared:
W/librs: Timeout reached waiting for response
E/librs: Cancel request failed: Invalid argumentThe warning log seems to be similar to issue #3612.
But after switching librealsense to v2.20.0, and build the capture again, It worked correctly. It even could deliver 1280x720x30fps RGBD stream, since the D415 was enumerated as USB 3.2 device.
matkatz
commented
5 years ago I was finally able to reproduce the issue and able to see where is it coming from. From a reason that I’m not yet understand, the USB bulk transfer started to fail when the frame resolution size was larger than ~600KB.
To solve that I changed the read buffer size from default hard coded size to a value that is determined according to the current stream resolution.
Reducing the buffer size allows to configure and stream lower resolutions in case of “Out of memory” error.
I also tried to reassemble the frame from few smaller USB payloads but that didn’t work for me (I think it requires isochronous endpoint which is not supported by Android).
BTW, a reboot to the device that the issue was reproduced on (Pixel 2 - Android 9), eliminated the issue.
I will keep looking into this but first I want to apply the fix if it's mitigates the issue.
Once again I will ask you to check my fix before I push it to the development branch.
This is the branch: android-read-length-fix And it is also available in the dev repo (https://dl.bintray.com/intel-realsense/librealsense-dev) on bintray, version: 2.23.0.13
Thanks in advance!
Mad-Thanos
commented
5 years ago That's great! I'd like to try it out later in this day
dreifelix
commented
5 years ago The android-read-length-fix branch fails to stream using the camera sample app, even at lowest resolution at 6 fps when tested on my S7 + D435.
Mad-Thanos
commented
5 years ago I got a similar result to the one from @dreifelix. I was running the example app capture on Huawei P20 + librealsense-dev v2.23.0.13.
Here is the code snippet to config and start the stream:
The critical error in logcat:
2019-06-17 23:22:47.432 26320-27063/com.intel.realsense.capture W/librs: bulk_transfer returned error, endpoint: 130, error: Out of memory, number: 12
2019-06-17 23:22:47.432 26320-27063/com.intel.realsense.capture E/librs: USB transfer failed, the max read buffer size is smaller than the current resolution requires, try to configure lower resolutionThanks again for your help!
@dreifelix, do you see the same behavior with the Pixel? did you try to reboot the phone? Also, I see some strange issues with specific configurations (6 FPS) on my platforms when connecting via USB2. If you get to test it again please try to config only depth 640x480x30 as first step.
I also want to share with you the reason for trying to solve this issue and not revert the backend changes.
This is the performance of v2.23.0 (when it does work (: )
While this is v2.20.0 results:
I'll keep digging...
dreifelix
commented
5 years ago Rebooting helped! :)
In my S7, I am able to stream up to 60 fps when running 2 streams at a time, and up to 30 fps when running all 3 streams simultaneously, at the lowest resolution (424x240 for color, 480x270 for infrared and depth). At 640x480 resolution, only infrared works, up to 30 fps.
I noticed the performance improvement as well. I wasn't able to reach the above frame rates in the android_bulk_transfer branch - the streams were freezing intermittently at 30 fps when running simultaneously.
As you noted, I am unable to run any of the streams at 6 fps at any resolution, but that's fine :) I will try it on the Pixel 3 when I get the chance - i'm just borrowing the device from another project team.
Looking forward to have your changes merged in the next release. Thanks so much for looking into this!
matkatz
commented
5 years ago Thanks @dreifelix I still need to understand what is bringing the phone to this state.
@GucciPrada, I guess reboot didn't help with the P20?
Mad-Thanos
commented
5 years ago Hi @matkatz I've never tried to reboot the phone. I'll try it today.
Mad-Thanos
commented
5 years ago Sorry for this late report!
I've tried the librealsense-dev 2.23.0.13, ran capture on Huawei P20 for 640x480@30fps RGBD.
Before restarting the phone, I got the same result as before: the app cannot receive any frame correctly.
After rebooting the phone, I got a different result: The color stream was streaming correctly. But the depth stream was stuck after several frames were delivered. I pasted the log below:
Then I tried to kill the app several times or tried to keep the D415 plugged into the phone, then reopen the capture app, everything was ok again. That was wired!
And one more trial, after the phone was rebooted, I reinstalled the capture app through Android Studio. Then I got the same error result as the very beginning, i.e. both color and depth stream was stuck while the D415 plugged in. So I tried to reboot the phone and reopen or kill the capture app with the D415 plugged in, sometimes the app could work correctly.
In a word, it's not stable and I didn't find a consistent way to make the app work correctly.
matkatz
commented
5 years ago Thanks @GucciPrada. I don't have a solution for the "Out of memory" state yet. Will update once I'll have some news regarding this issue.
Mad-Thanos
commented
5 years ago OK, thanks!
matkatz
commented
4 years ago @xtrawurst let's continue the conversation on this thread.
I'm trying to revert the bulk transfer changes and use USB requests instead. After removing some locks and extra copies I'm getting the same performance as I got with bulk transfer, hopefully without the OOM error. Since I'm not capable to reproduce that error on my devices I wanted to ask you to help me verify that I'm on the right direction. This is the branch: android-usb-request You can also consume it by changing the bintray repo to: https://dl.bintray.com/intel-realsense/librealsense-dev And specify version 2.24.0.67
rafaelspring
commented
4 years ago @matkatz Great, I'll give it a try and let you know!
rafaelspring
commented
4 years ago @matkatz Confirmed! I no longer see the issue on my Samsung Galaxy Tab S4, even when using Depth @ 1280x720 and RGB @ 1920x1080. Will test on other devices too.
Quick question: What's, roughly, the processing being done CPU-sided on each frame in libRS after the USB transmission completes? At full size capture the frame rate on my S4 is below 30 fps and I'm trying to figure out if it's a) The USB transfer taking a long time, b) CPU-sided processing in libRS or c) CPU-sided processing on our end.
Mad-Thanos
commented
4 years ago @matkatz Glad to hear this. I'll give it a try today.
@xtrawurst May I ask, what frame rate specifically you got at full-size capture(Depth @ 1280x720 and RGB @ 1920x1080)?
matkatz
commented
4 years ago @xtrawurst Thanks! This is good news, except that now I get OOM error on high bandwidth configuration with devices that worked with no issues. So I need to check that now... ):
As for processing, depends on the image format you request, a conversion may be done over the CPU. For example requesting RGB8 requires more CPU than YUYV which is the native format and doesn't requires any processing except a single memcopy.
On the USB transactions I see strange behavior with the color stream on high resolutions (720p & 1080p). In both resolutions the transaction time seems to be longer than the time measured on the depth stream although the resolution, bytes per pixel and frequency are identical for the 720p. Currently with both 720p and 1080p color I get 15 FPS, while depth streams @ ~27-30 FPS
Mad-Thanos
commented
4 years ago Hi @matkatz
I got the same good news as @xtrawurst, that the example app capture running on Huawei P20 device can deliver 1920x1080 Color and 1280x720 Depth both at 15fps:
it can also deliver 640x360 Depth@23-26fps and 1280x720 Color @15fps. The firmware version is still 5.11.04 on my D415 camera. I'll update it to the newest and test it again.
Yes, it's really weird that the color stream ran at a much lower frame rate than the depth stream with the same resolution and pixel size.
matkatz
commented
4 years ago @GucciPrada Thanks for the update, I hope to push this change to the development branch soon, but first I need to understand why my RK3399 started to show OOM error.
ndiremdjian
commented
4 years ago Hi Guys, Not sure if this issue is fully SW related, I was experiencing the same on Sony Xperia but I came to suspect my USB3 cable (I used so far a 2m from nedis). With this cable the capture app last about 27s before crashing with the bulk_transfer error above at 60fps VGA. Lowering frame rate was just delaying the crash (30 fps about 50s etc...)
I then switched for another cable (1m from belkin) and I wasn't able to reproduce the error anymore.
Sounds a bit too magic and witchware but still, if it may help one of you...
Cheers
Mad-Thanos
commented
4 years ago Hi @ndiremdjian Which version of librealsense and firmware did you use? When built with librealsense v2.22.0 and firmware 5.11.4, the capture app, running at VGA@30fps, didn't crash on my phone with a USB3.1 type-c to type-c cable.
And thanks for your tips, I will try some other cables.
ndiremdjian
commented
4 years ago Hi I have the same versions as yours. My initial cable was a 2m 3.1 type c to type c 5Gbps. My second cable was a Thunderbolt 3 USB-C 20Gbps with backward compatibility USB3.1 10Gbps. With the later, no error is showing up even at 60fps VGA with Depth and both IR streams on. Will try to add RGB as well.
Cheers
rafaelspring
commented
4 years ago @matkatz Thanks for the information. Any updates on the Android-USB front?
matkatz
commented
4 years ago I will try to merge #4428 to the development branch before our next release.
rafaelspring
commented
4 years ago @matkatz Thanks
On the USB transactions I see strange behavior with the color stream on high resolutions (720p & 1080p). In both resolutions the transaction time seems to be longer than the time measured on the depth stream although the resolution, bytes per pixel and frequency are identical for the 720p. Currently with both 720p and 1080p color I get 15 FPS, while depth streams @ ~27-30 FPS
Has this been resolved?
matkatz
commented
4 years ago Not yet, currently both implementations shows this behavior.
RealSenseCustomerSupport
commented
4 years ago @GucciPrada @xtrawurst The latest SDK v2.30 has improved android stability according to our testing. Could you please try if you still get any issue? Thanks!
Mad-Thanos
commented
4 years ago Yes, I will try it in the following days. Thank you!
RealSenseCustomerSupport
commented
4 years ago @GucciPrada Thanks! Looking forward to your update.
RealSenseCustomerSupport
commented
4 years ago @GucciPrada @xtrawurst Any update with SDK v2.30? Thanks!
kafan1986
commented
4 years ago @matkatz @RealSenseCustomerSupport I have received same OOM error on SDK 2.30.0 on android 9 (S922x SBC). My colour config is 1920x1080x15fps and depth is 1280x720x15. It works OK for few minutes and then crashes.
I/librs: endpoint 132 read buffer size: 4148224 I/librs: USB pipe 132 reset successfully I/librs: usb_request_queue returned error, endpoint: 132 error: Out of memory, number: 12 E/librs: failed to submit UVC request, error: -11
Mad-Thanos
commented
4 years ago Sorry for this late reply.
I ran the RS Camera App v2.30.0.1174 on two Android Phones, Honor V20 and Huawei P30 Pro.
The app could not open a single DEPTH stream(1280x720@15fps or 30fps) on those phones.
Specifically, when I tried changing to the 15fps configuration on the Settings page, the App showed me a toast message:"Failed to set streaming configuration". Here is the captured log file when I changed the single DEPTH stream from 1280x720@6fps to 1280x720@15fps.
But it could open a single 1280x720@6fps DEPTH stream or RGBD stream(RGB8-1280x720@6fps and DEPTH-1280x720@6fps).
Here is the log.txt while runing the 1280x720@6fps RGBD stream.
I guess the reason is: the connection type is considered as USB 2.1
I might need to try more Android Phones or USB cables.
kafan1986
commented
4 years ago @matkatz @RealSenseCustomerSupport At 15 fps with RGB at 1280x720 and Depth at 1280x720, the camera will throw lots of error and would eventually would stop producing frames, after few minutes. The only way I could make it work stable for longer duration was to set FPS at 6, at 6 Hz no error is thrown for a long time and I could stream RGB at rather highest resolution of 1920x1080 and Depth at 1280x720 simultaneously without crashes for longer period. After around 3-4 hours, I saw the depth frame was getting updated but somehow the RGB stream was stuck on some previous frame. Stats displayed 0 fps for RGB stream. I was able to resume the camera at same 1920x1080 resolution, without any restart or reset by simply going to the settings activity and then going back to preview activity.
Also FYIP, my device is USB 3.0, but the test app displays it as 3.2, not sure if it affects stability in any way. BTW, I am using latest 05.11.15.00 firmware and 2.30.0 SDK.
The error which was thrown after 3-4 hours was this:
2019-12-02 06:45:18.321 7396-7804/com.intel.realsense.camera E/librs: failed to submit UVC request, error: -11
2019-12-02 06:45:20.719 7396-7804/com.intel.realsense.camera E/librs: uvc streamer watchdog triggered on endpoint: 132I really need this to work at 15 Hz stable on android. Do confirm whether solving this issue is high on your priority or not, someone is actively looking into this issue. I need to make some business decision regarding my product design components accordingly.
@rafaelspring Did you ever share your hacked libusb code to Matkatz, which did not face same "out of memory" issue. I saw a previous thread regarding the same.
RealSenseCustomerSupport
commented
4 years ago @GucciPrada As your connection type is USB 2.1, only 6fps is supported for 1280x720. If you want higher fps such as 15fps or 30fps, you need switch to USB3 connection.
@kafan1986 What's your android device when you get the issue with latest v2.30? And could you please provice the full logcat log when the issue happened? Thanks!
Issue Description
short version:
I ran the wrappers/android/examples/capture app on a Android Phone(has a usb 3.1 compatible type-c female interface) with D415 plugged in. But there was only black screen on the UI, which means the text "Connect to a Realsense Camera" was gone but no video data rendered. And at the same time, I found in the logcat of Android studio there reported constantly a plenty of warning messages:
the full log was something as follow:
Click me to expand
``` 2019-05-29 00:01:45.036 27981-28098/com.intel.realsense.capture D/librs DeviceWatcher: Device: /dev/bus/usb/003/002 added successfully 2019-05-29 00:01:49.237 27981-27981/com.intel.realsense.capture I/librs: Found UVC Device vid: id- vid- 8086 pid- ad3 mi- 0 unique_id- /dev/bus/usb/003/002 path- /dev/bus/usb/003/002 susb specification- 320 2019-05-29 00:01:49.237 27981-27981/com.intel.realsense.capture I/librs: Found UVC Device vid: id- vid- 8086 pid- ad3 mi- 3 unique_id- /dev/bus/usb/003/002 path- /dev/bus/usb/003/002 susb specification- 320 2019-05-29 00:01:49.239 27981-27981/com.intel.realsense.capture I/librs: Found UVC Device vid: id- vid- 8086 pid- ad3 mi- 0 unique_id- /dev/bus/usb/003/002 path- /dev/bus/usb/003/002 susb specification- 320 2019-05-29 00:01:49.239 27981-27981/com.intel.realsense.capture I/librs: Found UVC Device vid: id- vid- 8086 pid- ad3 mi- 3 unique_id- /dev/bus/usb/003/002 path- /dev/bus/usb/003/002 susb specification- 320 2019-05-29 00:01:49.241 27981-27981/com.intel.realsense.capture D/librs capture example: try start streaming 2019-05-29 00:01:49.241 27981-27981/com.intel.realsense.capture I/librs: Found UVC Device vid: id- vid- 8086 pid- ad3 mi- 0 unique_id- /dev/bus/usb/003/002 path- /dev/bus/usb/003/002 susb specification- 320 2019-05-29 00:01:49.241 27981-27981/com.intel.realsense.capture I/librs: Found UVC Device vid: id- vid- 8086 pid- ad3 mi- 3 unique_id- /dev/bus/usb/003/002 path- /dev/bus/usb/003/002 susb specification- 320 2019-05-29 00:01:49.276 27981-28475/com.intel.realsense.capture W/librs: bulk_transfer returned error, endpoint: �, error: Out of memory ...... many many repeated warning message as the above line..... ```My Question is: Has anybody ever seen this problem before? What should I do to fix this?
Longer version:
My setup:
"Android Phone Huawei P20 + USB 3.1 Gen 2 C Male to C Male cable + Realsense D415". I've smiply checked the cable, it is compliant with USB 3.1 Gen 2 10/20Gbps 5A 100W. I've also checked the Huawei P20's type c interface by plugging it into my Macbook and checking the 'USB 3.0 Bus' section of System Report of the MacBook. It showed the Speed is up to 5Gbps of "Huawei P20+ the c-to-c cable" combination. I guess that means either Huawei P20 or my Macbook has a USB 3.1 Gen 1 or USB 3.0 interface.
What I did and what I saw:
I downloaded the librealsense-2.22.0.aar and librealsense-2.22.0.zip from dl.bintray.com. After that, I import them into the android project of wrappers/android, making both examples/native-example and examples/capture Modules to depend on them.
Then I modified the examples/native-example's source code to enumerate the device, the sensors and the stream profiles of each sensor. And save the enumerated result to a log file: usb3.2-P20-enumerate-log.txt In a word, I could tell that the highest profiles (1280x720xZ16x30Hz Depth stream and the 1920x1080xRGB8x30Hz Color stream) are available.
In the next step, I build and ran examples/capture on the Huawei P20. After granting the USB permission the capture app, the screen was black and the message “27981-28475/com.intel.realsense.capture W/librs: bulk_transfer returned error, endpoint: �, error: Out of memory” was flooding the Logcat.