Android support for t265

[TheBricktop]

I already know that there is some work being done at it 5213 I have some questions though : What is the main reason the tracking camera does not work on Android devices while other d400 devices work fine? Is there any specific thing that needs to be patched or is there some conflicts in the firmware ? This feature is increasingly more vital for my reasearch so I'm really interested in making it finally work for android devices.

[eoffermann]

I'm chomping at the bit, too - I'm working on a project that'll be hosted on Android once the t265 is supported there and at the moment it's a mess of bad workarounds to let me work until that happens.

The short answer though is that the t265 bears little resemblance to the D4xx family. They're very different beasts. The d4xx series are basically special purpose USB cameras. You plug them in, you get images. If you need to do something with those images, you use librealsense, pull the frames into OpenCV or both. For t265, you're not talking to a simple UVC camera - you're talking to a Movidius chip. Intel used libtm as an interface to get the stream of positional data from it and libtm is built for Windows and Linux.

That's really all there is to it - they're making the t265 behave like a native device on Android. Once 5213 lands, I think we'll be able to use it on Android, MacOS, etc. (It looks like it's currently possible to get it working yourself if you switch to a dev branch and apply the 5213 PR to it. It looks like people are still uncovering conflicts during testing though.)

[TheBricktop]

okay then, il try to build the dev branch and try if it works.

[radfordi]

My understanding is that now that #5213 has been merged, that there shouldn't be anything significant blocking getting T265 working on Android. We are not currently testing on Android, so we can't confirm that, but we'd be happy to hear your feedback if you have tried the current development branch there.

[neilyoung]

@TheBricktop Let us know please. I tried 2 or 3 weeks ago to no avail. This lead me to the question, if there is still a rooted device required. This was denied, but I didn't have the time to double check it.

For me it did not work, but it might be that it was a 2.31 or 2.30 development, when I tried last time.

[radfordi]

FWIW, @neilyoung, we no longer need to use libusb on Android, so a rooted device shouldn't be required.

[neilyoung]

@radfordi Yes, thanks. This is my state of knowledge too. Maybe if @TheBricktop signals success, I will try again. That would really be a big point, if that would work.

[neilyoung]

@radfordi OK, for me the situation is unchanged to what I have seen a couple of weeks ago: Android not working.

I'm having here not the most modern device, a Google Nexus 6P with Android 8.1. I compiled the sample and did run the "camera" app. The camera is connected via a USB-C/USB-3.1 cable with the Android device. But as far as I can see from the logs, there is a problem with USB (sic!)

getUsbDevice: failed to locate USB device, VID: 0x8086, PID: 0x0000

2020-01-22 19:30:14.582 14876-14876/? I/zygote64: Late-enabling -Xcheck:jni
2020-01-22 19:30:14.689 14876-14876/com.intel.realsense.camera W/ActivityThread: Application com.intel.realsense.camera is waiting for the debugger on port 8100...
2020-01-22 19:30:14.699 14876-14876/com.intel.realsense.camera I/System.out: Sending WAIT chunk
2020-01-22 19:30:21.224 14876-14883/com.intel.realsense.camera I/zygote64: Debugger is active
2020-01-22 19:30:21.421 14876-14876/com.intel.realsense.camera I/System.out: Debugger has connected
2020-01-22 19:30:21.421 14876-14876/com.intel.realsense.camera I/System.out: waiting for debugger to settle...
2020-01-22 19:30:21.622 14876-14876/com.intel.realsense.camera I/System.out: waiting for debugger to settle...
2020-01-22 19:30:22.632 14876-14876/com.intel.realsense.camera I/chatty: uid=10117(com.intel.realsense.camera) identical 5 lines
2020-01-22 19:30:22.832 14876-14876/com.intel.realsense.camera I/System.out: waiting for debugger to settle...
2020-01-22 19:30:23.034 14876-14876/com.intel.realsense.camera I/System.out: debugger has settled (1439)
2020-01-22 19:30:26.504 14876-14876/com.intel.realsense.camera D/librs: Librealsense VERSION: 2.32.1
2020-01-22 19:30:26.617 14876-14876/com.intel.realsense.camera W/librs UsbUtilities: getUsbDevice: failed to locate USB device, VID: 0x8086, PID: 0x0000
2020-01-22 19:30:26.617 14876-14967/com.intel.realsense.camera I/librs MessagesHandler: handleMessage: realsense device attached
2020-01-22 19:30:26.617 14876-14967/com.intel.realsense.camera I/librs Enumerator: notifyOnAttach
2020-01-22 19:30:26.646 14876-14968/com.intel.realsense.camera D/OpenGLRenderer: HWUI GL Pipeline
2020-01-22 19:30:26.702 14876-14968/com.intel.realsense.camera I/Adreno: QUALCOMM build                   : 8e59954, I0be83d0d26
    Build Date                       : 09/22/17
    OpenGL ES Shader Compiler Version: EV031.21.02.00
    Local Branch                     : O17A
    Remote Branch                    : 
    Remote Branch                    : 
    Reconstruct Branch               : 
2020-01-22 19:30:26.703 14876-14968/com.intel.realsense.camera D/vndksupport: Loading /vendor/lib64/hw/gralloc.msm8994.so from current namespace instead of sphal namespace.
2020-01-22 19:30:32.830 14876-14968/com.intel.realsense.camera I/Adreno: PFP: 0x00000000, ME: 0x00000000
2020-01-22 19:30:32.844 14876-14968/com.intel.realsense.camera I/zygote64: android::hardware::configstore::V1_0::ISurfaceFlingerConfigs::hasWideColorDisplay retrieved: 0
2020-01-22 19:30:32.846 14876-14968/com.intel.realsense.camera I/OpenGLRenderer: Initialized EGL, version 1.4
2020-01-22 19:30:32.846 14876-14968/com.intel.realsense.camera D/OpenGLRenderer: Swap behavior 2
2020-01-22 19:30:32.869 14876-14968/com.intel.realsense.camera D/vndksupport: Loading /vendor/lib64/hw/android.hardware.graphics.mapper@2.0-impl.so from current namespace instead of sphal namespace.
2020-01-22 19:30:32.870 14876-14968/com.intel.realsense.camera D/vndksupport: Loading /vendor/lib64/hw/gralloc.msm8994.so from current namespace instead of sphal namespace.
2020-01-22 19:30:32.923 14876-14876/com.intel.realsense.camera I/Choreographer: Skipped 372 frames!  The application may be doing too much work on its main thread.

[radfordi]

@neilyoung, Whatever you are running above is filtering on vendor ID 0x8086, but T265 uses 0x8087. Not only that, we need to find the Movidius device at some point which uses neither.

I would suggest trying a simple all C++ command line program like rs-pose to start with to verify that the basics work.

[neilyoung]

Hmm. Ok. I was running the project from the Android branch of the SDK. There are several apps, one of them is called "camera"

[superjayman]

How about Unity3D Android implementation? Will the unity package be updated for Android support? or how will it work?

[neilyoung]

There is a file /librealsense/wrappers/android/librealsense/src/main/java/com/intel/realsense/librealsense/UsbUtilities.java, which obviously contains the hardcoded 0x8086 tag. I changed it and recompiled everything to no avail so far.

[neilyoung]

OK, I managed to adapt a sample so that it looks for 0x8087. Result is the same "No device connected".

2020-01-23 12:03:43.939 15976-15976/com.example.realsense_java_example W/librs UsbUtilities: getUsbDevice: failed to locate USB device, VID: 0x8087, PID: 0x0000
2020-01-23 12:03:43.940 15976-15976/com.example.realsense_java_example D/librs: Librealsense VERSION: 2.32.1
2020-01-23 12:03:43.940 15976-16132/com.example.realsense_java_example I/librs MessagesHandler: handleMessage: realsense device attached
2020-01-23 12:03:43.940 15976-16132/com.example.realsense_java_example I/librs Enumerator: notifyOnAttach
2020-01-23 12:03:43.954 15976-16133/com.example.realsense_java_example D/librs: Librealsense VERSION: 2.32.1
2020-01-23 12:03:43.990 15976-16135/com.example.realsense_java_example D/OpenGLRenderer: HWUI GL Pipeline
2020-01-23 12:03:44.058 15976-16135/com.example.realsense_java_example I/Adreno: QUALCOMM build                   : 8e59954, I0be83d0d26
    Build Date                       : 09/22/17
    OpenGL ES Shader Compiler Version: EV031.21.02.00
    Local Branch                     : O17A
    Remote Branch                    : 
    Remote Branch                    : 
    Reconstruct Branch               : 
2020-01-23 12:03:44.059 15976-16135/com.example.realsense_java_example D/vndksupport: Loading /vendor/lib64/hw/gralloc.msm8994.so from current namespace instead of sphal namespace.
2020-01-23 12:03:52.611 15976-16135/com.example.realsense_java_example I/Adreno: PFP: 0x00000000, ME: 0x00000000
2020-01-23 12:03:52.634 15976-16135/com.example.realsense_java_example I/zygote64: android::hardware::configstore::V1_0::ISurfaceFlingerConfigs::hasWideColorDisplay retrieved: 0
2020-01-23 12:03:52.636 15976-16135/com.example.realsense_java_example I/OpenGLRenderer: Initialized EGL, version 1.4
2020-01-23 12:03:52.636 15976-16135/com.example.realsense_java_example D/OpenGLRenderer: Swap behavior 2
2020-01-23 12:03:52.674 15976-16135/com.example.realsense_java_example D/vndksupport: Loading /vendor/lib64/hw/android.hardware.graphics.mapper@2.0-impl.so from current namespace instead of sphal namespace.
2020-01-23 12:03:52.675 15976-16135/com.example.realsense_java_example D/vndksupport: Loading /vendor/lib64/hw/gralloc.msm8994.so from current namespace instead of sphal namespace.
2020-01-23 12:04:02.653 15976-16133/com.example.realsense_java_example W/System.err: java.lang.RuntimeException: No device connected
2020-01-23 12:04:02.654 15976-16133/com.example.realsense_java_example W/System.err:     at com.intel.realsense.librealsense.Pipeline.nStart(Native Method)
2020-01-23 12:04:02.655 15976-16133/com.example.realsense_java_example W/System.err:     at com.intel.realsense.librealsense.Pipeline.start(Pipeline.java:10)
2020-01-23 12:04:02.657 15976-16133/com.example.realsense_java_example W/System.err:     at com.example.realsense_java_example.MainActivity.stream(MainActivity.java:92)
2020-01-23 12:04:02.657 15976-16133/com.example.realsense_java_example W/System.err:     at com.example.realsense_java_example.MainActivity.access$100(MainActivity.java:22)
2020-01-23 12:04:02.658 15976-16133/com.example.realsense_java_example W/System.err:     at com.example.realsense_java_example.MainActivity$2.run(MainActivity.java:82)
2020-01-23 12:04:02.658 15976-16133/com.example.realsense_java_example W/System.err:     at java.lang.Thread.run(Thread.java:764)

[neilyoung]

I think it doesn't matter, which device is attempted to be resolved. The attachment of a T.265 is detected, the attach notification is sent up, the stream is attempted to be started. But this fails.

2020-01-23 12:03:43.940 15976-16132/com.example.realsense_java_example I/librs MessagesHandler: handleMessage: realsense device attached
2020-01-23 12:03:43.940 15976-16132/com.example.realsense_java_example I/librs Enumerator: notifyOnAttach

[TheBricktop]

@neilyoung are you testing the dev branch?

[neilyoung]

Yepp

[TheBricktop]

Goddamit

[radfordi]

@neilyoung, if you look in the main jni/ directory, you can see that, for example, we only wrap VideoStreamProfile, and not MotionStreamProfile, etc. The Android Java wrappers look like they haven't been updated for T265 in any way. This is why I suggest testing the C++ interface not the Java one. See this example that uses the C++ interface to librealsense. You should be able to trivially modify this function and make it specifically look for T265s.

[neilyoung]

@radfordi Thanks Jim. Maybe if I have some spare time. Right now Android is not prio 1 here.

Thanks anyway for the update.

[radfordi]

Actually you don't even need to touch the code. If you run that example you can see, by the number of devices it prints (with T265 connected and disconnected), if it's working.

[neilyoung]

@radfordi Right, but in any event I would need the Java bindings.

[neilyoung]

@radfordi I must be missing something. I have changed the JNI function to

extern "C" JNIEXPORT jint JNICALL
Java_com_example_realsense_1native_1example_MainActivity_nGetCamerasCountFromJNI(JNIEnv *env, jclass type) {
    rs2::context ctx;
    return ctx.query_all_sensors().size();
}

Camera connected, attach notification comes, but the sensor list is empty.

[smartynenko]

OK, I managed to adapt a sample so that it looks for 0x8087. Result is the same "No device connected".

My camera reports that VendorId is actually 0x03E7. I can connect using this VendorId, but not sure how to open open a stream and start getting pose data

[radfordi]

The T265 uses a chip made by Movidius (0x03E7). When librealsense detects it, it will upload the firmware to the T265 and then the vendor will change to Intel (0x8087).

[smartynenko]

It seems to be going back and forth depending on what you connect last. I connected camera to my laptop, ran realsense-viewer and camera vendor id changed to Intel. Then I reconnected it to android device and it went back to 0x03E7. Is there anything in android build that makes camera switch vendor id, or this is due to the fact that I build out of development branch?

[radfordi]

Correct, @smartynenko. This is by design. Each time the device is plugged in, the firmware needs to be uploaded to the device. When this happens the vendorid/productid will change.

My understanding is that Android support should mostly be in place after our recent large refactor of the USB subsystem in librealsense, but that it has never been tested.

[smartynenko]

there is downloadFirmware() task in build.gradle file for camera example which will download firmware files for d4xx and sr3xx cameras, similarly camera app will check if attached device is one of those two series before attempt to update. can librealsense reset t265 to its default version? (I can't see to be able to locate fw update code in the library)

[radfordi]

Each backend (mf, rsusb, v4l) has a hook for booting T265. It looks like the android backend uses the rsusb one.

[smartynenko]

I updated few makefiles to include files from src\tm2 directory, not android fails to upload fw binaries into camera (usbhost.c: usb_device_bulk_transfer -> "bulk_transfer returned error, endpoint: 1, error: Out of memory, number: 12"). Any suggestions about how to fix this?

[smartynenko]

made some progress with boot sending updated firmware, device started reporting itself as " Intel(R) RealSense(TM) Tracking Camera T265 Vendor Id: 8087 Product Id: b37"

Now I am stuck at getting response from DEV_SET_LOW_POWER_MODE request. Any suggestions about what to check?

2020-03-02 13:21:49.048 31212-31212/com.intel.realsense.multicam D/librs: tm2_info::create 0xa5438dac 2020-03-02 13:21:49.048 31212-31212/com.intel.realsense.multicam D/librs: tm2_info::get_device_data 0xa5438dac 2020-03-02 13:21:49.049 31212-31212/com.intel.realsense.multicam D/librs: Creating a T265 device 2020-03-02 13:22:22.141 31212-31212/com.intel.realsense.multicam D/librs: Successfully opened and claimed interface 0 2020-03-02 13:23:30.953 31212-31212/com.intel.realsense.multicam D/librs: Sending message DEV_SET_LOW_POWER_MODE length 8 2020-03-02 13:26:37.030 31212-31212/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 8 2020-03-02 13:28:37.309 31212-31212/com.intel.realsense.multicam W/librs: bulk_transfer returned error, endpoint: 130, error: No such device, number: 19

[radfordi]

Great progress getting the device to boot, @smartynenko. The LOW_POWER command is the first one that gets sent, so I suspect whatever you're running into is a more general problem. Nonetheless, you might try commenting out the sending of that command as the device should work without it, IIRC.

[smartynenko]

that helped. previous error was cased by me trying to send/receive messages separately few secs between each other, once i removed break points most of the messages went through

2020-03-02 15:37:03.797 7554-7584/com.intel.realsense.multicam D/librs: tm2_info::create 0xa5438dac 2020-03-02 15:37:03.797 7554-7584/com.intel.realsense.multicam D/librs: tm2_info::get_device_data 0xa5438dac 2020-03-02 15:37:03.797 7554-7584/com.intel.realsense.multicam D/librs: Creating a T265 device 2020-03-02 15:37:03.797 7554-7584/com.intel.realsense.multicam D/librs: Successfully opened and claimed interface 0 2020-03-02 15:37:03.797 7554-7584/com.intel.realsense.multicam D/librs: Sending message DEV_SET_LOW_POWER_MODE length 8 2020-03-02 15:37:03.797 7554-7584/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 8 2020-03-02 15:37:03.798 7554-7584/com.intel.realsense.multicam D/librs: Received DEV_SET_LOW_POWER_MODE with length 8 2020-03-02 15:37:03.798 7554-7584/com.intel.realsense.multicam D/librs: Sending message DEV_GET_DEVICE_INFO length 6 2020-03-02 15:37:03.798 7554-7584/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 54 2020-03-02 15:37:03.798 7554-7584/com.intel.realsense.multicam D/librs: Received DEV_GET_DEVICE_INFO with length 54 2020-03-02 15:37:03.798 7554-7584/com.intel.realsense.multicam I/librs: Serial: 944222110076 2020-03-02 15:37:03.798 7554-7584/com.intel.realsense.multicam I/librs: Connection type: 2.1 2020-03-02 15:37:03.798 7554-7584/com.intel.realsense.multicam I/librs: Firmware version: 0.2.0.926 2020-03-02 15:37:03.799 7554-7584/com.intel.realsense.multicam D/librs: Making a sensor 0x976ff810 2020-03-02 15:37:03.799 7554-7584/com.intel.realsense.multicam D/librs: Sending message DEV_LOG_CONTROL length 8 2020-03-02 15:37:03.799 7554-7584/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 8 2020-03-02 15:37:03.799 7554-7584/com.intel.realsense.multicam D/librs: Received DEV_LOG_CONTROL with length 8 2020-03-02 15:37:03.800 7554-7618/com.intel.realsense.multicam D/librs: Sending message DEV_GET_AND_CLEAR_EVENT_LOG length 6 2020-03-02 15:37:03.800 7554-7618/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 32776 2020-03-02 15:37:03.800 7554-7618/com.intel.realsense.multicam W/librs: bulk_transfer returned error, endpoint: 130, error: Success, number: 0 2020-03-02 15:37:03.800 7554-7618/com.intel.realsense.multicam E/librs: Received 0 but header was 200 bytes (max_response_size was 32776) 2020-03-02 15:37:03.800 7554-7618/com.intel.realsense.multicam I/librs: Got bad response, stopping log_poll 2020-03-02 15:37:03.800 7554-7584/com.intel.realsense.multicam D/librs: Sending message DEV_GET_TEMPERATURE length 6 2020-03-02 15:37:03.800 7554-7584/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 1024 2020-03-02 15:37:03.802 7554-7584/com.intel.realsense.multicam W/librs: bulk_transfer returned error, endpoint: 130, error: No data available, number: 61 2020-03-02 15:37:03.803 7554-7584/com.intel.realsense.multicam E/librs: Bulk response error RS2_USB_STATUS_OTHER 2020-03-02 15:37:03.803 7554-7619/com.intel.realsense.multicam D/librs: Sending message DEV_GET_TIME length 6 2020-03-02 15:37:03.803 7554-7619/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 16 2020-03-02 15:37:03.803 7554-7619/com.intel.realsense.multicam D/librs: Received DEV_GET_TIME with length 16 2020-03-02 15:37:03.803 7554-7584/com.intel.realsense.multicam D/librs: Sending message DEV_GET_TEMPERATURE length 6 2020-03-02 15:37:03.803 7554-7584/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 1024 2020-03-02 15:37:03.804 7554-7584/com.intel.realsense.multicam W/librs: bulk_transfer returned error, endpoint: 130, error: No data available, number: 61 2020-03-02 15:37:03.804 7554-7584/com.intel.realsense.multicam E/librs: Bulk response error RS2_USB_STATUS_OTHER 2020-03-02 15:37:03.804 7554-7619/com.intel.realsense.multicam D/librs: Sending message DEV_GET_TIME length 6 2020-03-02 15:37:03.804 7554-7619/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 16 2020-03-02 15:37:03.804 7554-7619/com.intel.realsense.multicam D/librs: Received DEV_GET_TIME with length 16 2020-03-02 15:37:03.804 7554-7584/com.intel.realsense.multicam D/librs: Sending message DEV_GET_SUPPORTED_RAW_STREAMS length 6 2020-03-02 15:37:03.805 7554-7584/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 1024 2020-03-02 15:37:03.805 7554-7584/com.intel.realsense.multicam W/librs: bulk_transfer returned error, endpoint: 130, error: No data available, number: 61 2020-03-02 15:37:03.805 7554-7584/com.intel.realsense.multicam E/librs: Bulk response error RS2_USB_STATUS_OTHER

I am trying to adapt multicam example to get pose date by changing enableStream call

//config.enableStream(StreamType.DEPTH, 640, 480); config.enableStream(StreamType.POSE, StreamFormat.POSE); Pipeline pipe = mPipelines.get(i); // try statement needed here to release resources allocated by the Pipeline:start() method try (PipelineProfile pp = pipe.start(config)) {}

code compiles and starts successfully, but then files to star a pipe with new config

I am missing anything else? thanks

[radfordi]

I assume you're starting with this function:

    private void configAndStart() throws Exception {
        for(int i = 0; i < mPipelines.size(); i++) {
            try (Config config = new Config()) {
                config.enableDevice(deviceList.createDevice(i).getInfo(CameraInfo.SERIAL_NUMBER));
                config.enableStream(StreamType.DEPTH, 640, 480);
                mPipelines.get(i).start(config);
            }
        }
    }

I would guess you'd need to do something like this:

    private void configAndStart() throws Exception {
            int i = 0;
            try (Config config = new Config()) {
                config.enableStream(StreamType.DEPTH, 640, 480);
                mPipelines.get(i).start(config);
            }
            i=1;
            try (Config config = new Config()) {
                config.enableStream(StreamType.POSE);
                mPipelines.get(i).start(config);
            }
    }

In particular note that I removed the usage of enableDevice as it's broken for T265 until #5615 or something like it gets merged.

I don't see why DEV_GET_SUPPORTED_RAW_STREAMS wouldn't return successfully.

[smartynenko]

It fails in a different way. Should camera still work if w/o making call to enableDevice ?

2020-03-02 16:38:35.857 14372-14372/com.intel.realsense.multicam D/librs: Sending message DEV_GET_EXTRINSICS length 7 2020-03-02 16:38:35.858 14372-14372/com.intel.realsense.multicam D/librs: Receiving message with max_response_size 57 2020-03-02 16:38:35.858 14372-14372/com.intel.realsense.multicam D/librs: Received DEV_GET_EXTRINSICS with length 57 2020-03-02 16:38:35.859 14372-14372/com.intel.realsense.multicam D/librs: Stopping sensor 2020-03-02 16:38:35.900 14372-14372/com.intel.realsense.multicam D/librs: Destroying T265 device 2020-03-02 16:38:35.902 14372-14372/com.intel.realsense.multicam D/librs: Iterate available devices - config can not be resolved. Couldn't resolve requests 2020-03-02 16:38:35.902 14372-14372/com.intel.realsense.multicam I/librs: tm_boot: received list with 1 device(s) 2020-03-02 16:38:35.902 14372-14372/com.intel.realsense.multicam I/librs: Found device, vid=32903 2020-03-02 16:38:35.902 14372-14372/com.intel.realsense.multicam I/librs: Picked 1/1 devices 2020-03-02 16:38:35.902 14372-14372/com.intel.realsense.multicam D/librs: tm2_info created for 0xa5437b3c 2020-03-02 16:38:35.902 14372-14372/com.intel.realsense.multicam D/librs: tm2_info::get_device_data 0xa5437b3c 2020-03-02 16:38:35.902 14372-14372/com.intel.realsense.multicam D/librs: tm2_info destroyed for 0xa5437b3c 2020-03-02 16:38:40.903 14372-14372/com.intel.realsense.multicam D/librs: tm2_info destroyed for 0xa5437aec 2020-03-02 16:38:40.905 14372-14372/com.intel.realsense.multicam E/[SM]LibrsMulticamExampl: configAndStart, depth pipeline start error: No device connected 2020-03-02 16:38:40.905 14372-14372/com.intel.realsense.multicam E/[SM]LibrsMulticamExampl: configAndStart, pose pipeline start error: Index: 1, Size: 1 2020-03-02 16:38:40.906 14372-14372/com.intel.realsense.multicam D/[SM]LibrsMulticamExampl: start: streaming started successfully 2020-03-02 16:38:40.947 14372-14447/com.intel.realsense.multicam D/OpenGLRenderer: HWUI GL Pipeline 2020-03-02 16:38:40.964 14372-14372/com.intel.realsense.multicam I/librs Enumerator: onReceive: USB_CONTROL_PERMISSION 2020-03-02 16:38:40.966 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getDevices: requesting list of USB devices matching vId=0x8086 2020-03-02 16:38:40.968 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getUsbDevices: Enumerating Usb Devices... 2020-03-02 16:38:40.971 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getUsbDevices: Unexpected USB Device, VID: 0x8087, PID: 0x0b37 2020-03-02 16:38:40.973 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getUsbDevices: failed to locate USB device, VID: 0x8086, PID: 0x0000 2020-03-02 16:38:40.975 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getDevices: requesting list of USB devices matching vId=0x8087 2020-03-02 16:38:40.978 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getUsbDevices: Enumerating Usb Devices... 2020-03-02 16:38:40.979 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getDevices: requesting list of USB devices matching vId=0x03e7 2020-03-02 16:38:40.982 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getUsbDevices: Enumerating Usb Devices... 2020-03-02 16:38:40.985 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getUsbDevices: Unexpected USB Device, VID: 0x8087, PID: 0x0b37 2020-03-02 16:38:40.987 14372-14372/com.intel.realsense.multicam W/[SM] librs UsbUtilities: getUsbDevices: failed to locate USB device, VID: 0x03e7, PID: 0x2150 2020-03-02 16:38:40.987 14372-14372/com.intel.realsense.multicam D/[SM] librs UsbUtilities: hasUsbPermission 2020-03-02 16:38:40.989 14372-14394/com.intel.realsense.multicam I/librs MessagesHandler: handleMessage: realsense device attached 2020-03-02 16:38:40.989 14372-14394/com.intel.realsense.multicam I/librs Enumerator: notifyOnAttach 2020-03-02 16:38:40.990 14372-14372/com.intel.realsense.multicam W/librs: wait_for_frames cannot be called before start() 2020-03-02 16:38:40.990 14372-14372/com.intel.realsense.multicam E/[SM]LibrsMulticamExampl: streaming, error: wait_for_frames cannot be called before start() 2020-03-02 16:38:40.992 14372-14394/com.intel.realsense.multicam D/librs DeviceWatcher: Enumerating USB devices... 2020-03-02 16:38:40.992 14372-14394/com.intel.realsense.multicam D/librs DeviceWatcher: USB device[Name: Intel(R) RealSense(TM) Tracking Camera T265 Vendor Id: 8087 Product Id: b37]

[smartynenko]

I am getting some traction with pose data by adopting native example. So far main glitches are

1) Unreliable camera detection (most likely caused either by USB, or my hacky approach to make original code work) 2) Intermittent pipeline failures caused by frame delays. @radfordi do you have any insights into what could cause those?

10x!

[radfordi]

I recommend going "native" and using the C++ instead of the Java API, if that's what you mean.

Unreliable camera detection (most likely caused either by USB

This could easily be code, but I'd also check your cable.

Intermittent pipeline failures caused by frame delays.

With respect to frame delays, I'm not sure sure what you mean. Can you provide more any more details?

[smartynenko]

correct, I am using c++ instead of Java API to communicate with the camera except for initial bootstrap

Unreliable detection -> it feels like one either needs to wait for some time time or perform additional initialization step before camera can be used

2020-03-02 20:08:15.873 16445-16445/com.example.realsense_native_example I/librs Enumerator: onReceive: android.hardware.usb.action.USB_DEVICE_DETACHED 2020-03-02 20:08:15.874 16445-16466/com.example.realsense_native_example I/librs MessagesHandler: handleMessage: realsense device detached 2020-03-02 20:08:15.874 16445-16466/com.example.realsense_native_example I/librs Enumerator: notifyOnDetach 2020-03-02 20:08:15.874 16445-16445/com.example.realsense_native_example I/librs Enumerator: onReceive: android.hardware.usb.action.USB_DEVICE_ATTACHED 2020-03-02 20:08:15.875 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getDevices: requesting list of USB devices matching vId=0x8086 2020-03-02 20:08:15.882 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Enumerating USB devices... 2020-03-02 20:08:15.882 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getUsbDevices: Enumerating Usb Devices... 2020-03-02 20:08:15.882 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: USB device[Name: Intel(R) RealSense(TM) Tracking Camera T265 Vendor Id: 8087 Product Id: b37] 2020-03-02 20:08:15.882 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Intel Devices: 1, Movidius Devices: 0 2020-03-02 20:08:15.883 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Removing device: /dev/bus/usb/001/002 2020-03-02 20:08:15.883 16445-16466/com.example.realsense_native_example D/librs: RemoveUsbDevice, previous device count: 1 2020-03-02 20:08:15.883 16445-16466/com.example.realsense_native_example D/librs: RemoveUsbDevice, removing device: /dev/bus/usb/001/002, descriptor: 40 2020-03-02 20:08:15.884 16445-16466/com.example.realsense_native_example D/librs: RemoveUsbDevice, current device count: 0 2020-03-02 20:08:15.884 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getUsbDevices: Unexpected USB Device, VID: 0x8087, PID: 0x0b37 2020-03-02 20:08:15.884 16445-16466/com.example.realsense_native_example D/UsbDeviceConnectionJNI: close 2020-03-02 20:08:15.885 16445-16466/com.example.realsense_native_example W/librs: tm_boot: received request with empty list of devices 2020-03-02 20:08:15.885 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Device: /dev/bus/usb/001/002 removed successfully 2020-03-02 20:08:15.886 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getUsbDevices: failed to locate USB device, VID: 0x8086, PID: 0x0000 2020-03-02 20:08:15.887 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getDevices: requesting list of USB devices matching vId=0x8087 2020-03-02 20:08:15.890 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Adding device: /dev/bus/usb/001/003 2020-03-02 20:08:15.890 16445-16466/com.example.realsense_native_example D/librs: AddUsbDevice, previous device count: 0 2020-03-02 20:08:15.890 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getUsbDevices: Enumerating Usb Devices... 2020-03-02 20:08:15.890 16445-16466/com.example.realsense_native_example D/librs: AddUsbDevice, adding device: /dev/bus/usb/001/003, descriptor: 40 2020-03-02 20:08:15.890 16445-16466/com.example.realsense_native_example D/librs: AddUsbDevice, create 2020-03-02 20:08:15.891 16445-16466/com.example.realsense_native_example D/librs: AddUsbDevice, notify 2020-03-02 20:08:15.891 16445-16466/com.example.realsense_native_example D/librs: AddUsbDevice, current device count: 1 2020-03-02 20:08:15.891 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getDevices: requesting list of USB devices matching vId=0x03e7 2020-03-02 20:08:15.891 16445-16466/com.example.realsense_native_example I/librs: tm_boot: received list with 1 device(s) 2020-03-02 20:08:15.891 16445-16466/com.example.realsense_native_example I/librs: Found device, vid=32903 2020-03-02 20:08:15.891 16445-16466/com.example.realsense_native_example I/librs: Picked 1/1 devices 2020-03-02 20:08:15.892 16445-16466/com.example.realsense_native_example D/librs: tm2_info created for 0x978c76ec 2020-03-02 20:08:15.892 16445-16466/com.example.realsense_native_example D/librs: tm2_info destroyed for 0x978c76ec 2020-03-02 20:08:15.892 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Device: /dev/bus/usb/001/003 added successfully 2020-03-02 20:08:15.893 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getUsbDevices: Enumerating Usb Devices... 2020-03-02 20:08:15.895 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getUsbDevices: Unexpected USB Device, VID: 0x8087, PID: 0x0b37 2020-03-02 20:08:15.897 16445-16445/com.example.realsense_native_example W/[SM] librs UsbUtilities: getUsbDevices: failed to locate USB device, VID: 0x03e7, PID: 0x2150 2020-03-02 20:08:15.897 16445-16445/com.example.realsense_native_example D/[SM] librs UsbUtilities: hasUsbPermission 2020-03-02 20:08:15.899 16445-16466/com.example.realsense_native_example I/librs MessagesHandler: handleMessage: realsense device attached 2020-03-02 20:08:15.899 16445-16466/com.example.realsense_native_example I/librs Enumerator: notifyOnAttach 2020-03-02 20:08:15.902 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Enumerating USB devices... 2020-03-02 20:08:15.902 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: USB device[Name: Intel(R) RealSense(TM) Tracking Camera T265 Vendor Id: 8087 Product Id: b37] 2020-03-02 20:08:15.902 16445-16466/com.example.realsense_native_example D/librs DeviceWatcher: Intel Devices: 1, Movidius Devices: 0 2020-03-02 20:08:15.909 16445-16445/com.example.realsense_native_example W/librs: wait_for_frames cannot be called before start() 2020-03-02 20:08:15.910 16445-16445/com.example.realsense_native_example A/libc: /usr/local/google/buildbot/src/android/ndk-release-r21/external/libcxx/../../external/libcxxabi/src/abort_message.cpp:72: abort_message: assertion "terminating with uncaught exception of type rs2::wrong_api_call_sequence_error: wait_for_frames cannot be called before start()" failed 2020-03-02 20:08:15.911 16445-16445/com.example.realsense_native_example A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 16445 (_native_example)

frame delays -> (some times app crashes when delay exceeds certain threshold, I will send longs once I will see it next time)

2020-03-02 20:07:33.843 16106-16153/com.example.realsense_native_example D/librs: CallbackStarted,Pose,196,DispatchedAt,1583197653843.407959 2020-03-02 20:07:33.843 16106-16153/com.example.realsense_native_example D/librs: CallbackStarted,Pose,196,DispatchedAt,1583197653843.628906 2020-03-02 20:07:33.843 16106-16153/com.example.realsense_native_example D/librs: CallbackFinished,Pose,195,DispatchedAt,1.5832e+12 2020-03-02 20:07:33.844 16106-16153/com.example.realsense_native_example D/librs: Frame Callback [Pose#195] overdue. (Duration: 4.00391ms, FPS: 200, Max Duration: 4ms) 2020-03-02 20:07:33.844 16106-16153/com.example.realsense_native_example D/librs: CallbackFinished,Pose,0,DispatchedAt,1.5832e+12

[smartynenko]

Could you check what could be causing below crash conditions? Streaming would work for a few seconds and then abort for no apparent reason

2020-03-07 00:49:42.150 11760-11888/com.example.realsense_native_example D/librs: Received DEV_GET_TIME with length 16 2020-03-07 00:49:42.150 11760-11888/com.example.realsense_native_example D/librs: T265 time synced, host_ns: 1583560126817370880 2020-03-07 00:49:42.658 11760-11888/com.example.realsense_native_example D/librs: Sending message DEV_GET_TIME length 6 2020-03-07 00:49:42.659 11760-11888/com.example.realsense_native_example D/librs: Receiving message with max_response_size 16 2020-03-07 00:49:42.659 11760-11888/com.example.realsense_native_example D/librs: Received DEV_GET_TIME with length 16 2020-03-07 00:49:42.659 11760-11888/com.example.realsense_native_example D/librs: T265 time synced, host_ns: 1583560126817370880

--------- beginning of crash

2020-03-07 00:49:42.799 11760-11760/com.example.realsense_native_example A/libc: /usr/local/google/buildbot/src/android/ndk-release-r21/external/libcxx/../../external/libcxxabi/src/abort_message.cpp:72: abort_message: assertion "terminating with uncaught exception of type rs2::error: Frame didn't arrive within 15000" failed 2020-03-07 00:49:42.814 11760-11760/com.example.realsense_native_example A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 11760 (_native_example)

[b4be1]

Dear @smartynenko, @radfordi, @neilyoung, could you please share an example native project that correctly detects T265 on an Android phone?

I followed your discussions in #4648, #5213, and what I did so far is I built wrappers/android SDK on the development branch using ./gradlew buildRelease. Then I linked the resulting .aar and .so libraries librealsense-2.35.0 to the native_example. However, I am getting exactly the same output as when using the pre-built librealsense-2.34.0:

W/librs UsbUtilities: getUsbDevice: failed to locate USB device, VID: 0x8086, PID: 0x0000

This warning comes from wrappers/android/librealsense/.../UsbUtilities.java, where VendorID is hard-coded. However, I thought that files from src/android should be used instead, that employ the rsusb backend. Does it mean that my build is incorrect? Or should I not use native_example but write some communication code from scratch? Or should I tweak the VendorID?

This is my first Android project, so please excuse my naivety.

[neilyoung]

Sorry, I never was able to make this run, so I abandoned it.

[smartynenko]

native_example in master repo has funky set up. In particular build.gradle references pre-built arr files, so unless you change it to your local output, you will end up banging your head trying to understand why your changes are not making it to the final app. I would suggest to review my pull request # 6011 to see what needs to be changed (just fyi, I have not done any development since 2.33.1 came out, so it might not compile with latest SDK)

[vladHoloArch]

I was able to make the T265 work on android using just Java's api (no jni calls), though the usage is kind of 'primitive' meaning I only use the pose

[neilyoung]

though the usage is kind of 'primitive' meaning I only use the pose

Out of curiosity: What other use of the T265 do you have in mind?

[vladHoloArch]

I meant that I don't address all the available api such as getting the fisheye frame and occupancy map

[smartynenko]

I was able to make the T265 work on android using just Java's api (no jni calls), though the usage is kind of 'primitive' meaning I only use the pose

Glad to hear it worked for you, hopefully there are more features coming. I had troubles getting 6DF data with java api and switched to jni (btw, one of the Intel guys mentioned that they did not expose full API in Java yet). What version of Android you are on?

[smartynenko]

I meant that I don't address all the available api such as getting the fisheye frame and occupancy map

Just FYI, video stream is only available on USB 3, which is not supported on lower end models

[vladHoloArch]

@smartynenko I’m not using the java api intel provided, I address the device like any other usb device and do bulk transfers to and from it’s endpoints. I’m not sure about the android version but I don’t think it makes too much of a difference in this case.

[b4be1]

@smartynenko

native_example in master repo has funky set up. In particular build.gradle references pre-built arr files, so unless you change it to your local output, you will end up banging your head trying to understand why your changes are not making it to the final app. I would suggest to review my pull request # 6011 to see what needs to be changed (just fyi, I have not done any development since 2.33.1 came out, so it might not compile with latest SDK)

Thanks, I moved .aar and .so files by hand and added .aar to dependencies in build.gradle. I also switched off all downloading from maven in build.gradle and put my .so files into build/jni. So, it is similar to your configuration and I am confident that this part is working correctly.

Looking at your pull request, I see that you adjusted UsbUtitilites.java to recognize Intel and Movidius devices. This is the piece that I was missing.

@vladHoloArch

I was able to make the T265 work on android using just Java's api (no jni calls), though the usage is kind of 'primitive' meaning I only use the pose

This would already be sufficient for my application. Is your code publicly available? I saw a link in one of the threads, but the link was not accessible anymore.

@neilyoung Thanks for your reply. I am also considering to switch to Raspberry PI if I don't manage to make Android work till the end of the week.

[b4be1]

@smartynenko I managed to compile and run native_app from your branch, but unfortunately I am getting the following error

D/librs DeviceWatcher: Enumerating USB devices... found USB device[Name: Movidius MA2X5X Vendor Id: 3e7 Product Id: 2150] I/librs: Found a T265 to boot W/librs: bulk_transfer returned error, endpoint: 1, error: Out of memory, number: 12

Do you know a solution? Seems that bulk_transfer is sending too much to fit into the memory.

[smartynenko]

@smartynenko I managed to compile and run native_app from your branch, but unfortunately I am getting the following error

D/librs DeviceWatcher: Enumerating USB devices... found USB device[Name: Movidius MA2X5X Vendor Id: 3e7 Product Id: 2150] I/librs: Found a T265 to boot W/librs: bulk_transfer returned error, endpoint: 1, error: Out of memory, number: 12

Do you know a solution? Seems that bulk_transfer is sending too much to fit into the memory.

Are you rebuilding librealsense2.so as well? I would suggest to look into usbhost.c and usbhost.h, I added condition to check android version, but it was only tested on the phone I had access to

[b4be1]

Yes, I also rebuild librealsense2.so.

Ok, thanks, I will take a look at usbhost.c and usbhost.h.