search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.53k stars 4.81k forks source link

Possible to set custom FPS of 10? #5328

Closed kafan1986 closed 4 years ago

kafan1986 commented 4 years ago
Required Info  
Camera Model { D415}
Firmware Version {05.11.15.00}
Operating System & Version {Android}
Platform {Arm-Odroid}
SDK Version { 2.30.0 }
Language {C++/Java }
Segment {others }

I tried running colour(1280x720x15fps) and depth stream(1280x720x15fps) together on android using Odroid N2 SBC and after around 62 minutes, the stream gives Frame didn't arrive error. The system throws error sooner when the color stream is set at 1920x1080x15fps resolution. If it helps can I set the FPS to lower value? I would prefer something around 10 rather than available option of 6, which I think is too less for my use case.

Also, can I reset the camera if it starts giving me error and re-run with same config? I am not able to hardware reset it properly. Currently only reboot seems to allow the rerunning of camera with same config.

I have posted the code and adb logs below.

The issue looks related to https://github.com/IntelRealSense/librealsense/issues/4091

I truly need some answers to make the stream stable. My last question did not grant a single answer. That's why I am opening a new issue. Desperate times, desperate measures.

CODE:

private void stream() throws Exception {

        mPipeline = new Pipeline();

        Config config = new Config();
        try(DeviceList devices = mRsContext.queryDevices()) {
            if (devices.getDeviceCount() == 0) {
                return;
            }

            try (Device device = devices.createDevice(0)) {
                if(device == null){
                    Log.e(TAG, "failed to create device");
                    return;
                }

                if (!device.isInAdvancedMode()){
                    device.toggleAdvancedMode(true);
                    if (CommonConstants.getInstance().DEBUG) {
                        Log.d(TAG, "Device not in advance mode. Set to advance mode now!");
                    }
                } else {
                    if (CommonConstants.getInstance().DEBUG) {
                        Log.d(TAG, "Device already in advance mode");
                    }
                }

                try {
                    byte[] presetData = getByteArrayFromResource(FacialProcessingApp.getContext(), R.raw.high_res_high_density_preset, "high_res_high_density_preset.json");
                    device.loadPresetFromJson(presetData);
                    Log.d(TAG, "Preset Set!");
                } catch (IOException e) {
                    Log.e(TAG, "failed to set preset, failed to open preset file, error: " + e.getMessage());
                } catch (Exception e) {
                    Log.e(TAG, "failed to set preset, error: " + e.getMessage());
                }

                config.disableAllStreams();
                config.enableStream(StreamType.COLOR, 0, CommonConstants.getInstance().colourStreamWidth, CommonConstants.getInstance().colourStreamHeight, StreamFormat.YUYV, CommonConstants.getInstance().depthCamFPS);
                config.enableStream(StreamType.DEPTH,0,CommonConstants.getInstance().depthStreamWidth,CommonConstants.getInstance().depthStreamHeight,StreamFormat.Z16,CommonConstants.getInstance().depthCamFPS);
            }
        }

        mPipeline.start(config);

        int frameCounter  = 0;
        while (!mStreamingThread.isInterrupted())
        {
            if (depthCameraConnected) {
                try {
                    try (FrameReleaser fr = new FrameReleaser()) {
                        FrameSet frames = mPipeline.waitForFrames(DEFAULT_TIMEOUT).releaseWith(fr);
                        mIsStreaming = true;
                        frameCounter++;
                        if (frameCounter <= 30) {
                            continue;
                        } else {
                            frameCounter = 100;
                        }

                        FrameSet processedSet = frames.applyFilter(mDecimationFilter).releaseWith(fr)
                                .applyFilter(mTemporalFilter).releaseWith(fr)
                                .applyFilter(mHoleFillingFilter).releaseWith(fr);

                        Frame depth = processedSet.first(StreamType.DEPTH).releaseWith(fr);
                        DepthFrame depthFrame = depth.as(Extension.DEPTH_FRAME);
                        depthFrame.releaseWith(fr);

                        Frame colour = processedSet.first(StreamType.COLOR).releaseWith(fr);
                        VideoFrame colourFrame = colour.as(Extension.VIDEO_FRAME);
                        colourFrame.releaseWith(fr);

                        LibraryLoader.getInstance().getPeopleCountingInterface().SetDepthCamValues(depthFrame,colourFrame);

                        synchronized (Streamer.this) {
                            mCameraCombinedFrameChain[mChainIdx] = new CameraCombinedFrame(colourFrame, depthFrame);
                            mCameraFrameReady = true;
                            Streamer.this.notify();
                        }

                    }
                } catch (Exception e) {
                    Log.e(TAG, "streaming, error: " + e.getMessage());
                    e.printStackTrace();
                }
            }
        }

        mPipeline.stop();
        mPipeline.close();
        mPipeline = null;
    }

ADB LOGS (When the error appeared):

2019-11-27 07:24:25.503 26820-26974/com.test.app W/librs: usb_request_queue returned error, endpoint: 132 error: Out of memory, number: 12
2019-11-27 07:24:25.503 26820-26974/com.test.app E/librs: failed to submit UVC request, error: -11
2019-11-27 07:24:25.546 26820-26971/com.test.app W/librs: usb_request_queue returned error, endpoint: 130 error: Out of memory, number: 12
2019-11-27 07:24:25.546 26820-26971/com.test.app E/librs: failed to submit UVC request, error: -11
2019-11-27 07:24:25.557 26820-26974/com.test.app W/librs: usb_request_queue returned error, endpoint: 132 error: Out of memory, number: 12
2019-11-27 07:24:25.557 26820-26974/com.test.app E/librs: failed to submit UVC request, error: -11
2019-11-27 07:24:26.179 26820-26834/com.test.app I/sense.retailap: Background concurrent copying GC freed 315(63KB) AllocSpace objects, 5(8MB) LOS objects, 49% free, 10MB/21MB, paused 2.424ms total 464.110ms
2019-11-27 07:24:26.433 26820-26974/com.test.app E/librs: uvc streamer watchdog triggered on endpoint: 132
2019-11-27 07:24:26.433 26820-26974/com.test.app I/librs: USB pipe 132 reset successfully
2019-11-27 07:24:46.406 3068-3161/? D/MDnsDS: MDnsSdListener::Monitor poll timed out
2019-11-27 07:24:46.406 3068-3161/? D/MDnsDS: Going to poll with pollCount 1
2019-11-27 07:24:54.488 26820-26966/com.test.app W/librs: control_transfer returned error, index: 768, error: Broken pipe, number: 32
2019-11-27 07:27:01.564 26820-26966/com.test.app W/librs: control_transfer returned error, index: 768, error: Broken pipe, number: 32
2019-11-27 07:27:01.628 26820-26966/com.test.app I/chatty: uid=10062(com.test.app) Thread-3 identical 1 line
2019-11-27 07:27:01.688 26820-26966/com.test.app W/librs: control_transfer returned error, index: 768, error: Broken pipe, number: 32
2019-11-27 07:29:26.546 26820-26971/com.test.app W/librs: usb_request_queue returned error, endpoint: 130 error: Out of memory, number: 12
2019-11-27 07:29:26.546 26820-26971/com.test.app E/librs: failed to submit UVC request, error: -11
2019-11-27 07:29:27.554 26820-26971/com.test.app E/librs: uvc streamer watchdog triggered on endpoint: 130
2019-11-27 07:29:27.554 26820-26971/com.test.app I/librs: USB pipe 130 reset successfully
2019-11-27 07:29:29.896 26820-26953/com.test.app E/MRApp_1.1 :Streamer: streaming, error: Frame didn't arrived within 3000
MartyG-RealSense commented 4 years ago

Though 10 FPS is not officially supported, a workaround for a custom FPS that has been tried is to run at 30 FPS and only capture every nth frame in order to simulate the desired frame rate. The link below details a user who used this method.

https://github.com/IntelRealSense/librealsense/issues/3169

Alternatively, if you have errors at 6 FPS then using 15 FPS, which is officially supported, will often make the errors disappear.

A hardware reset can be performed automatically within scripting with the hardware_reset() instruction.

https://github.com/IntelRealSense/librealsense/issues/2161#issuecomment-408779150

kafan1986 commented 4 years ago

@MartyG-RealSense @matkatz Can you confirm whether the hardware reset will solve this issue? My previous experience somehow made me believe that when this error is encountered then the particular configuration gets locked for further use, till a system reboot is made.

MartyG-RealSense commented 4 years ago

One of the Intel guys on this forum can give a better answer about hardware_reset() than I can. If you find that a hardware reset does not work for you though, an alternative approach that can be tried is a USB reset.

https://github.com/IntelRealSense/librealsense/issues/4113#issuecomment-513649319

kafan1986 commented 4 years ago

@MartyG-RealSense Currently I have written some code to perform hardware reset when it encounters "frame not received" error. I am testing with colour stream at 1920x1080. The thing during my first testing I noticed the frame gets stuck after few minutes. Basically each gives me same colour frame (haven't check the depth frame) again and again, without throwing any "frame not received" error. Is their any id that can be compared with the last received colour frame, to check whether it changes or not, so to that I can trigger hardware reset.

Also during such session, I kept getting following logs on adb. I did not see such logs when working with 720p colour stream.

2019-11-28 10:50:29.448 4407-4443/com.random.app:activity E/librs: Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:50:30.489 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:43.059 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 98 lines
2019-11-28 10:50:43.199 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:43.336 4407-4443/com.random.app:activity W/librs: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:50:43.345 4407-4443/com.random.app:activity E/librs: Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:50:44.392 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:47.217 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 22 lines
2019-11-28 10:50:47.345 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:47.485 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:47.626 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 1 line
2019-11-28 10:50:47.772 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:47.904 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:56.866 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 71 lines
2019-11-28 10:50:57.008 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:50:57.112 4407-4443/com.random.app:activity W/librs: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:50:57.129 4407-4443/com.random.app:activity E/librs: Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:50:58.169 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:10.781 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 98 lines
2019-11-28 10:51:10.915 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:11.055 4407-4443/com.random.app:activity W/librs: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:51:11.057 4407-4443/com.random.app:activity E/librs: Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:51:12.098 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:17.112 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 40 lines
2019-11-28 10:51:17.253 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:17.391 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:17.664 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 2 lines
2019-11-28 10:51:17.772 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:17.909 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:24.541 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 52 lines
2019-11-28 10:51:24.684 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:24.808 4407-4443/com.random.app:activity W/librs: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:51:24.820 4407-4443/com.random.app:activity E/librs: Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:51:25.842 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:38.498 4407-4444/com.random.app:activity I/chatty: uid=10062(com.random.app) Thread-6 identical 98 lines
2019-11-28 10:51:38.632 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
2019-11-28 10:51:38.730 4407-4443/com.random.app:activity W/librs: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:51:38.734 4407-4443/com.random.app:activity E/librs: Error during time_diff_keeper polling: set_xu(ctrl=1) failed! Last Error: Success
2019-11-28 10:51:39.774 4407-4444/com.random.app:activity W/librs: control_transfer returned error, index: 768, error: Device or resource busy, number: 16
MartyG-RealSense commented 4 years ago

I researched your new question very carefully. In regard to detecting a problem as a condition for reset, I wonder if you could use try_wait_for frames. This returns a False value when timeout occurs instead of throwing an exception.

https://github.com/IntelRealSense/librealsense/issues/2422#issuecomment-423254709

kafan1986 commented 4 years ago

@MartyG-RealSense Another question, currently I am using the D415 module, the one without case. In the original hardware(cased one), their is heatsink plate touching the back of the vision processor board on two chips: Vision D4 processor and ISP. ISP remains cool to touch and D4 heatsup after around an hour of operation, my guess around 60-65 degree celcius, I haven't checked the temperature through API, as currently there are none on android, maybe I can call it through the JNI. So the question is does it needs heatsink for long operation? I am intending to use it indoors 24x7 at ambient room temperature. Same query for the camera module, specially the region behind the laser projector.

MartyG-RealSense commented 4 years ago

Assuming a glitch does not occur in the computing device and its USB or its OS during a long-run, the 400 Series cameras can run indefinitely as long as the cameras remain within recommended temperature tolerances (35 degrees C maximum officially in the data sheet document, though in the real world you may get away with 42 degrees before problems manifest). So it is important to draw heat away from the camera board.

The data sheet says "The Vision Processor D4 Board should be screw mounted on to a heat sink or a heat dissipating structure element using screw forks on Board. Thermal conductive tape (electrically non-conductive) should cover the entire back side area (non-component side) of the ASIC Board for thermal transfer onto heat sink or heat dissipating structure element".

The data sheet can be obtained as a PDF from the link below. Page 82 describes recommendations for designing a heat sink for the depth module.

https://dev.intelrealsense.com/docs/intel-realsense-d400-series-product-family-datasheet?elq_cid=1206596&erpm_id=1567221

kafan1986 commented 4 years ago

I researched your new question very carefully. In regard to detecting a problem as a condition for reset, I wonder if you could use try_wait_for frames. This returns a False value when timeout occurs instead of throwing an exception.

#2422 (comment)

The android api does not have a method that exposes try_wait_for_frames. Looks like I need to call it via JNI. If the bandwidth permits, the team should expose rest of the APIs on Android in future.

Also, the colour camera is unstable on Android, and resetting the device/usb hub feels more like a hack for a device of such calibre, which is designed to run 24x7. Also I am not sure about effectiveness of this device reset approach, although I will test this soon and report back.

In any case, I hope Intel looks into this and sorts out underlying issue. The problem is that the previous threads that reported the issue, acknowledges the problem but mentions about cluelessness about the underlying cause of major "Out of Memory" issue. Maybe the team can think about bringing in some additional help from some expert.

Also, is there a chance that using infrared camera stream instead of color camera would improve stability on Android? Is that path worth exploring? Given I am already unable to use 1920x1080 resolution with color camera for stability issues, I can then use 1280x720 infrared camera stream, rather than same resolution stream from color camera, if it helps it in making it stable. In such share some links on how to remove laser projector pattern from the infrared camera.

MartyG-RealSense commented 4 years ago

Thanks for the detailed feedback! If you post your feature request about API exposition as a separate issue on this forum then it will enable the RealSense team to tag it specifically as a feature request and track it if they believe that it is something that they might be able to add at a future date.

If a member of the RealSense user community is able to write their own code to address a particular issue and would like it to be incorporated into the SDK, they are welcome to submit their code as a 'pull request' (PR) so that the RealSense development team can look at it and try to merge it into the SDK.

https://github.com/IntelRealSense/librealsense/pulls

I do not have the specialist knowledge about the Android wrapper to comment reliability on whether a certain tactic with it will work, so hopefully one of the Intel RealSense team members on this forum will be able to address your questions about using the IR sensor. Good luck!

kafan1986 commented 4 years ago

@MartyG-RealSense Thanks for your help. I am not sure whether you are part of intel team or rather an independent open source contributor. But atleast you replied on this thread, when no one else did.

The request for 10 fps was due to the fact that D415 is practically unusable on android, inspite of being supported atleast for last 1 year. I believed that lowering fps on android would help but it did not. Even 6 fps stream eventually faces the issue, not to mention that 6 fps brings additional problem of blurring which, may be corrected to an extent by lowering exposure time. Not sure how to go about that using Android API.

One would believe a product launched by Intel would have better support and more robustness, specially when it is already more than 2 years the product was introduced. But looks like it is not, probably because it is not intel's core line of products. I have seen better support for product launched by much smaller companies or young startups. Only shame is Intel launched their product at a very reasonable price. Intel needs to think whether they want this product to be regarded as just another DYI project for enthusiast or want to provide it for actual production environment.

Intel now keeps adding new line of products like "i" series etc. Lets hope someone from intel goes after actual product stability, before adding more half supported and unstable products. Sorry for the rant, I have literally pulled my hairs for last 20 days fixing and trying different approaches and posted on threads with my findings. The least you can expect is a reply from someone from engineering team that they are looking for a solution, but I don't see any activity from the developer team. They conveniently avoid replying to questions that they don't have answers too, keeping users in total dark. I would open new thread regarding that.

MartyG-RealSense commented 4 years ago

@kafan1986 Thanks very much for your further feedback. I am not an Intel RealSense employee. As an end-user myself, I do very much understand your frustration

An Intel RealSense team member responds to every case at least once, in order to be able to close it after checking if the user still requires assistance. There is no predictable schedule for when this first response occurs though. I respond myself if I can offer at least an educated opinion / best guess, but there are inevitably gaps in my knowledge that mean that some cases are best responded to by RealSense team members with specialist knowledge. Sometimes RealSense team members also need to refer questions to colleagues with specialist knowledge.

Other than the custom FPS approach discussed earlier about capturing every "nth" frame, an alternative approach that some RealSense users have used is to use a timer. One user developed a RealSense ROS node that is controlled by a 'rate'. The link below provides an explanation about 'Rates' in ROS.

https://roboticsbackend.com/ros-rate-roscpy-roscpp/

Another user set up a timer in QT instead of using ROS.

If you are aiming for 10 FPS, the best built-in compromise for avoiding errors that typically occur at 6 FPS is to use the 15 FPS speed.

Computer vision is becoming a key sector for Intel as fields such as AI and autonomous vehicles mature, and RealSense will have a presence at the forthcoming CES 2020 show at the start of January.

Intel aims for RealSense products to run on as many different hardware and software platforms as possible and so it is very open and non-proprietary, minimizing the number of pre-requisites that need to be installed in order for it to work. Inevitably when attempting to "be everywhere and on everything", there are cases where problems occur that require some extra work to resolve.

The RealSense team try to address issues as quickly as possible, though some issues require a greater amount of time and development steps (e.g gradually building towards a solution over multiple SDK releases). A fix may be especially slow to arrive if it has to be built into the firmware driver instead of the SDK, due to the need for the firmware to be rock-solid reliable before a new version is released to avoid "bricking" cameras.

Again, I truly understand your criticisms, and I appreciate that you provide constructive and well expressed feedback.