Closed jomiham closed 1 year ago
MartyG-RealSense
commented
1 year ago Hi @jomiham Which method did you use to install the librealsense SDK, please? If you built it from source code with CMake, have you patched the kernel? And if you have patched it, did you use the patch script called patch-realsense-ubuntu-lts-hwe.sh as this script is used for patching kernel 5.15 on Ubuntu 22.04 Jammy.
jomiham
commented
1 year ago Hi @MartyG-RealSense,
The sdk etc is installed with apt-get install (but I saw now that librealsense2-udev-rules is behind for some reason):
librealsense2-dbg/jammy,now 2.53.1-0~realsense0.8251 amd64 [installed]
librealsense2-dev/jammy,now 2.53.1-0~realsense0.8251 amd64 [installed]
librealsense2-dkms/jammy,now 1.3.19-0ubuntu1 all [installed]
librealsense2-gl/jammy,now 2.53.1-0~realsense0.8251 amd64 [installed]
librealsense2-net/jammy,now 2.53.1-0~realsense0.8251 amd64 [installed,automatic]
librealsense2-udev-rules/now 2.51.1-0~realsense0.7526 amd64 [installed,upgradable to: 2.53.1-0~realsense0.8251]
librealsense2-utils/jammy,now 2.53.1-0~realsense0.8251 amd64 [installed]
librealsense2/jammy,now 2.53.1-0~realsense0.8251 amd64 [installed,automatic]Okay, so the kernel patch should be already bundled within the packages of 2.53.1. Thanks very much for the information.
The udev rules handle devices in Ubuntu.
If the problem occurs after hardware reset as the issue title indicates, this could be because a camera is sometimes not re-detected at the start of the second half of a reset after it has been disconnected in the first half of the reset. This could be because of a USB related problem.
jomiham
commented
1 year ago If the problem occurs after hardware reset as the issue title indicates, this could be because a camera is sometimes not re-detected after it has been disconnected in the first half of a reset.
Yes, my guess is also that it is somehow related to the reconnect sequence/timing after the reset. From the stacktrace, it looks like the rs2_config_resolve() call internally uses device_hub::wait_for_device() so I would expect it to return once the device had reconnected, but maybe there is a better way of "polling" for when the device is available again?
MartyG-RealSense
commented
1 year ago It is possible to check if a camera is currently active ("busy"), as discussed at https://github.com/IntelRealSense/librealsense/issues/2240
Alternatively, you can listen for disconnection and reconnection events. So if a reconnection event is detected then that would indicate that the camera is likely available.
C++ https://github.com/IntelRealSense/librealsense/issues/931
Python https://github.com/IntelRealSense/librealsense/issues/4212
It is good practice to have a sleep period for a certain time after a hardware reset before initiating a subsequent instruction in order to ensure that the camera has completed the reset and is ready. I see that you are already doing this. A hardware reset should take 2 to 3 seconds to complete from start to finish. I believe that 2 seconds would be represented by a period of 2000 and 3 seconds by 3000.
jomiham
commented
1 year ago This could be because of a USB related problem.
Possibly, we sometimes get warnings in syslog (more details at the bottom of the main post), but I have seen several runs with SIGSEGV core dumps even when there are no such USB related warnings as well.
And interestingly, I basically only see the warnings (and the error codes from rs2_config_resolve() when we wait close to 500ms (which is slightly longer than the camera usually takes to re-appear on the USB bus)
MartyG-RealSense
commented
1 year ago Typically, rs2_delete_device() is not used with hardware reset. When a hardware reset is initiated and the camera disconnects, it will be removed from the list of attached devices upon disconnection anyway.
jomiham
commented
1 year ago Typically, rs2_delete_device() is not used with hardware reset. When a hardware reset is initiated and the camera disconnects, it will be removed from the list of attached devices upon disconnection anyway.
ok. I have removed the rs2_delete_device() calls after reset but still got intermittent SIGSEGVs.
I assume it doesn't matter which device we reset (depth/color in this case) since it will reset the entire camera?
MartyG-RealSense
commented
1 year ago hardware_reset resets the entire camera and not individual sensors, yes.
MartyG-RealSense
commented
1 year ago Unless there is a problem with camera activation that occurs consistently then performing a hardware reset after initial enabling of the pipeline is likely to be unnecessary. Plus, if the sensors have been successfully defined and gotten then the pipeline is already proven to be accessible.
jomiham
commented
1 year ago Unless there is a problem with camera activation that occurs consistently then performing a hardware reset after initial enabling of the pipeline is likely to be unnecessary. Plus, if the sensors have been successfully defined and gotten then the pipeline is already proven to be accessible.
I believe the initial restart is more to ensure that we have full control of what settings etc are used. But, unfortunately, we sometimes have intermittent hanging/freezing that requires us to restart the cameras on-demand as well so we need to be able to do this regardless.
MartyG-RealSense
commented
1 year ago If the entire port is reset with a general non-RealSense Ubuntu bash script instead of resetting the camera device specifically then it is not necessary for the camera to be detectable in order to complete the reset. An example of such a script is at https://github.com/IntelRealSense/librealsense/issues/8393
jomiham
commented
1 year ago If the entire port is reset with a general non-RealSense Ubuntu bash script instead of resetting the camera device specifically then it is not necessary for the camera to be detectable in order to complete the reset. An example of such a script is at #8393
I'm not sure I follow what you mean with "complete the reset"..? Wouldn't there still be a sensitive period while starting up regardless of how the reset is done (i.e. we still need to know when it is safe to start polling)?
jomiham
commented
1 year ago Alternatively, you can listen for disconnection and reconnection events. So if a reconnection event is detected then that would indicate that the camera is likely available.
C++ #931
Python #4212
Yes a callback event when the device is available would be great, but is there an actively maintained java wrapper? I found a reference to an android wrapper but I'm not sure if that would work for a generic java environment?
jomiham
commented
1 year ago So far, the only (temporary) workaround I have is to wait a "really long time" after resetting to ensure that we don't try to access it during this sensitive restart time. I have seen the odd SIGSEGV even when waiting for 1-2 seconds and I'm guessing it could be even longer in cold temperatures etc.
The drawback is that the system is quite time sensitive so we don't want to wait any longer than we have to (especially when restarting while running). But since waiting too short can crash the whole system, I need to have quite a bit of margin.
I assume the expected behaviour for rs2_config_resolve() / rs2_config_can_resolve() in the cases I get SIGSEGVs is to simply return an error code? I have seen both No device connected and Couldn't resolve requests errors so far (when it doesn't crash of course).
In that case, we can go back to waiting for e.g. 1s and then try/retry resolving until it succeeds.
Would it be optimistic to hope for such a bugfix in the near future?
MartyG-RealSense
commented
1 year ago What I mean by "complete the reset" is that the camera hardware reset process depends on the camera being able to be detected again once it has been disconnected, otherwise the reset process will fail at the halfway point.
When an Ubuntu bash script is used to reset the port in general instead of a specific device attached to the port, it can be assumed that if the camera is able to be detected after the port reset then it should be able to be detected within a couple of seconds. To demonstrate this, you could launch the RealSense Viewer tool and then plug the camera in and observe how long it takes for the camera to appear in the Viewer's options side-panel.
So a safe period to wait after running the Ubuntu port reset script before trying to access the camera may be 3 seconds.
Regarding a Java wrapper, there is a non-Android one that a RealSense user contributed at the link below but it is now several years old. Also, this wrapper is not part of the official RealSense SDK and is only available at the below link's 'forked' version of the SDK which will be hugely out of date compared to the current official SDK version.
https://github.com/edwinRNDR/librealsense/tree/master/wrappers/java
This wrapper was hardly ever used by RealSense users, who developed in Java via the Android wrapper instead.
You may be able to put an exception catch mechanism in your code so that if an error occurs, the program catches it but keeps running instead of crashing. An example of such a mechanism in Java code is at https://github.com/IntelRealSense/librealsense/issues/3295
extern "C"
JNIEXPORT void JNICALL
Java_com_example_realsense_1app_MainActivity_stopStreaming(JNIEnv *env, jobject instance) {
try {
if (!streaming)
return;
streaming = false;
if (frame_thread.joinable())
frame_thread.join();
}
catch (const std::exception& ex)
{
jclass jcls = env->FindClass("java/lang/Exception");
env->ThrowNew(jcls, ex.what());
}What I mean by "complete the reset" is that the camera hardware reset process depends on the camera being able to be detected again once it has been disconnected, otherwise the reset process will fail at the halfway point.
When an Ubuntu bash script is used to reset the port in general instead of a specific device attached to the port, it can be assumed that if the camera is able to be detected after the port reset then it should be able to be detected within a couple of seconds. To demonstrate this, you could launch the RealSense Viewer tool and then plug the camera in and observe how long it takes for the camera to appear in the Viewer's options side-panel.
So a safe period to wait after running the Ubuntu port reset script before trying to access the camera may be 3 seconds.
Sorry if I am being slow here but looking through that discussion, the author used the usb reset script to simulate an USB error and then hardware_reset() + wait to recover from it?
You may be able to put an exception catch mechanism in your code so that if an error occurs, the program catches it but keeps running instead of crashing. An example of such a mechanism in Java code is at https://github.com/IntelRealSense/librealsense/issues/3295
extern "C" JNIEXPORT void JNICALL Java_com_example_realsense_1app_MainActivity_stopStreaming(JNIEnv *env, jobject instance) { try { if (!streaming) return; streaming = false; if (frame_thread.joinable()) frame_thread.join(); } catch (const std::exception& ex) { jclass jcls = env->FindClass("java/lang/Exception"); env->ThrowNew(jcls, ex.what()); }
This might work if we had a JNI C layer, but the native calls are handled by JNA and from what I can find it does not allow catching SEGV signals.
MartyG-RealSense
commented
1 year ago In general, Ubuntu port reset scripts have been used by RealSense users to reset the camera in situations where hardware_reset was not a suitable solution in their particular project. Only a very small number have used it though, with the majority opting for hardware_reset instead.
Using a USB reset script is also discussed at https://github.com/IntelRealSense/librealsense/issues/8274#issuecomment-770196280
An alternative to using rs2_config_resolve may be to instead use rs2_pipeline_get_active_profile to determine whether the pipeline is active. It returns a valid result only when the pipeline is active - between calls to start() and stop(). If the camera is reset with hardware_reset after the pipeline has been started then the pipeline should continue to be open so long as the camera reconnects within 5 seconds of disconnection.
As you installed librealsense from packages, there should not be the timing issues that can occur if the SDK is built from source code with the RSUSB Backend.
As far as I am aware, temperature does not have a bearing on the time taken to complete a reset.
The camera should operate normally so long as its internal temperature in degrees C is above 0 and less than a maximum of 42. Whilst the camera can operate with an internal temperature greater than 42 degrees C, problems may begin to manifest and above 60 degrees C, the firmware driver's laser safety mechanism will shut the laser off. The laser will also be shut off if the internal temperature falls below zero.
jomiham
commented
1 year ago An alternative to using rs2_config_resolve may be to instead use rs2_pipeline_get_active_profile to determine whether the pipeline is active. It returns a valid result only when the pipeline is active - between calls to start() and stop().
In my case, I cannot make any assumptions on the camera's state when starting up (which is why we reset it).
If the camera is reset with hardware_reset after the pipeline has been started then the pipeline should continue to be open so long as the camera reconnects within 5 seconds of disconnection.
Where is the pipeline state preserved if the camera is reset? If the program (that uses the SDK to communicate with the camera) is restarted and then resets the camera, is there some other state in kernel drivers or similar that I should be aware of?
MartyG-RealSense
commented
1 year ago A pipeline has frames delivered to it by the source that is providing the frames (such as a live camera or a pre-recorded bag file), and if there are no new frames received after 5 seconds then the pipeline time-outs, typically resulting in the crashing of a program with the error message RuntimeError: Frame didn't arrive within 5000.
Once a pipeline has been started then it can continue to be open for this time period if the camera stops delivering frames whilst it waits for a new frame, but the pipeline may close after a period of 5 seconds without arrival of frames has passed. If new frames arrive within 5 seconds of the last received frame - for example, because the camera has reconnected - then the pipeline continues from the point that it left off when a frame was last received.
MartyG-RealSense
commented
1 year ago Hi @jomiham Do you have an update about this case that you can provide, please? Thanks!
jomiham
commented
1 year ago Hi @MartyG-RealSense No real update unfortunately. We are trying a longer wait time after the reset as a workaround but still hoping for a fix to the underlying SIGSEGV issue.
MartyG-RealSense
commented
1 year ago In regard to the problem seeming to manifest after upgrading from 2.30.0, there was a case at https://github.com/IntelRealSense/librealsense/issues/8154#issuecomment-764980206 where a RealSense user upgraded to SDK version 2.41.0 and was getting segfaults whenever they stopped the stream, whilst the problem did not occur in version 2.40.0.
MartyG-RealSense
commented
1 year ago Hi @jomiham Do you require further assistance with this case, please? Thanks!
jomiham
commented
1 year ago No, I think we have to settle for the "long wait" workaround until the bug is fixed. Thank you @MartyG-RealSense!
Will this issue will be updated when/if the underlying bug is fixed (that causes a SIGSEGV instead of returning an error)?
MartyG-RealSense
commented
1 year ago It appears that you are the only RealSense user who has reported using the rs2_config_resolve() instruction during the history of this support forum. A bug-fix would therefore likely be considered to be a low development priority unfortunately, especially as you are also using Java (a language that the RealSense SDK does not officially support with a wrapper).
MartyG-RealSense
commented
1 year ago Case closed due to no further assistance required at this time.
Issue Description
I am migrating a system from an older setup with ubuntu 18.04 to a new with 22.04 and am having intermittent trouble with the librealsense integration.
The system uses two D435:s to get camera and depth feeds and the setup process is basically this:
rs2_hardware_reset()+rs2_delete_device()(to get to a known state)What I'm seeing is that often, in step 7
rs2_config_resolve()will fail with an error code. I assume this is some timing issue after the reset and usually everything starts working after one or two automatic retries.However, sometimes one of the retries will throw a SIGSEGV and core dump. (Note: For the sample program below, I only have one camera so it is not due to multiple cameras)
From what I can see, it seems that the problem is on this line https://github.com/IntelRealSense/librealsense/blob/v2.53.1/src/ds5/ds5-device.cpp#L733
since
all_device_infosdoes not contain any device withmi==0(se stack trace below).Changing how long I wait after reset seems to have an effect and 500ms is arbitrary based on how quickly the camera usually pops back up on the USB bus, but given that reset time could vary with temperatures etc. I need to find a "safe" solution.
What am I missing here? It feels like the system has happened on a timing edge case that triggered a bug. But apart from the SIGSEGV bug, I would really appreciate any hint if there is a better way of doing the setup that will ensure that the system always starts in the same state every time without risking timing issues? On Ubuntu 18, we used SDK v2.30 and I am starting to suspect that we have had the same timing issue there, but with different symptoms since the code that triggers the SIGSEGV seems to be added after v2.30...
Core dump stack trace
Sample
I don't know if it helps, but this is the basic sample program that sometimes recreates the core dump (maybe one in 10/20 runs):
A successful run can look like this:
And a SIGSEGV like this:
/var/log/syslog
If everything runs without any errors, I get a clean reset in
/var/log/syslog:But when I get an error code back from
rs2_config_resolve()(regardless of if it will SIGSEGV or not), I see some extra warnings: