search

ifm / libo3d3xx

Driver and utilities for IFM Efector O3D3xx depth cameras
Apache License 2.0
38 stars 39 forks source link

Framegrabber timeout error #145

Open robinbansal08 opened 3 years ago

robinbansal08 commented 3 years ago

I am running my application over the O3D303 (after cross compilation), and getting such framegrabber timeout errors sometimes (occurs out of sudden at a time when my application was running fine and was able to grab previous frames successfully). I tried frame grabber reset but that didn't help. I have to do camera reboot to get it work again as my application is not able to get the images and keep getting frame grabber timeout errors. My camera is set to software triggered and not free run.

Can you please provide any suggestions related to this issue @graugans 

E1029 12:30:41.621506  1317 frame_grabber.cpp:343] Bad ticket: 1001L000000007
1001!
W1029 12:30:41.621618  1317 frame_grabber.cpp:348] Are you software triggering in free run mode?
==> Trigger difference: 2565ms
W1029 12:30:42.615631  1311 frame_grabber.cpp:193] Timeout waiting for image buffer from camera
Timeout waiting for camera!
Restarting FrameGrabber...
I1029 12:30:42.629215  1311 frame_grabber.cpp:72] Camera connection info: ip=192.168.0.19, port=50010
W1029 12:30:42.630647  1317 frame_grabber.cpp:429] Exception: Thread interrupted
I1029 12:30:42.630798  1317 frame_grabber.cpp:432] Framegrabber thread done.
W1029 12:30:43.634541  1311 frame_grabber.cpp:193] Timeout waiting for image buffer from camera
Timeout waiting for camera!
W1029 12:30:43.637190  1319 frame_grabber.cpp:429] Exception: Thread interrupted
I1029 12:30:43.637830  1319 frame_grabber.cpp:432] Framegrabber thread done.

o3d3xx::Camera::Ptr cam_;
o3d3xx::ImageBuffer::Ptr img_;
o3d3xx::FrameGrabber::Ptr fg_;

void init_cam()
{
    cam_ = std::make_shared<o3d3xx::Camera>(camera_IP);
    img_ = std::make_shared<o3d3xx::ImageBuffer>();
    fg_ = std::make_shared<o3d3xx::FrameGrabber>(cam_, o3d3xx::IMG_AMP|o3d3xx::IMG_RDIS);
    cout<<"Cam init done"<<endl;
}

int getImageFromCamera(Mat& image, pcl::PointCloud<pcl::PointXYZI>::Ptr& cloud_xyzi)
{
    fg_->SWTrigger();

    if (! fg_->WaitForFrame(img_.get(), 1000))
    {
        cout << RED<<"Timeout waiting for camera!" << RESET <<std::endl;
        std::cout << "Restarting FrameGrabber..." << std::endl;
        //exit (-1);
        fg_.reset(new o3d3xx::FrameGrabber(cam_, o3d3xx::IMG_AMP|o3d3xx::IMG_RDIS));
        if (! fg_->WaitForFrame(img_.get(), 1000))
        {
            cout << RED<<"Timeout waiting for camera!" << RESET <<std::endl;
            exit (-1);
        }
        return 0;
    }

    pcl::copyPointCloud(*(img_->Cloud()), *cloud_xyzi);
    image = img_->DepthImage();

    return 1;
}
{
    "o3d3xx": {
        "libo3d3xx": "705",
        "Date": "Thu Jun 17 13:02:13 2021",
        "HWInfo": {
            "Connector": "#!02_A300_C40_04246016_008023176",
            "Diagnose": "#!02_D322_C34_04250651_008026824",
            "MiraSerial": "8470-81f7-0e63-0720",
            "MACAddress": "00:02:01:41:3B:B5",
            "Frontend": "#!02_F342_C34_19_00105_008023607",
            "Mainboard": "#!02_M381_C41_03901679_008023690",
            "Illumination": "#!02_I300_001_04201465_008001175"
        },
        "SWVersion": {
            "IFM_Software": "1.71.7444",
            "Main_Application": "O2I_1.20.7428-0-g91c60a74be",
            "IFM_Recovery": "unversioned",
            "Algorithm_Version": "3.0.5",
            "Calibration_Device": "00:02:01:41:3b:b5",
            "Calibration_Version": "1.0.2",
            "Linux": "Linux version 3.14.34-rt31-yocto-standard-00048-gc8112a91aeb2-d",
            "Diagnostic_Controller": "O2I_1.20.7428-0-g91c60a74be"
        },
        "Device": {
            "Name": "New sensor",
            "Description": "",
            "ActiveApplication": "1",
            "PcicTcpPort": "50010",
            "PcicProtocolVersion": "3",
            "IOLogicType": "1",
            "IODebouncing": "true",
            "IOExternApplicationSwitch": "0",
            "SessionTimeout": "30",
            "ServiceReportPassedBuffer": "15",
            "ServiceReportFailedBuffer": "15",
            "ExtrinsicCalibTransX": "0",
            "ExtrinsicCalibTransY": "0",
            "ExtrinsicCalibTransZ": "0",
            "ExtrinsicCalibRotX": "0",
            "ExtrinsicCalibRotY": "0",
            "ExtrinsicCalibRotZ": "0",
            "EvaluationFinishedMinHoldTime": "10",
            "SaveRestoreStatsOnApplSwitch": "true",
            "IPAddressConfig": "0",
            "PasswordActivated": "false",
            "OperatingMode": "1",
            "DeviceType": "1:2",
            "ArticleNumber": "O3D303",
            "ArticleStatus": "AD",
            "UpTime": "25.1308333333333",
            "ImageTimestampReference": "1603975097",
            "TemperatureFront1": "3276.6999999999998",
            "TemperatureFront2": "3276.6999999999998",
            "TemperatureIMX6": "32.812000274658203",
            "TemperatureIllu": "39.200000000000003",
            "PNIODeviceName": "",
            "EthernetFieldBus": "1",
            "EthernetFieldBusEndianness": "0",
            "EnableAcquisitionFinishedPCIC": "false"
        },
        "Net": {
            "MACAddress": "",
            "NetworkSpeed": "0",
            "StaticIPv4Address": "192.168.0.19",
            "StaticIPv4SubNetMask": "255.255.255.0",
            "StaticIPv4Gateway": "192.168.0.201",
            "UseDHCP": "false"
        },
        "Apps": [
            {
                "Name": "Box Detection Application",
                "Description": "",
                "TriggerMode": "2",
                "PcicTcpResultSchema": "{ \"layouter\": \"flexible\", \"format\": { \"dataencoding\": \"ascii\" }, \"elements\": [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"blob\", \"id\": \"normalized_amplitude_image\" }, { \"type\": \"blob\", \"id\": \"distance_image\" }, { \"type\": \"blob\", \"id\": \"x_image\" }, { \"type\": \"blob\", \"id\": \"y_image\" }, { \"type\": \"blob\", \"id\": \"z_image\" }, { \"type\": \"blob\", \"id\": \"confidence_image\" }, { \"type\": \"blob\", \"id\": \"diagnostic_data\" }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "PcicEipResultSchema": "{ \"layouter\": \"flexible\", \"format\": { \"dataencoding\": \"binary\", \"order\": \"big\" }, \"elements\" : [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"records\", \"id\": \"models\", \"elements\": [ { \"type\": \"int16\", \"id\": \"boxFound\" }, { \"type\": \"int16\", \"id\": \"width\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"height\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"length\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"xMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"zMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yawAngle\" }, { \"type\": \"int16\", \"id\": \"qualityLength\" }, { \"type\": \"int16\", \"id\": \"qualityWidth\" }, { \"type\": \"int16\", \"id\": \"qualityHeight\" } ] }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "PcicPnioResultSchema": "{\"layouter\" : \"flexible\", \"format\": { \"dataencoding\": \"binary\", \"order\": \"big\" }, \"elements\" : [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"records\", \"id\": \"models\", \"elements\": [ { \"type\": \"int16\", \"id\": \"boxFound\" }, { \"type\": \"int16\", \"id\": \"width\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"height\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"length\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"xMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"zMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yawAngle\" }, { \"type\": \"int16\", \"id\": \"qualityLength\" }, { \"type\": \"int16\", \"id\": \"qualityWidth\" }, { \"type\": \"int16\", \"id\": \"qualityHeight\" } ] }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "LogicGraph": "{\"IOMap\": {\"OUT1\": \"RFT\",\"OUT2\": \"AQUFIN\"},\"blocks\": {\"B00001\": {\"pos\": {\"x\": 200,\"y\": 200},\"properties\": {},\"type\": \"PIN_EVENT_IMAGE_ACQUISITION_FINISHED\"},\"B00002\": {\"pos\": {\"x\": 200,\"y\": 75},\"properties\": {},\"type\": \"PIN_EVENT_READY_FOR_TRIGGER\"},\"B00003\": {\"pos\": {\"x\": 600,\"y\": 75},\"properties\": {\"pulse_duration\": 0},\"type\": \"DIGITAL_OUT1\"},\"B00005\": {\"pos\": {\"x\": 600,\"y\": 200},\"properties\": {\"pulse_duration\": 0},\"type\": \"DIGITAL_OUT2\"}},\"connectors\": {\"C00000\": {\"dst\": \"B00003\",\"dstEP\": 0,\"src\": \"B00002\",\"srcEP\": 0},\"C00001\": {\"dst\": \"B00005\",\"dstEP\": 0,\"src\": \"B00001\",\"srcEP\": 0}}}",
                "Type": "Camera",
                "Index": "1",
                "Id": "476707713",
                "Imager": {
                    "AutoExposureReferencePointX": "88",
                    "AutoExposureReferencePointY": "66",
                    "AutoExposureReferenceType": "0",
                    "AutoExposureMaxExposureTime": "10000",
                    "AutoExposureReferenceROI": "{\"ROIs\":[{\"id\":0,\"group\":0,\"type\":\"Rect\",\"width\":130,\"height\":100,\"angle\":0,\"center_x\":88,\"center_y\":66}]}",
                    "Channel": "0",
                    "ClippingBottom": "263",
                    "ClippingLeft": "0",
                    "ClippingRight": "351",
                    "ClippingTop": "0",
                    "ContinuousAutoExposure": "false",
                    "EnableAmplitudeCorrection": "true",
                    "EnableFastFrequency": "false",
                    "EnableFilterAmplitudeImage": "false",
                    "EnableFilterDistanceImage": "true",
                    "EnableRectificationAmplitudeImage": "false",
                    "EnableRectificationDistanceImage": "false",
                    "UseSimpleBinning": "false",
                    "ExposureTime": "10000",
                    "ExposureTimeList": "2500;10000",
                    "ExposureTimeRatio": "4",
                    "FrameRate": "5",
                    "MinimumAmplitude": "2",
                    "Resolution": "1",
                    "ClippingCuboid": "{\"XMin\": -3.402823e+38, \"XMax\": 3.402823e+38, \"YMin\": -3.402823e+38, \"YMax\": 3.402823e+38, \"ZMin\": -3.402823e+38, \"ZMax\": 3.402823e+38}",
                    "SpatialFilterType": "1",
                    "SymmetryThreshold": "0",
                    "TemporalFilterType": "0",
                    "ThreeFreqMax2FLineDistPercentage": "80",
                    "ThreeFreqMax3FLineDistPercentage": "80",
                    "TwoFreqMaxLineDistPercentage": "80",
                    "Type": "upto30m_moderate",
                    "MaxAllowedLEDFrameRate": "4.9000000000000004",
                    "SpatialFilter": {
                        "Type": "1",
                        "TypeStr_": "Median Filter",
                        "MaskSize": "1",
                        "MaskSizeStr_": "5x5"
                    },
                    "TemporalFilter": {
                        "Type": "0",
                        "TypeStr_": "Off"
                    }
                }
            }
        ]
    }
}
robinbansal08 commented 3 years ago
graugans commented 3 years ago

@robinbansal08, have you tried to upload an image to the camera? This is what I do read from:

T: 1704.1475 P: 686 C: WARN S: imager M: virtual QByteArray ifm::PersistSnapshotData::getSnapshot() const: snapshot requested but no latest file present 
T: 1704.1489 P: 686 C: ERROR S: imager M: static ifm::ErrorStatus ifm::SnapshotProcessing::Private::convert(const QByteArray&, ifm::DistanceImage&) ( 101043 ) : "Invalid chunk type expected 600 received: 0"  
T: 1704.1493 P: 686 C: WARN S: imager M: virtual ifm::ErrorStatus ifm::SnapshotProcessing::retainSnapshot(const QByteArray&, ifm::SnapshotProcessing::ResolutionMode) Snapshot conversion error: Invalid chunk type expected 600 received: 0

Those SPI errors we do typically see when there is electromagnetic disturbance. 

T: 106779.4969 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 106779.4974 P: 662 C: INFO S: diagCtrl M: rx: 246707, tx: 246545, err: 1 
T: 106779.4977 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (unexpected pl_len) 
T: 106779.4980 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_HOST_SYNC 
T: 106779.4983 P: 662 C: INFO S: diagCtrl M: rx: 246707, tx: 246545, err: 3 
T: 106779.4986 P: 662 C: WARN S: diagCtrl M: host_rcv: invalid pdu (hdr_len) 
T: 706246.3254 P: 662 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ

From the log above it seems like the camera detected an situation where it is not able to perform any trigger. This can be caused by over temperature or an issue with the power-supply. Is the log from a situation where you not able to recover the framegrabber?

robinbansal08 commented 3 years ago
robinbansal08 commented 3 years ago

Here is the log from another camera, in which I got same framegrabber timeout error (as explained before) while my application was running fine for several hours and was able to capture previous frames successfully but failed later suddenly with framegrabber issue.

loadConfig: Redis connection problem for connecting via /tmp/redis-cache.sock
loadConfig: Redis connection problem for connecting via /tmp/redis-cache.sock
T: 11.5936 P: 423 C: WARN S: iom M: Program starting ... 
T: 11.5971 P: 419 C: WARN S: qt_dbus_web_bridge M: Starting ... 
T: 11.7942 P: 423 C: INFO S: iom M: DIAGC_DBUS_EVENT_DIAGC_BOOTED 
T: 11.8023 P: 423 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 11.8027 P: 423 C: WARN S: diagCtrl M: drv_cfg_hw_late_init[231]: calib_dac failed 
T: 11.8029 P: 423 C: INFO S: diagCtrl M: sysboot_hw_evt[139]: successfully init HW : ### 
T: 11.8032 P: 423 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 11.8034 P: 423 C: WARN S: diagCtrl M: do_conv_ana_out_calib[94]: invalid ana out calib params 
T: 11.8037 P: 423 C: WARN S: diagCtrl M: Pin reset  
T: 12.4094 P: 460 C: WARN S: suprge M: suprge: supervision activity starting, Bello the watchdog, is unfused ... 
T: 12.7586 P: 452 C: WARN S: gecco M: Starting ... 
T: 12.7664 P: 452 C: WARN S: gecco M: Build version        :  O2I_1.20.7428-0-g91c60a74be  
T: 12.7670 P: 452 C: WARN S: gecco M: ifm SW version       :  1.71.7444  
T: 12.7675 P: 452 C: WARN S: gecco M: ifm Recovery version :  unversioned  
T: 13.7110 P: 460 C: WARN S: suprge M: Registering services for the O3D3XX family of devices 
T: 13.7702 P: 452 C: ERROR S: gecco M: static void ifm::JsonSchemaPropertyReader::createDefaultFromSchema(const QString&, const QString&): initializing key default:time_read_only failed (god error 1010) 
T: 14.0034 P: 440 C: INFO S: GECOM M: DBUS connection successful 
T: 14.0881 P: 452 C: WARN S: gecco M: ifm::FieldbusDevice::FieldbusDevice(ifm::SmartCamera&): fieldbus type 1 is not allowed on this device 
T: 14.1437 P: 449 C: WARN S: imager M: Starting ... 
T: 14.2983 P: 423 C: INFO S: diagCtrl M: Stop VLED monitoring # 
T: 14.4189 P: 452 C: WARN S: gecco M: QDBusConnection: name 'com.ifm.suprge' had owner '' but we thought it was ':1.7' 
T: 14.4542 P: 449 C: ERROR S: imager M: void gelfi::SingleMotorError::print() const :  No motor  ( 28 )  
T: 14.4583 P: 449 C: WARN S: imager M: void gelfi::StepperMotor::initSystem(bool, const gelfi::StepperMotor*): No motor is available for this article 
T: 17.8466 P: 423 C: INFO S: diagCtrl M: set_illu_power 1 # 
T: 18.0893 P: 423 C: INFO S: diagCtrl M: Start VLED monitoring # 
T: 18.0943 P: 452 C: WARN S: gecco M: Imager is available again 
T: 18.0988 P: 449 C: WARN S: imager M: LRAE is available again 
T: 19.2069 P: 452 C: WARN S: gecco M: Finished imager configuration 
T: 1524.9848 P: 423 C: WARN S: diagCtrl M: First Trigger 
T: 15724.6401 P: 423 C: WARN S: diagCtrl M: spi_err_dump: SPI_ERR_IRQ 
T: 15724.6405 P: 423 C: INFO S: diagCtrl M: rx: 34133, tx: 34096, err: 1 
T: 15724.6408 P: 423 C: WARN S: diagCtrl M: host_rcv: invalid pdu (unexpected pl_len) 
T: 15726.9223 P: 449 C: ERROR S: imager M: ifm::ErrorStatus::ErrorCode ifm::TriggerControl::trigger(): IO manager trigger returned error code 200000, current state is 1 
T: 15726.9235 P: 423 C: WARN S: iom M: DIAGC_DBUS_EVENT_TRG_OVERRUN 
T: 15726.9251 P: 423 C: INFO S: diagCtrl M: main_task[81]: evt_call_funcs returned 6, evt ID: 15
robinbansal08 commented 3 years ago

ProcessDetails