Camera communication (high ping packets loss)

robinbansal08 commented 4 years ago

I'm running Ubuntu 16.04 together with the O3D303 camera.

A ping to the camera IP 192.168.0.51 shows 90% packet loss:

$ ping 192.168.0.51
PING 192.168.0.51 (192.168.0.51) 56(84) bytes of data.
64 bytes from 192.168.0.51: icmp_seq=26 ttl=64 time=0.164 ms
64 bytes from 192.168.0.51: icmp_seq=27 ttl=64 time=0.214 ms
64 bytes from 192.168.0.51: icmp_seq=28 ttl=64 time=0.218 ms
64 bytes from 192.168.0.51: icmp_seq=68 ttl=64 time=0.280 ms
64 bytes from 192.168.0.51: icmp_seq=69 ttl=64 time=0.163 ms
64 bytes from 192.168.0.51: icmp_seq=70 ttl=64 time=0.262 ms
64 bytes from 192.168.0.51: icmp_seq=71 ttl=64 time=0.169 ms
64 bytes from 192.168.0.51: icmp_seq=72 ttl=64 time=0.192 ms
64 bytes from 192.168.0.51: icmp_seq=73 ttl=64 time=0.179 ms
64 bytes from 192.168.0.51: icmp_seq=74 ttl=64 time=0.196 ms
64 bytes from 192.168.0.51: icmp_seq=75 ttl=64 time=0.138 ms
64 bytes from 192.168.0.51: icmp_seq=76 ttl=64 time=0.172 ms
64 bytes from 192.168.0.51: icmp_seq=77 ttl=64 time=0.141 ms
64 bytes from 192.168.0.51: icmp_seq=115 ttl=64 time=0.399 ms
64 bytes from 192.168.0.51: icmp_seq=116 ttl=64 time=0.168 ms
64 bytes from 192.168.0.51: icmp_seq=121 ttl=64 time=0.162 ms
64 bytes from 192.168.0.51: icmp_seq=155 ttl=64 time=0.324 ms
64 bytes from 192.168.0.51: icmp_seq=184 ttl=64 time=0.261 ms
64 bytes from 192.168.0.51: icmp_seq=189 ttl=64 time=0.149 ms
^C
--- 192.168.0.51 ping statistics ---
196 packets transmitted, 19 received, 90% packet loss, time 199685ms
rtt min/avg/max/mdev = 0.138/0.207/0.399/0.069 ms

When trying to use libo3d3xx command line utilities commands, most of the times, it fails with XMRPC call timed out error.

pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-ls --ip 192.168.0.51
Failed to list applications: XMLRPC call timed out
pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-ls --ip 192.168.0.51
Failed to list applications: XMLRPC call timed out
pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-ls --ip 192.168.0.51
* [1] id=1186278907, name=Box Detection Application, description=
pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-ls --ip 192.168.0.51
Failed to list applications: XMLRPC call timed out
pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-ls --ip 192.168.0.51
Failed to list applications: XMLRPC call timed out
pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-ls --ip 192.168.0.51
Failed to list applications: XMLRPC call timed out
pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-ls --ip 192.168.0.51
Failed to list applications: XMLRPC call timed out

I have already tried with rebooting, resetting the camera, but problem still persists.

There are several other O3D303 cameras connected in the same network, but they are working fine.

pactrispro@CLOUDLink:~/xpsSea$ o3d3xx-dump --ip 192.168.0.51 > camera_51_dump.json

Here is the result of it:


{
    "o3d3xx": {
        "libo3d3xx": "705",
        "Date": "Wed Jun  3 00:30:54 2020",
        "HWInfo": {
            "Connector": "#!02_A300_C40_04246016_008023176",
            "Diagnose": "#!02_D322_C34_04068012_008026824",
            "MiraSerial": "0670-81fe-0e78-0504",
            "MACAddress": "00:02:01:41:39:D5",
            "Frontend": "#!02_F342_C34_19_00156_008023607",
            "Mainboard": "#!02_M381_C41_04215040_008023690",
            "Illumination": "#!02_I300_001_04201465_008001175"
        },
        "SWVersion": {
            "IFM_Software": "1.6.2114",
            "Main_Application": "0.4.222",
            "Algorithm_Version": "1.2.19",
            "Calibration_Device": "00:02:01:41:39:d5",
            "ELDK": "GOLDENEYE_ELDK-2016-07-14_09-36-02-2e9219cce224332ecbe2f36934d2c3c7aa738c78",
            "Calibration_Version": "1.0.2",
            "Linux": "Linux version 3.11.0-00002-gd1bbe0b (eswadmin@dettlx79) (gcc version 4.7.2 (GCC) ) #1 SMP Fri Jun 24 16:34:26 CEST 2016",
            "Diagnostic_Controller": "0.3.207"
        },
        "Device": {
            "Name": "New sensor",
            "Description": "",
            "ActiveApplication": "1",
            "PcicTcpPort": "50010",
            "PcicProtocolVersion": "3",
            "IOLogicType": "1",
            "IODebouncing": "true",
            "IOExternApplicationSwitch": "0",
            "SessionTimeout": "30",
            "ServiceReportPassedBuffer": "15",
            "ServiceReportFailedBuffer": "15",
            "ExtrinsicCalibTransX": "0",
            "ExtrinsicCalibTransY": "0",
            "ExtrinsicCalibTransZ": "0",
            "ExtrinsicCalibRotX": "0",
            "ExtrinsicCalibRotY": "0",
            "ExtrinsicCalibRotZ": "0",
            "EvaluationFinishedMinHoldTime": "10",
            "SaveRestoreStatsOnApplSwitch": "true",
            "IPAddressConfig": "0",
            "PasswordActivated": "false",
            "OperatingMode": "1",
            "DeviceType": "1:2",
            "ArticleNumber": "O3D303",
            "ArticleStatus": "AD",
            "UpTime": "0.0508333333333333",
            "ImageTimestampReference": "0",
            "TemperatureFront1": "3276.6999999999998",
            "TemperatureFront2": "3276.6999999999998",
            "TemperatureIMX6": "30.2600002288818",
            "TemperatureIllu": "35.200000000000003",
            "PNIODeviceName": "",
            "EthernetFieldBus": "1",
            "EthernetFieldBusEndianness": "0",
            "EnableAcquisitionFinishedPCIC": "false"
        },
        "Net": {
            "MACAddress": "",
            "NetworkSpeed": "0",
            "StaticIPv4Address": "192.168.0.51",
            "StaticIPv4SubNetMask": "255.255.255.0",
            "StaticIPv4Gateway": "192.168.0.201",
            "UseDHCP": "false"
        },
        "Apps": [
            {
                "Name": "Box Detection Application",
                "Description": "",
                "TriggerMode": "2",
                "PcicTcpResultSchema": "{ \"layouter\": \"flexible\", \"format\": { \"dataencoding\": \"ascii\" }, \"elements\": [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"blob\", \"id\": \"normalized_amplitude_image\" }, { \"type\": \"blob\", \"id\": \"distance_image\" }, { \"type\": \"blob\", \"id\": \"x_image\" }, { \"type\": \"blob\", \"id\": \"y_image\" }, { \"type\": \"blob\", \"id\": \"z_image\" }, { \"type\": \"blob\", \"id\": \"confidence_image\" }, { \"type\": \"blob\", \"id\": \"diagnostic_data\" }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "PcicEipResultSchema": "{ \"layouter\": \"flexible\", \"format\": { \"dataencoding\": \"binary\", \"order\": \"big\" }, \"elements\" : [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"records\", \"id\": \"models\", \"elements\": [ { \"type\": \"int16\", \"id\": \"boxFound\" }, { \"type\": \"int16\", \"id\": \"width\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"height\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"length\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"xMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"zMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yawAngle\" }, { \"type\": \"int16\", \"id\": \"qualityLength\" }, { \"type\": \"int16\", \"id\": \"qualityWidth\" }, { \"type\": \"int16\", \"id\": \"qualityHeight\" } ] }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "PcicPnioResultSchema": "{\"layouter\" : \"flexible\", \"format\": { \"dataencoding\": \"binary\", \"order\": \"big\" }, \"elements\" : [ { \"type\": \"string\", \"value\": \"star\", \"id\": \"start_string\" }, { \"type\": \"records\", \"id\": \"models\", \"elements\": [ { \"type\": \"int16\", \"id\": \"boxFound\" }, { \"type\": \"int16\", \"id\": \"width\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"height\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"length\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"xMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"zMidTop\", \"format\": { \"scale\": 1000 } }, { \"type\": \"int16\", \"id\": \"yawAngle\" }, { \"type\": \"int16\", \"id\": \"qualityLength\" }, { \"type\": \"int16\", \"id\": \"qualityWidth\" }, { \"type\": \"int16\", \"id\": \"qualityHeight\" } ] }, { \"type\": \"string\", \"value\": \"stop\", \"id\": \"end_string\" } ] }",
                "LogicGraph": "{\"IOMap\": {\"OUT1\": \"RFT\",\"OUT2\": \"AQUFIN\"},\"blocks\": {\"B00001\": {\"pos\": {\"x\": 200,\"y\": 200},\"properties\": {},\"type\": \"PIN_EVENT_IMAGE_ACQUISITION_FINISHED\"},\"B00002\": {\"pos\": {\"x\": 200,\"y\": 75},\"properties\": {},\"type\": \"PIN_EVENT_READY_FOR_TRIGGER\"},\"B00003\": {\"pos\": {\"x\": 600,\"y\": 75},\"properties\": {\"pulse_duration\": 0},\"type\": \"DIGITAL_OUT1\"},\"B00005\": {\"pos\": {\"x\": 600,\"y\": 200},\"properties\": {\"pulse_duration\": 0},\"type\": \"DIGITAL_OUT2\"}},\"connectors\": {\"C00000\": {\"dst\": \"B00003\",\"dstEP\": 0,\"src\": \"B00002\",\"srcEP\": 0},\"C00001\": {\"dst\": \"B00005\",\"dstEP\": 0,\"src\": \"B00001\",\"srcEP\": 0}}}",
                "Type": "Camera",
                "Index": "1",
                "Id": "1186278907",
                "Imager": {
                    "AutoExposureReferencePointX": "88",
                    "AutoExposureReferencePointY": "66",
                    "AutoExposureReferenceType": "0",
                    "AutoExposureMaxExposureTime": "10000",
                    "AutoExposureReferenceROI": "",
                    "Channel": "0",
                    "ClippingBottom": "263",
                    "ClippingLeft": "0",
                    "ClippingRight": "351",
                    "ClippingTop": "0",
                    "ContinuousAutoExposure": "false",
                    "EnableAmplitudeCorrection": "true",
                    "EnableFastFrequency": "false",
                    "EnableFilterAmplitudeImage": "true",
                    "EnableFilterDistanceImage": "true",
                    "EnableRectificationAmplitudeImage": "false",
                    "EnableRectificationDistanceImage": "false",
                    "UseSimpleBinning": "false",
                    "ExposureTime": "10000",
                    "ExposureTimeList": "2500;10000",
                    "ExposureTimeRatio": "4",
                    "FrameRate": "5",
                    "MinimumAmplitude": "2",
                    "Resolution": "1",
                    "ClippingCuboid": "{\"XMin\": -3.402823e+38, \"XMax\": 3.402823e+38, \"YMin\": -3.402823e+38, \"YMax\": 3.402823e+38, \"ZMin\": -3.402823e+38, \"ZMax\": 3.402823e+38}",
                    "SpatialFilterType": "1",
                    "SymmetryThreshold": "0",
                    "TemporalFilterType": "0",
                    "ThreeFreqMax2FLineDistPercentage": "80",
                    "ThreeFreqMax3FLineDistPercentage": "80",
                    "TwoFreqMaxLineDistPercentage": "80",
                    "Type": "upto30m_moderate",
                    "MaxAllowedLEDFrameRate": "4.9000000000000004",
                    "SpatialFilter": {
                        "Type": "1",
                        "TypeStr_": "Median Filter",
                        "MaskSize": "1",
                        "MaskSizeStr_": "5x5"
                    },
                    "TemporalFilter": {
                        "Type": "0",
                        "TypeStr_": "Off"
                    }
                }
            }
        ]
    }
}

theseankelly commented 4 years ago

Hey @robinbansal08 is it possible there's an IP address conflict on your network (perhaps DHCP Handed 192.168.0.55 to another device)? You could check by powering off the camera and seeing if ping works or fails.

Otherwise, the 1.6.2114 firmware release is several years old and it would be worth your time to upgrade and see if the issues persist.

@graugans any other thoughts from the embedded side?

graugans commented 4 years ago

@robinbansal08 please check the following:

Are there any O3D3xx devices with consecutive MAC addresses in your network. If so please remove all cameras which are close (+/-1) to the one you are testing. After this a full power cycle of any active network component is recommended. A solution to this is a FW update to version 1.30.x
Check if there is an issue with the Ethernet cable used. You can cross test with another working camera
If there is any active network component, like a switch, in between please also power cycle them and try a different port.

If the error is still present please contact your local ifm sales responsible and initiate an RMA

robinbansal08 commented 4 years ago

Thank you for your replies.

@graugans You are correct regarding the MAC addresses issue (+/-1). This camera (192.168.0.51) with the issue was working fine before, but this issue came up when I updated the firmware (to version 1.23.x) of few of the other cameras in the same network. Based on your suggestion, I have compared their MAC addresses and found one of the cameras (of which I updated the firmware) has consecutive sequence of the MAC address to the camera (192.168.0.51).

I have already replaced the camera(192.168.0.51) and now all the cameras in the network are working fine now.

I have also checked the MAC addresses of all the cameras in the same network and found around half of the cameras having MAC addresses +/-1 with other cameras.

Currently all cameras are working fine but I want to update their firmware as we have to run our applications over the camera. So if we update the firmware with the latest version 1.30.x, will this issue (ping packets loss) can still occur with the cameras with consecutive sequence MAC addresses ?

robinbansal08 commented 4 years ago

This issue is resolved, as per your suggestion I have updated the firmware of the cameras to 1.30.5309 and the cameras with sequence of +/-1 MAC addresses are working fine.

graugans commented 4 years ago

I am glad to hear the issue is solved. I'll close this, please reopen in case of any issues related to this topic.

ifm / libo3d3xx

Camera communication (high ping packets loss) #140