Error in the image processing loop

[mkaivs]

I have a pipeline that ran for hours before this error occurred.

66:52:26.688001436 11645 0x564a34d30590 ERROR               pylonsrc gstpylonsrc.c:2204:gst_pylonsrc_create:<camera> Error in the image processing loop.
66:52:26.688060908 11645 0x564a34d30590 INFO                 basesrc gstbasesrc.c:2839:gst_base_src_loop:<camera> pausing after gst_base_src_get_range() = error
66:52:26.688103115 11645 0x564a34d30590 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<camera> error: Internal data stream error.
66:52:26.688124190 11645 0x564a34d30590 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<camera> error: streaming stopped, reason error (-5)
66:52:26.688175968 11645 0x564a34d30590 INFO        GST_ERROR_SYSTEM gstelement.c:2145:gst_element_message_full_with_details:<camera> posting message: Internal data stream error.
66:52:26.688271008 11645 0x564a34d30590 INFO        GST_ERROR_SYSTEM gstelement.c:2172:gst_element_message_full_with_details:<camera> posted error message: Internal data stream error.
ERROR from element camera: Internal data stream error.
66:52:26.688349520 11645 0x564a34d30590 INFO                    task gsttask.c:316:gst_task_func:<camera:src> Task going to paused
Error details: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:ds-custom-pipeline/GstBin:source-bin-00/GstPylonsrc:camera:
streaming stopped, reason error (-5)
Returned, stopping playback

I looked into the source code and the error originated from this function:

static GstFlowReturn gst_pylonsrc_create (GstPushSrc *src, GstBuffer **buf)
{  
  GstPylonsrc *pylonsrc = GST_PYLONSRC (src);
  GENAPIC_RESULT res;
  size_t bufferIndex;
  PylonGrabResult_t grabResult;
  _Bool bufferReady;  
  GstMapInfo mapInfo;

  // Wait for the buffer to be filled  (up to 1 s)  
  res = PylonWaitObjectWait(pylonsrc->waitObject, 300000, &bufferReady);
  PYLONC_CHECK_ERROR(pylonsrc, res);
  if(!bufferReady) {
    GST_MESSAGE_OBJECT(pylonsrc, "Camera couldn't prepare the buffer in time. Probably dead.");    
    goto error;
  }

  res = PylonStreamGrabberRetrieveResult (pylonsrc->streamGrabber, &grabResult, &bufferReady);
  PYLONC_CHECK_ERROR(pylonsrc, res);
  if(!bufferReady) {
    GST_MESSAGE_OBJECT(pylonsrc, "Couldn't get a buffer from the camera. Basler said this should be impossible. You just proved them wrong. Congratulations!");    
    goto error;
  }

if(!pylonsrc->continuousMode) {
     // Trigger the next picture while we process this one
    if(PylonDeviceFeatureIsAvailable(pylonsrc->deviceHandle, "AcquisitionStatus")) {
    _Bool isReady = FALSE;
     do {
       res = PylonDeviceGetBooleanFeature(pylonsrc->deviceHandle, "AcquisitionStatus", &isReady);
       PYLONC_CHECK_ERROR(pylonsrc, res);
     } while (!isReady);
   }
    res = PylonDeviceExecuteCommandFeature(pylonsrc->deviceHandle, "TriggerSoftware");
   PYLONC_CHECK_ERROR(pylonsrc, res);
  }

  // Process the current buffer
  bufferIndex = (size_t) grabResult.Context;
  if(grabResult.Status == Grabbed) {        
    //TODO: See if I can avoid memcopy and record directly into the gst buffer map.

    // Copy the image into the buffer that will be passed onto the next GStreamer element
    *buf = gst_buffer_new_and_alloc(pylonsrc->payloadSize);
    gst_buffer_map(*buf, &mapInfo, GST_MAP_WRITE);
    orc_memcpy(mapInfo.data, grabResult.pBuffer, mapInfo.size);
    gst_buffer_unmap(*buf, &mapInfo);        

    // Release frame's memory
    res = PylonStreamGrabberQueueBuffer( pylonsrc->streamGrabber, grabResult.hBuffer, (void*) bufferIndex );
    PYLONC_CHECK_ERROR(pylonsrc, res);
  } else {
    GST_ERROR_OBJECT(pylonsrc, "Error in the image processing loop.");    
    goto error;
  }

I'm not sure what this error means and how to start fixing it. Please give me some explanation and pointers to help me deal with this error.

[mrstecklo]

@mkaivs, you are using an outdated code. Try pulling newest version. Also GST_DEBUG=2,pylonsrc:6 environment variable helps understanding errors

[mkaivs]

@mrstecklo, Thanks, I'm installing the newest code now. I will report back.

[MhdKAT]

Any updates on this @mkaivs ? I am having same issue

[joshdoe]

I didn't write this section of code, but have some guesses. It could be that there were missing packets due to network/system overloading, generating an incomplete buffer. To help identify this, I would add to that error statement the value of the grab result. If it's due to incomplete buffers, an element property could be added to ignore a certain number of incomplete buffers, or possibly fill in missing data from the previous frame. Do either of you have a way to reliably recreate this error condition?

[MhdKAT]

Thanks for the reply @joshdoe. Actually, I don't know if it is reproducible but I am getting this error when running this simple pipeline on Adlink Neon Vision Camera which has a basler acA2500-14uc USB3 camera connected to Jetson TX2 board. Pipeline : gst-launch-1.0 -v pylonsrc ! fpsfilter ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! nvegltransform ! nveglglessink sync=0. I have tried contacting Nvidia but according to them it is a camera "instability".

[thiesmoeller]

Hi @MhdKAT,

have you applied the recommendations in the installation readme in the pylon for Arm64 package?

e.g. Increase usbfs memory, increase transfer size in camera.

If not you generate a very high rate of smaller transfers, which could overload the host controller and can result in issues you observe

[MhdKAT]

Thank you @thiesmoeller I checked that usbfs was increased as in the installation recommandations' readme. Transfer size in the camera defaults to 65536. Is there a way to change it through plugin params ? I am currently setting the camera params through a saved config file but I don't think that param gets saved in that pfs file.

[mrstecklo]

@MhdKAT, every parameter that is displayed active in Pylon Viewer will be saved in PFS file. You can open PFS with text editor if you are unsure.

[MhdKAT]

Thank you @mrstecklo but I don't think it's there . Here is my PFS file : GainSelector All GainRaw 8 GainSelector All GainSelector All GainAuto Off GainSelector All GainSelector All AutoGainLowerLimitRaw 8 GainSelector All GainSelector All AutoGainUpperLimitRaw 127 GainSelector All BlackLevelSelector All BlackLevelRaw 0 BlackLevelSelector All GammaRaw 1000 BslColorSpaceMode sRGB Width 1296 Height 972 OffsetX 0 OffsetY 0 BinningHorizontalMode Average BinningHorizontal 2 BinningVerticalMode Average BinningVertical 2 ReverseX 0 ReverseY 0 PixelFormat YCbCr422_8 TestPattern Off SensorShutterMode Rolling OverlapMode On BslImmediateTriggerMode Off ExposureTimeRaw 22677 ExposureAuto Off AutoExposureTimeLowerLimitRaw 100 AutoExposureTimeUpperLimitRaw 100000 TriggerSelector FrameStart TriggerMode Off TriggerSelector FrameStart TriggerSelector FrameStart TriggerSource Software TriggerSelector FrameStart TriggerSelector FrameStart TriggerActivation RisingEdge TriggerSelector FrameStart ExposureMode Timed AcquisitionFramePeriodRaw 10 LightSourcePreset Daylight5000K BalanceRatioSelector Red BalanceRatioRaw 64 BalanceRatioSelector Green BalanceRatioRaw 77 BalanceRatioSelector Blue BalanceRatioRaw 103 BalanceRatioSelector Red BalanceWhiteAuto Off BslHueValue 0 BslSaturationValueRaw 1000 ContrastEnhancementRaw 0 BslContrastMode Linear BslBrightnessRaw 0 BslContrastRaw 0 PgiMode On_AutomaticNoiseReduction SharpnessEnhancementRaw 0 DefectPixelCorrectionMode On LineSelector Line1 LineMode Input LineSelector Line2 LineMode Input LineSelector Line1 LineSelector Line1 LineInverter 0 LineSelector Line2 LineInverter 0 LineSelector Line1 LineSelector Line1 LineSource UserOutput1 LineSelector Line2 LineSource UserOutput2 LineSelector Line1 LineSelector Line1 LineDebouncerTimeRaw 0 LineSelector Line2 LineDebouncerTimeRaw 0 LineSelector Line1 UserOutputSelector UserOutput1 UserOutputValue 0 UserOutputSelector UserOutput2 UserOutputValue 0 UserOutputSelector UserOutput1 AutoTargetBrightnessRaw 30 AutoTargetBrightnessDampingRaw 3 AutoBacklightCompensationRaw 0 AutoFunctionProfile Smart AutoFunctionROISelector ROI1 AutoFunctionROIWidth 1296 AutoFunctionROISelector ROI2 AutoFunctionROIWidth 1296 AutoFunctionROISelector ROI1 AutoFunctionROISelector ROI1 AutoFunctionROIHeight 972 AutoFunctionROISelector ROI2 AutoFunctionROIHeight 972 AutoFunctionROISelector ROI1 AutoFunctionROISelector ROI1 AutoFunctionROIOffsetX 0 AutoFunctionROISelector ROI2 AutoFunctionROIOffsetX 0 AutoFunctionROISelector ROI1 AutoFunctionROISelector ROI1 AutoFunctionROIOffsetY 0 AutoFunctionROISelector ROI2 AutoFunctionROIOffsetY 0 AutoFunctionROISelector ROI1 DeviceLinkSelector 0 DeviceLinkThroughputLimitMode On DeviceLinkSelector 0 DeviceLinkSelector 0 DeviceLinkThroughputLimit 163000000 DeviceLinkSelector 1 DeviceLinkThroughputLimit 0 DeviceLinkSelector 0 DeviceIndicatorMode Active

[mrstecklo]

@MhdKAT. Try adjusting Transfer Size a bit in Pylon Viewer. Probably some defaults are not exported to PFS. When property is represented in PFS it can be manually edited with text editor

[thiesmoeller]

It is not in the camera node map, but in the streamgrabber nodemap. It is a config param of the grab enginem

In pseudo code:

cam.StreamgrabberNodemap.MaxTransferSize = 4 1024 1024

[thiesmoeller]

I added the feature in my fork ... ( https://github.com/thiesmoeller/gst-plugins-vision/tree/feature/gst-pylonsrc_add_usb3_maxtransfersize ) You can now call to get e.g. 4MB transfers like described above:

gst-launch-1.0 pylonsrc camera=0 max-transfer-size=4194304 ! videoconvert ! autovideosink