ia.fetch_buffer() Infinite Loop

[luc-bot]

Seemingly randomly the function ia.fetch_buffer() is stuck in an infinite loop. I built a simple GUI which connects to a Photonfocus camera MV1312 with a start and stop button and window to display the images. If I start and stop the process multiple times it will get stuck on ia.fetch_buffer() seemingly randomly. with the example:

print('a') ia.fetch_buffer() print('b')

'b' never gets printed.

[mahwhat]

Hi @luc-bot ,

I once had problems with the acquisition being blocked.

To solve this, I added the buffer.queue() command after the buffer manipulation to allow the buffer to be used again in another acquisition.

[luc-bot]

Hi @mahwhat, is there a specific time after I call the buffer that I can queue it? If I trying queuing the buffer right after I grab the image I get an error that "GenTL does not support this operation"

With these lines of code: buffer = ia.fetch_buffer() img = buffer.payload.components[0].data buffer.queue()

If I queue the buffer right after I'm done manipulating the variable "img" then sometimes it'll get stuck in the infinite loop mentioned above.

[mahwhat]

Hi @luc-bot,

I normally enqueue the buffer after manipulation and process the image. Here's a snippet of my code:

buffer = ia.fetch_buffer()
component = buffer.payload.components[0]
 _img = component.data.reshape(component.height, component.width)
 self._pipeline(_img) # Here i process the image 
 buffer.queue()

[kazunarikudo]

@mahwhat Hi, thank you for the help you offered. I appreciate that. Regards, Kazunari.

[luc-bot]

I tried different methods to queue the buffer as shown in the setup page. With some new testing I think I narrowed down the cause of the error. It appears that sometimes my camera returns an error message instead of an image. In this event ia.fetch_buffer() gets stuck and I never get to the next line of code.

This took a significant of time to test because I have a few different cameras and the problem only occurs on certain systems, possibly a problem with the network settings?

[kazunarikudo]

A possible case that I can think of is, where Harvester faces an exceptional situation and forgets to queue the corrupted data containing buffer to the GenTL Producer. So here are two questions for you:

1. How did you recognize that "your camera returns an error message"?
2. Is the issue reproducible in both "manual queueing" and "automatic queueing based on the with statement"?

Thanks, Kazunari.

[kazunarikudo]

If you have a traceback that can be a piece of evidence for question no. 1, please paste it; it will be valuable for the investigation.

[luc-bot]

1. Alternatively to using harvesters I have accessed the camera with "eBUS Player" a software recommended by the camera manufacturer. While switching between my code, which is made with harvester, and this software I have noticed that only on instances when I receive these warnings do I get stuck in an infinite loop as described in my initial post. You can see the warnings that I am receiving in the attached image

2. Yes, the issue exists in both "manual queuing" and "automatic queuing based on the with statement"?.

Please let me know if you more details would be helpful for you or if you can suggest a more robust way for me to prove my hypothesis is correct that the warning messages are related to the bug.

Thank you.

[kazunarikudo]

Excuse me, the comments above do not mean the issue has been resolved. I am planning to prepare suggestions for further investigation.

[kazunarikudo]

@luc-bot Hi, this is just an idea for trial, you can prevent the fetch (or try_fetch) call from blocking you by setting a timeout period [s] to the timeout parameter:

try_fetch(timeout=0.1)  # the unit is [s]

After the specified period is elapsed and no valid image was delivered then the try_fetch call will return None and the fetch call will raise genicam.gentl.TimeoutException.

Harvester can't help you to prevent a situation where a buffer unexpectedly drops but the suggested approach above would make sense for you more or less.

[luc-bot]

Hi @kazunarikudo, previously, I was using version 1.3.2 and updated to version 1.3.6 to use try_fetch. Setting the timeout as you mentioned above seems to solve my previous issue, but there appear to be 2 new problems this has raised.

1. Updating to version 1.3.6 introduced a significant lag to the captured images based on how long it takes for the rest of my code to process the images.
2. As soon as I receive a buffer of "None" all buffers after the fact are "None" as well. If I stop and restart the image acquisition I still encounter this problem, the only solutions seems to be destroying and then creating the image acquirer. (This may have been an issue when I was using 1.3.2, but as my previous code crashed I couldn't verify this).

[kazunarikudo]

@luc-bot Hi, just a quick comment: Could you try the instruction that I mentioned here so that I can diagnose what's happening on your side? I need to confess that I do not know if there's a difference between 1.3.2 and 1.3.6 but I guess the lag you are facing is coming from a fact that sufficient buffers are not being delivered. In principle, Harvester just asks a GenTL Producer if it has a buffer that is ready to be fetched. In addition, when the Producer says "yes, I have," the buffer has already been on the computer so Harvester itself never drops it. Finally, you face the consecutive None because there's no buffer to be fetched; it eventually causes timeout and returns None to not block the execution. The consecutive timeout is just showing up as None.

[kazunarikudo]

One more thing, I would like to encourage you to try another setup such as another computer, another camera, another cable, etc. Of course, they may never be the production component but it should be worth trying to check if something can make difference. If such image acquisition quality is the best performance for Harvester then nobody would use that. ;-) Last but not least, could you tell if you have enabled jumbo-frame? I do not want to hear you've been using 1500 bytes that is by default! Thanks!

[michezio]

Hi, I think I am facing the same issue as the OP.

In my case I am using 2 cameras with the .cti producer from Matrix Vision (downloaded from here) in a multiprocessing environment. Since the full code is quite complex here is a stripped down version that produces the same behaviour:

Multiprocessing code

```python import traceback import multiprocessing import threading import ctypes import numpy as np import cv2 import time import os from harvesters.core import Harvester, TimeoutException def log(tag, *args, **kwargs): print("[ " + tag.ljust(18) + " ]", *args, flush=True, **kwargs) class GigeManagerProcess(multiprocessing.Process): def __init__(self, sources:int, stop_event:multiprocessing.Event): super().__init__() self.logtag = "GigE Manager" self.stop_event = stop_event self.sources = sources self.shape = (1024, 1280) self.frame_buffers = [multiprocessing.Array(ctypes.c_uint8, int(np.prod(self.shape)), lock=False) for _ in range(self.sources)] self.camera_threads = [] self.harvester = None def run(self): self.harvester = Harvester() self.harvester.add_file("mvGenTLProducer.cti") self.harvester.update() assert len(self.harvester.device_info_list) > 0, "No camera connected" log(self.logtag, "Connected GigE-Vision cameras:") for i, dev in enumerate(self.harvester.device_info_list): in_use = "-> " if i < self.sources else " " managed_by = f" managed by Camera Acquirer {i}" if i < self.sources else "" log(self.logtag, f"{in_use}[{i}] {dev.vendor} {dev.model} (S/N: {dev.serial_number}){managed_by}") for camera_id in range(self.sources): x = threading.Thread(target=self.camera_worker, args=(camera_id,), name=f"CameraAcquirer_{camera_id}", daemon=True) self.camera_threads.append(x) x.start() log(self.logtag, "Initialization completed") while not self.stop_event.is_set(): # doing some other stuff (~100 ms processing time) time.sleep(0.1) self.stop() def stop(self): self.stop_event.set() for t in self.camera_threads: if t.is_alive(): t.join() if self.harvester is not None: self.harvester.reset() log(self.logtag, "Terminated") def camera_worker(self, camera_id:int): logtag = f"Camera Acquirer {camera_id}" try: ia = self.harvester.create(camera_id) nm = ia.remote_device.node_map nm.Width.value = self.shape[1] nm.Height.value = self.shape[0] nm.PixelFormat.value = "Mono8" ia.start(run_as_thread=False) log(logtag, "Capturing") timeout_events = 0 while not self.stop_event.is_set(): try: buffer = ia.fetch(timeout=0.5) timeout_events = 0 component = buffer.payload.components[0] frame = component.data.reshape(component.height, component.width) buf = np.frombuffer(self.frame_buffers[camera_id], dtype=np.uint8) buf = buf.reshape(self.shape) buf[:] = frame except TimeoutException: timeout_events += 1 if timeout_events >= 3: log(logtag, f"Buffer timeout ({timeout_events}/3), there is a problem with the camera, ending the acquisition\n", traceback.format_exc()) self.stop_event.set() break else: log(logtag, f"Buffer timeout ({timeout_events}/3), frame skipped") finally: buffer.queue() except: log(logtag, f"Camera error\n", traceback.format_exc()) self.stop_event.set() return ia.destroy() log(logtag, "Terminated") if __name__ == "__main__": stop_event = multiprocessing.Event() acquirer = GigeManagerProcess(2, stop_event) acquirer.start() # other processes are initialized here while not stop_event.is_set(): frame_0 = np.frombuffer(acquirer.frame_buffers[0], dtype=np.uint8) frame_0 = frame_0.reshape(acquirer.shape) frame_1 = np.frombuffer(acquirer.frame_buffers[1], dtype=np.uint8) frame_1 = frame_1.reshape(acquirer.shape) # other processes access to these frames (read-only) and process them cv2.imshow("CAMERA 0 TEST", cv2.resize(frame_0, (None, None), fx=0.5, fy=0.5)) cv2.imshow("CAMERA 1 TEST", cv2.resize(frame_1, (None, None), fx=0.5, fy=0.5)) if cv2.waitKey(1) & 0xFF == 27: stop_event.set() acquirer.stop() cv2.destroyAllWindows() ```

Both threads sometimes get stuck in an infinite loop at buffer = ia.fetch(timeout=0.5). Unfortunately it happens rarely and randomly so it's hard to know exactly what's triggering the behavior. When the threads are stuck they generate no error and become non responsive so when I quit the application they remain active and I have to manually kill all the active python processes to get rid of them.

Anyway, I noticed that it's more likely to happen when there are many parallel processes (in my main code there are at least 6 more processes and it happens almost 50% of the time) or when the previous run has crashed or has been force closed.

I tried to run this code on both Windows 10 and Debian 11 on 3 different machines and a VM and the infinite loop happens ~5% of the time. If I enable Harvesters logs as suggested here I can see this line repeated over and over (with different memory addresses) while stuck in the infinite loop:

2022-08-03 14:01:06,492 :: harvesters :: WARNING :: incomplete or not available; discarded: <genicam.gentl.Buffer; proxy of <Swig Object of type 'std::shared_ptr< GenTLCpp::Buffer > *' at 0x7f5687869780> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: <genicam.gentl.GenTLProducer; proxy of <Swig Object of type 'std::shared_ptr< GenTLCpp::GenTLProducer > *' at 0x7f5687869f90> >

However, the test script suggested on that same post gives me OK result every time (except only once that somehow I managed to get an error right after I force closed a run that was stuck but unfortunately I've lost the log). Here's the log from it if it can be helpful:

Test log

``` 2022-08-03 16:15:14,224 :: harvesters :: INFO :: created: 2022-08-03 16:15:14,224 :: harvesters :: INFO :: added: libs/gentl_producers/mvGenTLProducer_linux.cti to 2022-08-03 16:15:14,225 :: harvesters :: DEBUG :: discarded device information: 2022-08-03 16:15:14,233 :: harvesters :: DEBUG :: initialized file: libs/gentl_producers/mvGenTLProducer_linux.cti 2022-08-03 16:15:14,240 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_System_10_1_0.xml;100000000;a6f5 2022-08-03 16:15:14,241 :: harvesters :: DEBUG :: created: /tmp/20220803161514_88bvmxty/MATRIXVISION_GenTL_System_10_1_0.xml 2022-08-03 16:15:14,244 :: harvesters :: DEBUG :: deleted: /tmp/20220803161514_88bvmxty/MATRIXVISION_GenTL_System_10_1_0.xml 2022-08-03 16:15:14,244 :: harvesters :: DEBUG :: opened: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f32df0c0> > 2022-08-03 16:15:14,321 :: harvesters :: DEBUG :: opened: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f32df4e0> > 2022-08-03 16:15:14,321 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_Interface_15_0_0.xml;100000000;12672 2022-08-03 16:15:14,322 :: harvesters :: DEBUG :: created: /tmp/20220803161514_520fq_h5/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:14,325 :: harvesters :: DEBUG :: deleted: /tmp/20220803161514_520fq_h5/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,360 :: harvesters :: DEBUG :: opened: /devices/pci0000:00/0000:00:11.0/0000:02:00.0/usb3 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f32dfa20> > 2022-08-03 16:15:15,361 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_Interface_15_0_0.xml;100000000;12672 2022-08-03 16:15:15,361 :: harvesters :: DEBUG :: created: /tmp/20220803161515_1gmfkqf1/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,365 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_1gmfkqf1/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,368 :: harvesters :: DEBUG :: opened: /devices/pci0000:00/0000:00:11.0/0000:02:02.0/usb1 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f32dfcc0> > 2022-08-03 16:15:15,368 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_Interface_15_0_0.xml;100000000;12672 2022-08-03 16:15:15,369 :: harvesters :: DEBUG :: created: /tmp/20220803161515_ws_7npi2/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,372 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_ws_7npi2/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,375 :: harvesters :: DEBUG :: opened: /devices/pci0000:00/0000:00:15.0/0000:03:00.0/usb2 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f32dff60> > 2022-08-03 16:15:15,375 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_Interface_15_0_0.xml;100000000;12672 2022-08-03 16:15:15,376 :: harvesters :: DEBUG :: created: /tmp/20220803161515_va3pzy89/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,379 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_va3pzy89/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,382 :: harvesters :: DEBUG :: opened: /devices/pci0000:00/0000:00:15.0/0000:03:00.0/usb4 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1901240> > 2022-08-03 16:15:15,382 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_Interface_15_0_0.xml;100000000;12672 2022-08-03 16:15:15,383 :: harvesters :: DEBUG :: created: /tmp/20220803161515_l0f_p52v/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,387 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_l0f_p52v/MATRIXVISION_GenTL_Interface_15_0_0.xml 2022-08-03 16:15:15,390 :: harvesters :: INFO :: updated: 2022-08-03 16:15:15,394 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_Device_8_0_0.xml;100000000;7edd 2022-08-03 16:15:15,395 :: harvesters :: DEBUG :: created: /tmp/20220803161515_ip0pcn_t/MATRIXVISION_GenTL_Device_8_0_0.xml 2022-08-03 16:15:15,396 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_ip0pcn_t/MATRIXVISION_GenTL_Device_8_0_0.xml 2022-08-03 16:15:15,397 :: harvesters :: DEBUG :: opened: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:15,401 :: harvesters :: DEBUG :: fetched url: Local:MER_GIGE_V1.3000.51.zip;40ba0000;4ff1 2022-08-03 16:15:15,480 :: harvesters :: DEBUG :: created: /tmp/20220803161515_juzkr_jp/MER_GIGE_V1.3000.51.zip 2022-08-03 16:15:15,491 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_juzkr_jp/MER_GIGE_V1.3000.51.zip 2022-08-03 16:15:15,680 :: harvesters :: DEBUG :: opened: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19019f0> > 2022-08-03 16:15:15,681 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_DataStream_12_0_0.xml;100000000;9844 2022-08-03 16:15:15,683 :: harvesters :: DEBUG :: created: /tmp/20220803161515_tlqry7h8/MATRIXVISION_GenTL_DataStream_12_0_0.xml 2022-08-03 16:15:15,693 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_tlqry7h8/MATRIXVISION_GenTL_DataStream_12_0_0.xml 2022-08-03 16:15:15,694 :: harvesters :: DEBUG :: created: 2022-08-03 16:15:15,695 :: harvesters :: INFO :: created: for MER-131-75GM-P(00:21:49:03:ae:27) by 2022-08-03 16:15:15,702 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_Device_8_0_0.xml;100000000;7edd 2022-08-03 16:15:15,703 :: harvesters :: DEBUG :: created: /tmp/20220803161515_xzm0ygk9/MATRIXVISION_GenTL_Device_8_0_0.xml 2022-08-03 16:15:15,706 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_xzm0ygk9/MATRIXVISION_GenTL_Device_8_0_0.xml 2022-08-03 16:15:15,706 :: harvesters :: DEBUG :: opened: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:15,711 :: harvesters :: DEBUG :: fetched url: Local:MER_GIGE_V1.3000.51.zip;40ba0000;4ff1 2022-08-03 16:15:15,791 :: harvesters :: DEBUG :: created: /tmp/20220803161515_jhq8tk5q/MER_GIGE_V1.3000.51.zip 2022-08-03 16:15:15,803 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_jhq8tk5q/MER_GIGE_V1.3000.51.zip 2022-08-03 16:15:15,991 :: harvesters :: DEBUG :: opened: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f190b7b0> > 2022-08-03 16:15:15,993 :: harvesters :: DEBUG :: fetched url: Local:MATRIXVISION_GenTL_DataStream_12_0_0.xml;100000000;9844 2022-08-03 16:15:15,995 :: harvesters :: DEBUG :: created: /tmp/20220803161515_kt0m23pb/MATRIXVISION_GenTL_DataStream_12_0_0.xml 2022-08-03 16:15:16,008 :: harvesters :: DEBUG :: deleted: /tmp/20220803161515_kt0m23pb/MATRIXVISION_GenTL_DataStream_12_0_0.xml 2022-08-03 16:15:16,009 :: harvesters :: DEBUG :: created: 2022-08-03 16:15:16,011 :: harvesters :: INFO :: created: for MER-131-75GM-P(00:21:49:03:ae:25) by 2022-08-03 16:15:16,012 :: harvesters :: DEBUG :: allocated: 1310720 bytes by 2022-08-03 16:15:16,013 :: harvesters :: DEBUG :: allocated: 1310720 bytes by 2022-08-03 16:15:16,013 :: harvesters :: DEBUG :: allocated: 1310720 bytes by 2022-08-03 16:15:16,014 :: harvesters :: DEBUG :: announced: *' at 0x7ff3f190be70> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911180> > 2022-08-03 16:15:16,014 :: harvesters :: DEBUG :: announced: *' at 0x7ff3f190bf30> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911210> > 2022-08-03 16:15:16,015 :: harvesters :: DEBUG :: announced: *' at 0x7ff3f190bf90> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19112a0> > 2022-08-03 16:15:16,016 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190be70> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19112a0> > 2022-08-03 16:15:16,017 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bf30> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911330> > 2022-08-03 16:15:16,019 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bf90> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911360> > 2022-08-03 16:15:16,026 :: harvesters :: DEBUG :: launched thread: 2022-08-03 16:15:16,031 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,032 :: harvesters :: INFO :: started acquisition: 2022-08-03 16:15:16,039 :: harvesters :: DEBUG :: started streaming: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911480> > 2022-08-03 16:15:16,045 :: harvesters :: DEBUG :: allocated: 1310720 bytes by 2022-08-03 16:15:16,045 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,046 :: harvesters :: DEBUG :: allocated: 1310720 bytes by 2022-08-03 16:15:16,046 :: harvesters :: DEBUG :: allocated: 1310720 bytes by 2022-08-03 16:15:16,052 :: harvesters :: DEBUG :: announced: *' at 0x7ff3f19114e0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911750> > 2022-08-03 16:15:16,053 :: harvesters :: DEBUG :: announced: *' at 0x7ff3f19114b0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911900> > 2022-08-03 16:15:16,052 :: harvesters :: DEBUG :: fetched: 0 (#1); *' at 0x7ff3f19115a0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19118d0> > 2022-08-03 16:15:16,053 :: harvesters :: DEBUG :: announced: *' at 0x7ff3f1911720> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911960> > 2022-08-03 16:15:16,054 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f19114e0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911930> > 2022-08-03 16:15:16,055 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f19114b0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911990> > 2022-08-03 16:15:16,056 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911720> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19119c0> > 2022-08-03 16:15:16,067 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,073 :: harvesters :: DEBUG :: launched thread: 2022-08-03 16:15:16,078 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,088 :: harvesters :: INFO :: started acquisition: 2022-08-03 16:15:16,088 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,089 :: harvesters :: DEBUG :: started streaming: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911b10> > 2022-08-03 16:15:16,100 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,106 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f19115d0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911cc0> > 2022-08-03 16:15:16,110 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,110 :: harvesters :: DEBUG :: fetched: 0 (#1); *' at 0x7ff3f1911b40> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911e40> > 2022-08-03 16:15:16,122 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,134 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911a80> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911bd0> > 2022-08-03 16:15:16,134 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,138 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,149 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,160 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,161 :: harvesters :: DEBUG :: fetched: 1 (#2); *' at 0x7ff3f1911ae0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911e10> > 2022-08-03 16:15:16,161 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bd80> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19115d0> > 2022-08-03 16:15:16,177 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,177 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,189 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,201 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,203 :: harvesters :: DEBUG :: fetched: 1 (#2); *' at 0x7ff3f1911a80> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911db0> > 2022-08-03 16:15:16,214 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911c90> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911b70> > 2022-08-03 16:15:16,219 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,219 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,236 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,236 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,247 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,252 :: harvesters :: DEBUG :: fetched: 2 (#3); *' at 0x7ff3f1911ae0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911de0> > 2022-08-03 16:15:16,265 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bd80> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911ae0> > 2022-08-03 16:15:16,265 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,269 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,281 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,294 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,298 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,303 :: harvesters :: DEBUG :: fetched: 2 (#3); *' at 0x7ff3f1911c60> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911d80> > 2022-08-03 16:15:16,303 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911cf0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19119f0> > 2022-08-03 16:15:16,320 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,321 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,333 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,337 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,349 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,352 :: harvesters :: DEBUG :: fetched: 0 (#4); *' at 0x7ff3f1911d20> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911cc0> > 2022-08-03 16:15:16,364 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,365 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bde0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911b10> > 2022-08-03 16:15:16,375 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,389 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,392 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,403 :: harvesters :: DEBUG :: fetched: 0 (#4); *' at 0x7ff3f1911a50> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911de0> > 2022-08-03 16:15:16,412 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911960> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911d50> > 2022-08-03 16:15:16,414 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,433 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,434 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,446 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,451 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,452 :: harvesters :: DEBUG :: fetched: 1 (#5); *' at 0x7ff3f1911600> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911e70> > 2022-08-03 16:15:16,459 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f19119c0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911a50> > 2022-08-03 16:15:16,467 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,470 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,482 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,492 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,503 :: harvesters :: DEBUG :: fetched: 1 (#5); *' at 0x7ff3f1911c90> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911cc0> > 2022-08-03 16:15:16,509 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bd80> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911540> > 2022-08-03 16:15:16,514 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,530 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,530 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,541 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,553 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,553 :: harvesters :: DEBUG :: fetched: 2 (#6); *' at 0x7ff3f1911ba0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911e10> > 2022-08-03 16:15:16,553 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911360> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19113c0> > 2022-08-03 16:15:16,566 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,570 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,581 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,591 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,602 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,603 :: harvesters :: DEBUG :: fetched: 2 (#6); *' at 0x7ff3f1911cf0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911db0> > 2022-08-03 16:15:16,603 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bd80> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911b70> > 2022-08-03 16:15:16,616 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,620 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,643 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,644 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,652 :: harvesters :: DEBUG :: fetched: 0 (#7); *' at 0x7ff3f1911ba0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911de0> > 2022-08-03 16:15:16,653 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911bd0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911990> > 2022-08-03 16:15:16,656 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,673 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,673 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,684 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,696 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,701 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,713 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,714 :: harvesters :: DEBUG :: fetched: 0 (#7); *' at 0x7ff3f1911cf0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911ae0> > 2022-08-03 16:15:16,721 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bde0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19115d0> > 2022-08-03 16:15:16,725 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,735 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,746 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,758 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,761 :: harvesters :: DEBUG :: fetched: 1 (#8); *' at 0x7ff3f1911bd0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911de0> > 2022-08-03 16:15:16,767 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f19119c0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19112a0> > 2022-08-03 16:15:16,786 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,786 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,797 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,808 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,808 :: harvesters :: DEBUG :: fetched: 1 (#8); *' at 0x7ff3f1911960> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911e10> > 2022-08-03 16:15:16,809 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bde0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911c30> > 2022-08-03 16:15:16,823 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,827 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,837 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,849 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,853 :: harvesters :: DEBUG :: fetched: 2 (#9); *' at 0x7ff3f1911bd0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911ae0> > 2022-08-03 16:15:16,863 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,864 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bd80> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911a50> > 2022-08-03 16:15:16,874 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,885 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,903 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,903 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,909 :: harvesters :: DEBUG :: fetched: 2 (#9); *' at 0x7ff3f1911c00> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911e70> > 2022-08-03 16:15:16,914 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,920 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911960> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911d20> > 2022-08-03 16:15:16,926 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,931 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,942 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,952 :: harvesters :: DEBUG :: fetched: 0 (#10); *' at 0x7ff3f1911600> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911ae0> > 2022-08-03 16:15:16,953 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,953 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f19119c0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911c00> > 2022-08-03 16:15:16,971 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,975 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,986 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:16,997 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,007 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,008 :: harvesters :: DEBUG :: fetched: 0 (#10); *' at 0x7ff3f1911420> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911ae0> > 2022-08-03 16:15:17,015 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911b40> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911cf0> > 2022-08-03 16:15:17,022 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,026 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,048 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,054 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,060 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,061 :: harvesters :: DEBUG :: fetched: 1 (#11); *' at 0x7ff3f1911600> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911db0> > 2022-08-03 16:15:17,072 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,073 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f190bde0> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911600> > 2022-08-03 16:15:17,084 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,089 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,101 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,103 :: harvesters :: DEBUG :: fetched: 1 (#11); *' at 0x7ff3f1911540> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911e70> > 2022-08-03 16:15:17,103 :: harvesters :: DEBUG :: queued: *' at 0x7ff3f1911960> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911ab0> > 2022-08-03 16:15:17,104 :: harvesters :: DEBUG :: terminated thread: 2022-08-03 16:15:17,104 :: harvesters :: DEBUG :: going to join thread: 2022-08-03 16:15:17,115 :: harvesters :: DEBUG :: joined thread: 2022-08-03 16:15:17,117 :: harvesters :: DEBUG :: stopped streaming: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19119f0> > 2022-08-03 16:15:17,118 :: harvesters :: DEBUG :: revoked: *' at 0x7ff3f190be70> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f19112a0> > 2022-08-03 16:15:17,118 :: harvesters :: DEBUG :: revoked: *' at 0x7ff3f190bf30> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911a50> > 2022-08-03 16:15:17,118 :: harvesters :: DEBUG :: revoked: *' at 0x7ff3f190bf90> > :: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911ab0> > 2022-08-03 16:15:17,119 :: harvesters :: INFO :: stopped acquisition: 2022-08-03 16:15:17,319 :: harvesters :: DEBUG :: timeout: elapsed 0.01 sec. 2022-08-03 16:15:17,319 :: harvesters :: DEBUG :: closed: MER-131-75GM-P(00:21:49:03:ae:27)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:27) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911540> > 2022-08-03 16:15:17,357 :: harvesters :: DEBUG :: fetched: 2 (#12); *' at 0x7ff3f1901b40> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911420> > 2022-08-03 16:15:17,359 :: harvesters :: INFO :: released resources: 2022-08-03 16:15:17,360 :: harvesters :: DEBUG :: terminated thread: 2022-08-03 16:15:17,360 :: harvesters :: DEBUG :: going to join thread: 2022-08-03 16:15:17,361 :: harvesters :: DEBUG :: joined thread: 2022-08-03 16:15:17,363 :: harvesters :: DEBUG :: stopped streaming: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f190be70> > 2022-08-03 16:15:17,365 :: harvesters :: DEBUG :: revoked: *' at 0x7ff3f19114e0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911420> > 2022-08-03 16:15:17,366 :: harvesters :: DEBUG :: revoked: *' at 0x7ff3f19114b0> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911a20> > 2022-08-03 16:15:17,367 :: harvesters :: DEBUG :: revoked: *' at 0x7ff3f1911720> > :: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911d50> > 2022-08-03 16:15:17,367 :: harvesters :: INFO :: stopped acquisition: 2022-08-03 16:15:17,565 :: harvesters :: DEBUG :: closed: MER-131-75GM-P(00:21:49:03:ae:25)_Stream_0 :: MER-131-75GM-P(00:21:49:03:ae:25) :: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: *' at 0x7ff3f1911720> > 2022-08-03 16:15:17,623 :: harvesters :: INFO :: released resources: 2022-08-03 16:15:17,623 :: harvesters :: DEBUG :: being reset: 2022-08-03 16:15:17,623 :: harvesters :: INFO :: flushed file list: 2022-08-03 16:15:17,624 :: harvesters :: DEBUG :: discarded device information: 2022-08-03 16:15:17,627 :: harvesters :: DEBUG :: closed: 00:0c:29:b2:d9:ce_ens33 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:17,627 :: harvesters :: DEBUG :: closed: /devices/pci0000:00/0000:00:11.0/0000:02:00.0/usb3 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:17,628 :: harvesters :: DEBUG :: closed: /devices/pci0000:00/0000:00:11.0/0000:02:02.0/usb1 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:17,628 :: harvesters :: DEBUG :: closed: /devices/pci0000:00/0000:00:15.0/0000:03:00.0/usb2 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:17,629 :: harvesters :: DEBUG :: closed: /devices/pci0000:00/0000:00:15.0/0000:03:00.0/usb4 :: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:17,630 :: harvesters :: DEBUG :: closed: {AF542A5A-E6D3-4f3d-9908-4A89AE21105A} :: :: None 2022-08-03 16:15:17,634 :: harvesters :: DEBUG :: closed: libs/gentl_producers/mvGenTLProducer_linux.cti 2022-08-03 16:15:17,635 :: harvesters :: INFO :: reset completed: . ---------------------------------------------------------------------- Ran 1 test in 3.411s OK ```

Some details that may be useful:

• Harvesters version 1.3.8
• Python version 3.8.12 (Linux) 3.8.10 (Windows)
• OS Windows 10 - Debian 11 kernel 5.10
• GenTL Producer mvGenTL_Acquire.cti version 2.46 from Matrix Vision
• Cameras 2x Daheng Imaging MER-131-75GM-P Monochrome 1280x1024 @ 10 fps

[michezio]

UPDATE

After some testing I found out that the usage of the method .wait() on a multiprocessing.Event increases the chances of the fetch method to get stuck. After I removed all the wait methods from my original code it got stuck with the same probability as the demo code I proposed, ~5-10% of the time. I am thinking of some sort of "desynchronization" happening between the process and the GenTL producer.

Also I tried to use run_as_thread=True just to check it out. It shows the same apparently random behavior, but this time, when relaunching the script after it got stuck and force quit, I always get this error:

Traceback (most recent call last):
  File "testing/harvesters_fetch_buffer_bug.py", line 72, in camera_worker
    ia = self.harvester.create(camera_id)
  File "/home/test/.venv/lib/python3.8/site-packages/harvesters/core.py", line 2899, in create
    return self._create_acquirer(device_proxy=device_proxy, config=config)
  File "/home/test/.venv/lib/python3.8/site-packages/harvesters/core.py", line 2916, in _create_acquirer
    device_proxy.open(_privilege)
  File "/home/test/.venv/lib/python3.8/site-packages/harvesters/core.py", line 214, in m
    return getattr(self._source_object, attribute)(*args)
  File "/home/test/.venv/lib/python3.8/site-packages/genicam/gentl.py", line 3189, in open
    return _gentl.Device_open(self, accessFlags)
_gentl.AccessDeniedException: GenTL exception: Requested operation is not allowed. (Message from the source: ) (ID: -1005)

which I can only recover from by killing all the active Python processes (eg using pkill python)

[laserK3000]

@michezio Did you solve your problem?

[michezio]

Unfortunately I'm not working on that project anymore.

Since I couldn't find a solution and it was an application that needed to be working 24/7 unmanaged, I "solved" it implementing a watchdog process that monitors the heartbeat of the processes using image acquirers. When they get stuck the watchdog kills and restarts them.

Actually I decided to reboot the whole system when It happened, since even when restarting the processes, sometimes the GenTL producer was unusable, so a full reboot was the only option (and in my case it was just max 15 seconds of down-time and was much more reliable).

[y2k0999]

I believe I am getting this error too. Except instead of 5-10% of the time as mentioned above, it happens every single time when I have the fps of my camera (~4000*3000 pixels) >1. If I set the fps to 1 then I don't get this error.