fetch_buffer keeps running until kernel dies and I can't get the data from hyperspectral camera

[AlmeidaAlin3]

Describe the Bug I have the hyperspectral camera Specim FX17, GigE version. The camera captures hyperspectral images and I need: (i) to capture images continuously and save them in the hard drive disk, with all the spectral layers (not only 3, like and RGB would be); and (ii) to capture the images and send to my python code so I can classify them with my A.I. models.

My camera do not have .cti files, but I'm trying to use the software of another camera vendor (specifically the Baumer camera software, which the Harvester site validates) and see if it is compatible with acquiring from the Specim camera; I expect it to work, given that everything should be truly GigEVision compliant and not manufacturer-specific.

When I add the Baumer .cti file and run the h.device_info_list[0] I get the Specim info printed into my code:

There are 1 devices in the list.
 first device:
 (id_='devicemodul70_f8_e7_b0_02_b3', vendor='Specim', model='FX17e', tl_type='GEV', user_defined_name=None, serial_number='076900041531', version=None)

Also, the Windows Task Manager shows a brief activity trough the Ethernet port which the camera is connected.

When I run ia.start_acquisition() sometimes the kernel dies, with no error messages. When it do not crashes and I try to get images using ia.fetch_buffer(is_raw=True), the code remains running for some time, showing no activity through Ethernet port, until eventually the kernel dies.

As my data is expected to be hyperspectral, I would need to reshape it after data acquisition, into many layers, but that's not a problem, once I could get the raw data.

I'd love to know if someone has a similar configuration or could give me some advice on how could I successfully get the data from the camera.

To Reproduce To reproduce the behavior I run this code in Jupyter Notebook on Anaconda with Python 3.6, Windows 10:

from harvesters.core import Harvester
import numpy as np

h = Harvester()

h.add_file(r'C:\...\bgapi2_gige.cti') #my path to cti file
h.update()
ia = h.create_image_acquirer(list_index=0)

ia.remote_device.node_map.get_node('AcquisitionMode').value = 'SingleFrame'

# Then start image acquisition.
ia.start_acquisition(run_in_background = True)
ia.fetch_buffer(is_raw=True)
with ia.fetch_buffer(is_raw=True) as buffer:
    component = buffer.payload.components[0]

Expected Behavior I expect that the kernel do not die and that running ia.fetch_buffer(is_raw=True) and then buffer.payload.components[0] I could acquire raw data captured by the camera.

Screenshots Here's the ethernet activity;

• 1st peak happens when I run ia = h.create_image_acquirer(list_index=0)
• 2nd peak happens when I run ia.start_acquisition() (which crashes sometimes)

Configuration

• OS: Windows 10
• Python: 3.6
• Harvester: 1.3.2]
• Camera: Specim FX17

Reproducibility

This phenomenon can be reproduced by other versions:

• [ ] Yes

Actions You Have Taken

• [X] I've read the Harvester FAQ page.

Additional context Add any other context about the problem here.

[kazunarikudo]

@AlmeidaAlin3 Hi, this is just out of curiosity, could you tell me why your code has two fetch_buffer calls? Is that intentional, or just a typo?

ia.fetch_buffer(is_raw=True)
with ia.fetch_buffer(is_raw=True) as buffer:
    component = buffer.payload.components[0]

Regards, Kazunari.

[kazunarikudo]

If you have an option, I’d recommend to give it a try with a GenTL Producer from MATRIX VISION.

[kazunarikudo]

And, even if your camera is a hyper-spectral, it can be reasonably arranged by Harvster if the layout follows the standard. Of course, I admit a case where it can’t be parsed due to a defect that Harvster has. If so, please excuse me!

[kazunarikudo]

Though it is not the issue at all, here's one more tip; you can rewrite the following code:

ia.remote_device.node_map.get_node('AcquisitionMode').value = 'SingleFrame'

with:

ia.remote_device.node_map.AcquisitionMode.value = 'SingleFrame'

There could have been a reason but this way should make your code more or less compact.

[kazunarikudo]

The fetch_buffer method call tries to wait until data is delivered to a buffer by default. The reason why no data is delivered is not the responsibility of the GenTL consumer side where Harvester resides; the data may not be delivered because a packet was dropped by an unexpected accident. If you can accept such a technical difficulty then you can pass the timeout parameter to the fetch_buffer method call. The unit is second, so if you call fetch_buffer(timeout=1.5) and no data is delivered until it passes 1.5 seconds, then it will throw genicam.gentl.TimeoutException. If you are not willing to catch any exception, then please give it a try with the try_fetch method. It also takes the timeout parameter, the unit is the same. It will return None instead of throwing an exception.

[kazunarikudo]

Note that the start_acquisition(run_in_background=True) method call lets an image acquisition thread run in the background; in the thread, the acquirer will keep fetching data until the image acquisition is stopped. If you do not have any particular reason to keep it running, just calling start_acquisition() should be the way to go.

[AlmeidaAlin3]

Hey @kazunarikudo! I appreciate your help :)

---

@AlmeidaAlin3 Hi, this is just out of curiosity, could you tell me why your code has two fetch_buffer calls? Is that intentional, or just a typo?

oops! it was a typo, sorry

---

If you have an option, I’d recommend to give it a try with a GenTL Producer from MATRIX VISION.

I tried it, but I'm not able to interpret the results.. One example of the files I get is:

_IMG00000000000000014515.480x224.Mono12PackedV2.raw.

when I open it using:

data = np.fromfile(fileobj,dtype=np.uint16)
data.size

I get a len of 80896 values; this is even multiple of 224 (my camera device has 224 channels)

Also the values are something like:

array([11532,  3084,  3121, 13324,  3084,  3142,   268,  3084,  3104,
       13068], dtype=uint16)

if I plot the results, trying to reshaping it, somehow (breaking each line with size 316 values, for example): The real image is a piece of rectangular plastic.

I'm just starting to test the suggestions for harvesters and keep you updated! thank you! :)

[kazunarikudo]

Hi, thank you for the update. Here are two questions:

1. Could you tell me the expected pixel format name? Which value do you set to node_map.PixelFormat.value? (Or, you could read the value if you do not explicitly set any value by yourself.)
2. Does the "kernel dies" phenomenon still occur?

[kazunarikudo]

I'd interpret that the data is Mono12Packed as far as I see the result from MATRIX VISION but I would like to make sure if it is the right answer you want to get.

[kazunarikudo]

One more question:

with ia.fetch() as buffer:
    print(buffer.payload.components)

Which number will the above code show you? I do not know about the camera you have but I guess it should consist of some (maybe 224?) components.

[kazunarikudo]

In addition, with the following code you can check how the payload is interpreted by the GenTL Producer:

with ia.fetch() as buffer:
    print(buffer.payload.components[0])

For example, a typical Mono8 payload, you will see:

512 x 512, Mono8, 262144 elements,
[  0   1   2 ... 252 253 254]

By the way, to achieve the expected result, we need to understand in advance that:

1. The camera must follow the GigE vision so that any GenTL Producer can parse the hyper-spectral data as expected, and
2. The GenTL Producer must be able to parse the hyper-spectral data as expected so that it can provide sufficient information to its consumer such as Harvester.

As far as I know, Harvester can handle multiple components; there could be an exception but I do not see a use case that brings up multiple components is not special at all. However, I know I will never be able to tell you that Harvester is perfect; please excuse me in advance!

If you fail at case no. 1, then it would take some time until they fix it; otherwise, you will need to stick with the SDK the camera manufacturer offers. If you fail at case no. 2, then you will need to try out another GenTL Producer but it may be more or less promising. If the issue is in Harvester, it should be much easier and I will try to take some time to fix the issue.

[kazunarikudo]

@AlmeidaAlin3 Hi, I have just resolved an issue that was revealed by the same camera model as yours. See #318. I have not yet officially released a version that fixes the issue but you can give that a try on the master branch. Thanks.

[AlmeidaAlin3]

Hi @kazunarikudo! what great news! 😍 🥇 thank you! I've been trying to use harvesters with this camera since I've posted but with no success. I'm still getting the jupyter freezing sometimes, though. @TouchTheFishy, we both are using FX17, I was wondering if would you mind sharing your code to access and get data from the camera, so I can test in my camera device? Also, what .cti file did you used? Thanks! 😄

[kazunarikudo]

Hi, concerning the issue where the fetch method call blocks you, this comment may improve the situation. As I mentioned there, Harvester can't prevent a situation where a buffer is dropped by accident but I would like to offer you a way to not have kept you waiting for a buffer that will never come.

[kazunarikudo]

@AlmeidaAlin3 Ah, excuse me, I had forgotten that I mentioned the timeout on a fetch method call. So do you mean it freezes even if the timeout parameter is set?

[AlmeidaAlin3]

Exactly, not always, but mostly of the time, it freezes then the kernel dies. Even if I set a high value for timeout, it crashes instantly.

[TouchTheFishy]

Hey @AlmeidaAlin3 I've worked quite extensively with both the FX17e and the FX10e and I know that they can be a piece of work 😅 I just need to check with my company if I'm allowed to share my code to outsiders but otherwise I can still give you some advices.

About the .cti I used the one from MvImpact. I don't know which OS you are using but on Ubuntu it was kinda tricky to find it once I installed it

[kazunarikudo]

@AlmeidaAlin3 Hi, excuse me, I had unintentionally interpreted that you've been observing "freeze" when it crashes. I have just finished reading the original description on the top and got some ideas for further investigation:

1. I can confirm the activity monitor says NI's name. Could you disable all of the filter drivers except MATRIX VISION? I do not want to involve unnecessary interference.
2. Could you give a Python >= 3.7 a try? Version 3.6 has reached its end of support at the end of last year.
3. Could you try running the script on ipython instead of Jupyter Notebook? I just would like to check the reproducibility.

I have one more thing that I had overlooked a fact that the crash can be observed when the start method call is executed. Once the call is executed, the buffers (memory) are allocated and the GenTL Producer starts to fill up the data in the background so I expect you should see constant network activity as long as the device transmits the data to the GenTL Producer. When the fetch call is executed, Harvester just points out the buffer that is ready to be passed to you so it does not involve any network activity because the data has been already delivered to your computer.

If you could put every Python script line into a *.py file and then execute the following command from Visual Studio (or other popular debuggers), then you could catch the exact location or library where the crash happened. However, it'd be too much for now. Let's try the above three items first.

[kazunarikudo]

@TouchTheFishy Hi, just a quick question: Have you ever observed any crashes on your setup so far? (I know you're working on Ubuntu but that's okay, I just would like to know a piece of reality.)

[TouchTheFishy]

Hey Kazunari, The only crashes I experienced were when I ran out of memory. But this happened only when I had quite large pictures (around 1600 x 1024 x 224 pixels) At that moment python just threw a 'Killed' at me and died. Other than that nothing as severe as a kernel crash.

[pers972]

Hey @AlmeidaAlin3 I've worked quite extensively with both the FX17e and the FX10e and I know that they can be a piece of work 😅 I just need to check with my company if I'm allowed to share my code to outsiders but otherwise I can still give you some advices.

About the .cti I used the one from MvImpact. I don't know which OS you are using but on Ubuntu it was kinda tricky to find it once I installed it

Please tell me what version of mvimpact do you use. I am using Specim FX10e and download mvimpact version 2.46.2. So my program freeze at step ia = h.create(0)

[kazunarikudo]

@pers972 Hi, I would appreciate it if you could check the reproducibility with other older MV versions; that would be sufficient if you could give it a try with the latest sub-minor version in every minor version you can download. Regards, Kazunari.

[kazunarikudo]

@pers972 Please use the latest Harvester version first for the suggested reproducibility test above.

[pers972]

@kazunarikudo I try MV 2.34.0 version and all working good. Thank you for your job!

[Eduardo-Assuncao-hub]

Hi all, I´m new to work with Harvester and the FX10e camera. I´m trying to perform acquisition of hyperspectral channels. For now, when I run the code, it can get the camera information, but after that it seems that the code freezes.
Follow some additional information:

OS: Windows 10 Harvesters version: 1.3.8 Python version: 3.7

Code: from harvesters.core import Harvester import numpy as np h = Harvester() h.add_file(r'C:\Program Files\MATRIX VISION\mvIMPACT Acquire\bin\x64\mvGenTLProducer.cti') h.update() print("device_info_list:", h.device_info_list) ia = h.create() with ia.fetch() as buffer: Print(buffer.payload.components[0])

Python console: device_info_list: [{'access_status': 1, 'displayname': 'Specim FX10e(70:f8:e7:b0:04:cb)', 'id': 'FX10e(70:f8:e7:b0:04:cb)', 'model': 'FX10e', 'parent': <genicam.gentl.Interface; proxy of <Swig Object of type 'std::shared_ptr< GenTLCpp::TLInterface > *' at 0x000001FF8F623B10> >, 'serial_number': '057000049029', 'tl_type': 'GEV', 'user_defined_name': ' ', 'vendor': 'Specim', 'version': 'Version 1.3'}]

I appreciate any help that you can give me.

[pers972]

Hi all,

I´m new to work with Harvester and the FX10e camera. I´m trying to perform acquisition of hyperspectral channels. For now, when I run the code, it can get the camera information, but after that it seems that the code freezes. Follow some additional information:

OS: Windows 10

Harvesters version: 1.3.8 Python version: 3.7

Code:

from harvesters.core import Harvester import numpy as np h = Harvester() h.add_file(r'C:\Program Files\MATRIX VISION\mvIMPACT Acquire\bin\x64\mvGenTLProducer.cti') h.update() print("device_info_list:", h.device_info_list) ia = h.create() with ia.fetch() as buffer: Print(buffer.payload.components[0])

Python console:

device_info_list: [{'access_status': 1, 'displayname': 'Specim FX10e(70:f8:e7:b0:04:cb)', 'id': 'FX10e(70:f8:e7:b0:04:cb)', 'model': 'FX10e', 'parent': <genicam.gentl.Interface; proxy of <Swig Object of type 'std::shared_ptr< GenTLCpp::TLInterface > *' at 0x000001FF8F623B10> >, 'serial_number': '057000049029', 'tl_type': 'GEV', 'user_defined_name': ' ', 'vendor': 'Specim', 'version': 'Version 1.3'}] I appreciate any help that you can give me.

Hi, @Eduardo-Assuncao-hub. I met with a similar problem. Try to change ia = h.create() on ia = h.create(0), or it will be problem with version of cti_file.

[Eduardo-Assuncao-hub]

Hi all,

I´m new to work with Harvester and the FX10e camera. I´m trying to perform acquisition of hyperspectral channels. For now, when I run the code, it can get the camera information, but after that it seems that the code freezes. Follow some additional information:

OS: Windows 10

Harvesters version: 1.3.8 Python version: 3.7

Code:

from harvesters.core import Harvester import numpy as np h = Harvester() h.add_file(r'C:\Program Files\MATRIX VISION\mvIMPACT Acquire\bin\x64\mvGenTLProducer.cti') h.update() print("device_info_list:", h.device_info_list) ia = h.create() with ia.fetch() as buffer: Print(buffer.payload.components[0])

Python console:

device_info_list: [{'access_status': 1, 'displayname': 'Specim FX10e(70:f8:e7:b0:04:cb)', 'id': 'FX10e(70:f8:e7:b0:04:cb)', 'model': 'FX10e', 'parent': <genicam.gentl.Interface; proxy of <Swig Object of type 'std::shared_ptr< GenTLCpp::TLInterface > *' at 0x000001FF8F623B10> >, 'serial_number': '057000049029', 'tl_type': 'GEV', 'user_defined_name': ' ', 'vendor': 'Specim', 'version': 'Version 1.3'}] I appreciate any help that you can give me.

Hi, @Eduardo-Assuncao-hub. I met with a similar problem. Try to change ia = h.create() on ia = h.create(0), or it will be problem with version of cti_file.

Hi @pers972. Thank you for the tip. The problem was the version of cti_file. I have another question that is not a Harvester issue . Do you have any information about how to convert Digital Number (DN) from the FX10e to reflectance?

[gsainsbury86]

I'm having similar problems with a SPCIM FX10. I tried using the current (3.03) mvAcquire and had the same problem - hanging when I try to create(). I downgraded to the most recent 2.X version (2.50.1) and wasn't able to see the device anymore after an h.update(). Finally I downgraded again to 2.34.0 as per @pers792's comment above and had the same problem (unable to connect).

Rolling forward again to 3.0.3, I still have the problem where it seems to hang indefinitely.

The below code will not print past "before create"

import os
from harvesters.core import Harvester

h = Harvester()

gentl_file = r"C:\Program Files\Balluff\ImpactAcquire\bin\x64\mvGenTLProducer.cti"

print(os.path.exists(gentl_file))

h.add_file(gentl_file)
h.update()
print(h.device_info_list)

print("before create")
with h.create() as ia:
    print("after create")
    with ia.fetch_buffer(is_raw=True) as buffer:
        print("after fetch")
        component = buffer.payload.components[0]
        print(component)