Document how to upload/download files to/from the camera

[aqc-carlodri]

What is the goal that you want to achieve by the request? Learn how to upload/download files to/from the camera using harvesters, for example color correction profiles or other LUTs.

Additional context I could not find anything in the documentation or examples showing how to do this.

[aqc-carlodri]

@kazunarikudo sorry for pinging you directly, do you have any insight or example about how to do this? This would greatly speed up our work with camera calibrations.

[jcormier]

Note there is no required standard interface for uploading/downloading files from a genicam camera. So this question is camera-dependent. Perhaps you could share the camera in questions xml and what parameter specifically you are trying to upload.

You really should be asking the camera's manufacturer instead.

[aqc-carlodri]

Thanks for the feedback @jcormier. I might be wrong but in the Genicam SFNC there is section 17 dedicated to File Access Control which to my understanding documents the feature naming standards for this process of uploading/downloading files from a camera.

My camera is a DALSA Linea 4K color camera.

[jcormier]

Okay so yes File Access control is a standardized interface however it is not a required one. So since your first post didn't ask about it specifically I assumed your camera didn't have it. Have you looked at your cameras xml and verified if the FileAccessControl node is there?

I think the SFNC outlines how its expected to be used. You should be able to read the FileSelector options to see which files are accessible.

[aqc-carlodri]

Yes indeed the FileAccessControl node is there and the camera supports it. The SFNC does mention some usage guidelines but they are too generic for me. I can see the FileSelector options and set the one that interests me, but then my problem is that I cannot find a way to retrieve the buffer.

This is what I have come up so far (and it works up to here with no errors):

rnmap = ia.remote_device.node_map
rnmap.FileSelector.value = "FlatFieldCoefficients0"  # the factory flat field coefficients
rnmap.FileOperationSelector.value = "Open"
rnmap.FileOpenMode.value = "Read"
rnmap.FileOperationExecute()

Then if I ask rnmap.FileAccessBuffer.length I get 32768.

Now I cannot find a way to successfully retrieve the buffer with rnmap.FileAccessBuffer.get(), or probably I don't understand how the get() and set() methods work. It could be that my question is more specific to Python/ctypes since I'm not sure how to pass the buffer to those methods (and what the buffer should be - a bytearray? a ctypes.byref() pointer? to what? ).

I tried something like this:

ffc_data = (ctypes.c_ubyte * rnmap.FileAccessBuffer.length)(0)
rnmap.FileAccessBuffer.get(ctypes.byref(ffc_data))

but I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[70], line 3
      1 ffc_data = (ctypes.c_ubyte * rnmap.FileAccessBuffer.length)(0)
      2 # ffc_data = ctypes.c_int64()
----> 3 rnmap.FileAccessBuffer.get(ctypes.byref(ffc_data))

File ~\AppData\Local\mambaforge\envs\data_analysis_env\Lib\site-packages\genicam\genapi.py:2229, in IRegister.get(self, buffer, Verify, IgnoreCache)
   2228 def get(self, buffer, Verify=False, IgnoreCache=False):
-> 2229     return _genapi.IRegister_get(self, buffer, Verify, IgnoreCache)

TypeError: Wrong number or type of arguments for overloaded function 'IRegister_get'.
  Possible C/C++ prototypes are:
    GENAPI_NAMESPACE::IRegister::Get(uint8_t *,int64_t,bool,bool)
    GENAPI_NAMESPACE::IRegister::Get(uint8_t *,int64_t,bool)
    GENAPI_NAMESPACE::IRegister::Get(uint8_t *,int64_t)

Thank you for any guidance you can give me on this.

[olofsjo1SICKAG]

I do not have the same camera as you have but in the past I have done something similar to this:

def read_file(node_map, filename):
    node_map.FileSelector.value = filename
    node_map.FileOpenMode.value = 'Read'
    node_map.FileOperationSelector.value = 'Open'
    node_map.FileOperationExecute.execute()

    offset = 0
    time.sleep(1)
    result = b''
    hunk_size = 4096
    while offset < node_map.FileSize.value:
        node_map.FileOperationSelector.value = 'Read'
        node_map.FileAccessOffset.value = offset
        node_map.FileAccessLength.value = hunk_size
        node_map.FileOperationExecute.execute()
        time.sleep(1)
        hunk = node_map.FileAccessBuffer.get(node_map.FileOperationResult.value)
        result += hunk
        offset += hunk_size
    time.sleep(1)
    node_map.FileOperationSelector.value = 'Close'
    node_map.FileOperationExecute.execute()
    time.sleep(1)
    return result

So it reads the buffer piece by piece, this size will probably vary depending on which camera you are using. Not sure if the sleeps are needed or not (probably not but maybe) and there might be an easier way or a different way in doing this as well. Write can be done in a similar manor.

I hope this can to some help for you.

[aqc-carlodri]

Thanks @olofsjo1SICKAG! I will try that right away.

UPDATE: sadly it doesn't work, it gives me a Node is not writable error at the node_map.FileOperationSelector.value = 'Read' at the beginning of the while loop.

[olofsjo1SICKAG]

And you know that the file in question are readable?

Otherwise I suggest you contact the camera manufacturer for assistance. Since I do not have knowledge about the camera you are using.

[aqc-carlodri]

Yes, the file is readable since I can download it with the DALSA proprietary software and also with the wxPropView MATRIXVISION tool.

[aqc-carlodri]

@olofsjo1SICKAG thanks to your code I was able to successfully read the file from the camera, thank you! The only change I needed was that I had to repeat the node_map.FileSelector.value assignment after every .execute(). The sleeps were also important. By the way, I could read the entire file in one single block by getting exactly node_map.FileSize.value bytes.

Now I'm trying to write a calibration file to the camera, and I'm having a hard time assigning the contents of the file to the buffer with the .set() method. Do you have a sample code for the write operation as well?

Thank you very much for your guidance.

What I have tried is:

with open(filename_to_upload, "rb") as f:
        raw_file_data = f.read()
# ...
node_map.FileAccessBuffer.set(raw_file_data)
node_map.FileOperationExecute.execute()

but what I get is the following:

InvalidParameterException: GenTL exception: Parameter not valid or out of range. (Message from the source: Could not write data to 0x8f00000(536 bytes). Status: GEV_STATUS_INVALID_HEADER.
) (ID: -1009)

[olofsjo1SICKAG]

Nice to hear that the read works for you. Yes I have an example on how a write can be done (once again you might have to adapt it for your camera):

def write_file(node_map, filename, buffer):
    node_map.FileSelector.value = filename
    node_map.FileOpenMode.value = 'Write'
    node_map.FileOperationSelector.value = 'Open'
    node_map.FileOperationExecute.execute()

    if not len(buffer) % 4 == 0:
        # Pad to multiple of 4
        b = bytearray(buffer)
        b.extend(bytearray(4 - (len(buffer) % 4)))
        buffer = bytes(b)
    offset = 0
    hunk_size = 4096
    time.sleep(1)
    while offset < len(buffer):
        node_map.FileOperationSelector.value = 'Write'

        node_map.FileAccessOffset.value = offset
        remaining_byte_count = len(buffer) - offset
        bytes_to_write = min(hunk_size, remaining_byte_count)
        print(f"Offset={offset}")
        print(f"Will write {bytes_to_write} bytes")

        node_map.FileAccessLength.value = bytes_to_write
        node_map.FileAccessBuffer.set(buffer[offset: offset + bytes_to_write])
        node_map.FileOperationExecute.execute()
        # time.sleep(1)
        offset += node_map.FileAccessLength.value
        if offset % (len(buffer) / 10) < hunk_size:
            print(f"File transfer progress: {int(100 * offset / len(buffer))}%")

    time.sleep(1)
    node_map.FileOperationSelector.value = 'Close'
    node_map.FileOperationExecute.execute()
    time.sleep(1)

[aqc-carlodri]

Thanks! That worked perfectly. The caveat that got me stuck was that while the download can be carried out in a single chunk, the upload needs to be chunked in 512 bytes, at least for my camera (DALSA Linea 4k Color).

For the rest, as above, I needed to "repeat" the FileSelector and FileOperationSelector assignments for each .execute() statement. Moreover, I had to add a 5ms sleep inside the while loop after the .execute() statement to avoid nodes not being writeable.

Thanks again @olofsjo1SICKAG, your help was invaluable!

[olofsjo1SICKAG]

The chunk size will vary from camera to camera and apparently some other implementation details regarding timing and selector setting as well. Happy to be of help and good luck with you application.

[thiesmoeller]

The genicam bindings contain a helper class FileAccess. Which is a friendly abstraction on top of the genicam c++ Filestream classes. Implementation is like your code above ;-)

Usage is roughly:

# filenames from ia.remote_device.node_map.FileSelector

gfa = genicam.FileAccess()

gfa.open(ia.remote_device.node_map, <filename>, 'r')
data = gfa.read()
gfa.close()

#or

gfa.open(ia.remote_device.node_map, <filename>, 'w')
gfa.write(data)
gfa.close()

[aqc-carlodri]

wow @thiesmoeller thanks! That looks much more comfortable, will try ASAP.

[aqc-carlodri]

hmm, it appears it doesn't like the node map argument:

gfa = genicam.genapi.FileProtocolAdapter.FileAccess()
gfa.open(ia.remote_device.node_map, "FlatFieldCoefficients0", "r")

gives:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[14], line 1
----> 1 gfa.open(ia.remote_device.node_map, "FlatFieldCoefficients0", "r")

File ...\Lib\site-packages\genicam\genapi.py:3128, in FileProtocolAdapter.FileAccess.open(self, nodemap, filename, openmode)
   3127 def open(self, nodemap, filename, openmode):
-> 3128     self.fpa.attach(nodemap)
   3129     self.filename = filename
   3130     self.fpa.openFile(filename,openmode)

File ...\Lib\site-packages\genicam\genapi.py:3094, in FileProtocolAdapter.attach(self, pInterface)
   3093 def attach(self, pInterface):
-> 3094     return _genapi.FileProtocolAdapter_attach(self, pInterface)

TypeError: in method 'FileProtocolAdapter_attach', argument 2 of type 'GENAPI_NAMESPACE::INodeMap *'

any suggestion @thiesmoeller?

[thiesmoeller]

@aqc-carlodri ... really sorry to say, that this won't work on the official genicam python version :-(

I wrote my comment from the base of a "this was working", it seems, that for harvesters/genicam side, no one ever tested this before.

The file access code has been developed by Basler AG years ago for the official genicam python bindings.

We forked out from the official python genicam repo to keep our naming conventions. Along the way we fixed and updated the SWIG mapping code and update to python > 3.x. ( https://github.com/basler/pypylon/blob/master/src/genicam/FileProtocolAdapter.i )

The changes are minimal, and are easiest to implement if applied on the SWIG/binary portion.

Pretty sure, that you can write a modified FileAccess Class that talks to the existing FileProtocolAdapter methods. Maybe some ctypes required ....

[aqc-carlodri]

No problem @thiesmoeller, thanks for the insight and the reference to the Basler fork of genicam.

[JacobBothell]

@aqc-carlodri sorry to resurrect an old thread, but how were you decoding the files once you had the bytes... trying to get a flatfield correction read in and out of a Teledyne Dalsa GigE camera

[aqc-carlodri]

Hi @JacobBothell, the bytes you read are a TIFF file, exactly as if you had read it from your hard drive. So you can decode them with anything that understands TIFF, e.g. from skimage.io import imread.