Jetson Nano Slow Image Acquisition

[NatePolizogo]

Hello, I am trying to use the VimbaPythonAPI with an Allied Vision ALVIUM 1800 U-1236c color camera alongside with a jetson nano. I've written a asynchronous acquisition script and it runs just fine on desktop, solid 7 fps. Trying to run the same code on nano it makes my code freeze after a few frames. I've edited '/boot/extlinux/extlinux.conf' by adding usbcore.usbfs_memory_mb=1024 but it did not solved the problem.

[NiklasKroeger-AlliedVision]

Freezing is very unexpected. I have only worked with the Nano for some short tests about a year ago and I remember also being slightly underwhelmed by the USB speed, but a complete freeze should not happen and to me indicates a problem with the code you are trying to run on the board.

In order to see if there is still room in the settings to improve the streaming speed it would be helpful if you could list the following settings.

• Installed Vimba version (most recent as of right now is Vimba 4.2. this did include some performance improvements for Linux system compared to older Vimba versions. So if your Vimba version is older please consider upgrading)
• Value of MaxTransferSize in your VimbaUSBTL.xml file (file is located next to the VimbaUSBTL.cti file. See content of the GENICAM_GENTL64_PATH environment variable to find out where exactly the transport layer is installed)
• The reported value of the feature DeviceLinkSpeed for the camera you are trying to use.

Regarding the freezing you are reporting, as I said I feel this might be more related to the code you are trying to run. Perhaps your code is only receiving incomplete frames and you are silently ignoring those so it looks like nothing is happening? To troubleshoot this a minimal code example would be very helpful.

I would however like to point out, that this does not appear to be a VimbaPython issue, but rather a more general Connection/Vimba issue. For those our support team is likely more experienced in handling the issue and can provide further insight. They might also have some further ideas on how to troubleshoot this that I am not aware of as I am not as experienced with embedded systems. So please feel free to also contact our support via the form on our website.

[NiklasKroeger-AlliedVision]

After writing this I noticed, that I never suggested the use of the DeviceLinkThroughputLimit feature. This is a feature, that reduces the amount of data sent (thereby also decreasing the number of recorded frames per second) to not overload the used communication bus. If the problem stems from incomplete frames due to too high bandwidth demands, it might be a helpful setting for you to reduce in order to be able to stream more than a few frames. I suggest you try reducing the value of this setting and see if it does improve the number of frames you are receiving. The 1236c is a camera which does need quite some bandwidth to transfer its data and your streaming might therefore be limited by the available bandwidth on the USB bus.

[NiklasKroeger-AlliedVision]

While skimming our knowledge base to answer #30 I remembered we also provide a short pdf on optimizing the Performance for Jetson boards. This does include a small section related to USB performance which might also be of interest for you. I do however believe that upgrading to Vimba 4.2 should be the first thing you attempt due to the mentioned improvements to the USB transport layer in that version.

[NatePolizogo]

I run the same code on a desktop pc and it works just fine. No incomplete frames and solid 7 fps. After some debuging I found out that the program does not freeze but all the frames acquired are incomplete, that's the reason I thought it freezed. Also, when I connect to nano through ssh on Visual Studio Code, that triggers incomple frames too. I use Vimba 4.1, I'd prefer to not updrage because I've developed an API on this version (I want to avoid incompatibilities between 4.1 and 4.2), but if the issue insists I'll do it. Although I'd prefer a solution without changing version.

The VimbaUSBTL.xml file looks like this: `<?xml version="1.0" standalone="no" ?>

True True ` As for the DeviceLinkSpeed, I don't know how to see it. My excact code is following: `"""BSD 2-Clause License Copyright (c) 2019, Allied Vision Technologies GmbH All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THE SOFTWARE IS PRELIMINARY AND STILL IN TESTING AND VERIFICATION PHASE AND IS PROVIDED ON AN “AS IS” AND “AS AVAILABLE” BASIS AND IS BELIEVED TO CONTAIN DEFECTS. A PRIMARY PURPOSE OF THIS EARLY ACCESS IS TO OBTAIN FEEDBACK ON PERFORMANCE AND THE IDENTIFICATION OF DEFECT SOFTWARE, HARDWARE AND DOCUMENTATION. """ import threading import sys import cv2 from typing import Optional from vimba import * import time import pdb def print_preamble(): print('///////////////////////////////////////////////////////') print('/// Vimba API Asynchronous Grab with OpenCV Example ///') print('///////////////////////////////////////////////////////\n') def print_usage(): print('Usage:') print(' python asynchronous_grab_opencv.py [camera_id]') print(' python asynchronous_grab_opencv.py [/h] [-h]') print() print('Parameters:') print(' camera_id ID of the camera to use (using first camera if not specified)') print() def abort(reason: str, return_code: int = 1, usage: bool = False): print(reason + '\n') if usage: print_usage() sys.exit(return_code) def parse_args() -> Optional[str]: args = sys.argv[1:] argc = len(args) for arg in args: if arg in ('/h', '-h'): print_usage() sys.exit(0) if argc > 1: abort(reason="Invalid number of arguments. Abort.", return_code=2, usage=True) return None if argc == 0 else args[0] def get_camera(camera_id: Optional[str]) -> Camera: with Vimba.get_instance() as vimba: if camera_id: try: return vimba.get_camera_by_id(camera_id) except VimbaCameraError: abort('Failed to access Camera \'{}\'. Abort.'.format(camera_id)) else: cams = vimba.get_all_cameras() if not cams: abort('No Cameras accessible. Abort.') return cams[0] def setup_camera(cam: Camera): with cam: # Enable auto exposure time setting if camera supports it try: cam.ExposureAuto.set('Off') except (AttributeError, VimbaFeatureError): #pdb.set_trace() pass # # Enable white balancing if camera supports it try: cam.BalanceWhiteAuto.set('Off') except (AttributeError, VimbaFeatureError): #pdb.set_trace() pass # Try to adjust GeV packet size. # This Feature is only available for GigE - Cameras. # try: # cam.GVSPAdjustPacketSize.run() # # while not cam.GVSPAdjustPacketSize.is_done(): # pass # # except (AttributeError, VimbaFeatureError): # pass # Query available, open_cv compatible pixel formats # prefer color formats over monochrome formats try: cv_fmts = intersect_pixel_formats(cam.get_pixel_formats(), OPENCV_PIXEL_FORMATS) color_fmts = intersect_pixel_formats(cv_fmts, COLOR_PIXEL_FORMATS) if color_fmts: cam.set_pixel_format(color_fmts[0]) else: mono_fmts = intersect_pixel_formats(cv_fmts, MONO_PIXEL_FORMATS) if mono_fmts: cam.set_pixel_format(mono_fmts[0]) else: abort('Camera does not support a OpenCV compatible format natively. Abort.') except: pdb.set_trace() class Handler: def __init__(self): self.shutdown_event = threading.Event() self.current_frame = None self.start = time.time() def __call__(self, cam: Camera, frame: Frame): ENTER_KEY_CODE = 13 key = cv2.waitKey(1) if key == ENTER_KEY_CODE: self.shutdown_event.set() return elif frame.get_status() == FrameStatus.Complete: self.current_frame = frame self.end = time.time() self.frame_time = self.end - self.start self.start = self.end print("FPS: " + str(1/self.frame_time) + " " + '{} acquired {}'.format(cam, frame), flush=True) img = frame.as_opencv_image() # msg = 'Stream from \'{}\'. Press to stop stream.' # font = cv2.FONT_HERSHEY_SIMPLEX # cv2.putText(img, "FPS: " + str(1/self.frame_time) + " " + str(frame), (10,450), font, 3, (0, 255, 0), 2, cv2.LINE_AA) # cv2.namedWindow(msg.format(cam), cv2.WINDOW_NORMAL) # cv2.resizeWindow(msg.format(cam), 1200, 900) # cv2.imshow(msg.format(cam), img) # cv2.imwrite(str(frame)+".jpg", img) # time.sleep(2) cam.queue_frame(frame) print('Frame Status: '.format(frame.as_opencv_image())) def get_img(self): if self.current_frame == None: return None return self.current_frame.as_opencv_image() def main(): print_preamble() cam_id = parse_args() with Vimba.get_instance(): with get_camera(cam_id) as cam: settings_file = 'settings/continuous_BGR8.xml' # Restore settings to initial value. try: cam.UserSetSelector.set('Default') except (AttributeError, VimbaFeatureError): abort('Failed to set Feature \'UserSetSelector\'') try: cam.UserSetLoad.run() print("--> All feature values have been restored to default") except (AttributeError, VimbaFeatureError): abort('Failed to run Feature \'UserSetLoad\'') # Start Streaming, wait for five seconds, stop streaming setup_camera(cam) # Load camera settings from file. # cam.load_settings(settings_file, PersistType.All) # print("--> Feature values have been loaded from given file '%s'" % settings_file) handler = Handler() # print(cam.DeviceLinkThroughoutLimit) try: # Start Streaming with a custom a buffer of 10 Frames (defaults to 5) cam.start_streaming(handler=handler, buffer_count=25) # while handler.get_img() == None: # # print("Not ready") # pass # time.sleep(2) # print("started") # while True: # msg = 'Stream from \'{}\'. Press to stop stream.' # cv2.namedWindow(msg.format(cam), cv2.WINDOW_NORMAL) # cv2.resizeWindow(msg.format(cam), 1200, 900) # cv2.imshow(msg.format(cam), handler.get_img()) handler.shutdown_event.wait() print(1) finally: cam.stop_streaming() if __name__ == '__main__': main() `

[NiklasKroeger-AlliedVision]

I use Vimba 4.1, I'd prefer to not updrage because I've developed an API on this version (I want to avoid incompatibilities between 4.1 and 4.2), but if the issue insists I'll do it. Although I'd prefer a solution without changing version.

I am not aware of any breaking changes from Vimba4.1 to Vimba4.2 so you should be safe to update with no need for you to change your API. But I do understand, that you are hesitant.

The first thing I would suggest you try is changing the value of MaxTransferSize in the VimbaUSBTL.xml file to use a value of 262144 (this is also the new default value in Vimba4.2 and above for Linux). In my experience this already had a great impact on the USB streaming performance. Also remember to uncomment the actual setting itself in that file (xml comments are indicated by <!-- THIS IS COMMENTED OUT --> ). Please perform the following change to apply the larger value for MaxTransferSize:

-    <!-- <MaxTransferSize>262144</MaxTransferSize> -->
+    <MaxTransferSize>262144</MaxTransferSize>

I hope this stabilizes your USB streaming. However you should definitely handle incomplete frames in your code as they can happen during normal operation. USB does not provide bandwidth guarantees and might occasionally fail to transfer all data for a frame in time. So having code that is able to work around such cases is definitely recommended.

One last thing: If you share code (especially larger sections like the VimbaUSBTL.xml file or your example code) you can let Github know so it will display the text in a more readable fashion (monotype, no line breaks, optionally with syntax highlighting) by surrounding it with three backticks (```) like this:

```
<your code goes in here>
```

More info and possibly other helpful tips can be found on this github help page.

[NatePolizogo]

I'm sorry for the code I was not aware about that syntax, yes I agree that I have to handle incomplete frames, but it does not help me somehow if all the frames are incomplete. Anyway, I'm flashing a fresh sdcard for nano right now to try the new Vimba version. By the way, I am using this camera for ML tasks and so I am using nvidia's Tensor RT that comes with nano's jetpack in python 3.6. It is really imposible to reinstall in on a later python version, and I had to make a patch file to make Vimba SDK run on python 3.6 last time. Is there any posibility to support python 3.6 for aarch64 devices as nano and jetson in general, that comes with all the nvidia goodies ?

[NiklasKroeger-AlliedVision]

I agree that I have to handle incomplete frames, but it does not help me somehow if all the frames are incomplete. Anyway, I'm flashing a fresh sdcard for nano right now to try the new Vimba version.

It is true, if all frames are incomplete then even handling them in a more elegant way would not be of any use. I sincerely hope, that Vimba 4.2 will help with the transmission as I have also experienced problems where only incomplete frames were transmitted in Vimba 4.1 and earlier. But since 4.2 the USB transfers on Linux improved for me dramatically. Have you already tried changing the MaxTransferSize in the VimbaUSBTL.xml file as suggested? If so, did that bring an improvement? As mentioned this was already a very big factor when I last tried this.

I am using this camera for ML tasks and so I am using nvidia's Tensor RT that comes with nano's jetpack in python 3.6. It is really imposible to reinstall in on a later python version, and I had to make a patch file to make Vimba SDK run on python 3.6 last time. Is there any posibility to support python 3.6 for aarch64 devices as nano and jetson in general, that comes with all the nvidia goodies ?

I understand and agree, this makes it hard to impossible to upgrade as you would use the nice NVidia support for all those other libraries. As explained in #16 there is unfortunately currently no plan to backport VimbaPython to Python3.6. I am afraid you would have to keep that patch file around. But since it is essentially only commenting out three lines, I believe this is manageable.