Recording 10 bit video @ high fps

[ysraja2415]

Hi @thiesmoeller

We are working with acA1300-200uc camera which can provide 10 bit information. 10 bit information @ high fps is important for our use case.

To extract 10 bit information from camera using the solution provided in the link .

But we are getting very less fps (not sure whether the method used in the code below to calculate fps is correct using time difference) compared to the resultant fps for the given camera settings.

For example for below code we are getting ResultingFrameRate = 143fps but fps calculated with time difference = 23 fps

Could you please help us in this case on below queries. code is attached at the end.

1. How to acquire 10 bit information at high fps.
2. Knowing sampling rate at which(fps) images are acquired is important in our case as we need this to plot fft to calculate a machine RPM. How to know the fps at which the images are acquired ?
3. Which GrabStrategy should be used for our case as we need continuous set of images with out loosing frames @ high fps to track changes in pixel values to plot fft?

cwd = os.getcwd()
frames_Path = os.path.join(cwd,'frames')
if os.path.isdir(frames_Path):
    shutil.rmtree(frames_Path)
!mkdir {frames_Path}

num_img_to_save = 500
img = pylon.PylonImage()
tlf = pylon.TlFactory.GetInstance()
camera = pylon.InstantCamera(tlf.CreateFirstDevice())
camera.Open()
camera.PixelFormat = "BayerBG10"
camera.Width = 1024
camera.Height = 1024
camera.ExposureTime = float(5000)
camera.StreamGrabber.MaxTransferSize = 4 * 1024 * 1024

print("Resultant fps : ",camera.ResultingFrameRate.GetValue())
#camera.AcquisitionMode.SetValue('Continuous')
#camera.AcquisitionFrameRateEnable.SetValue(True)
#camera.AcquisitionFrameRate.SetValue(143.0)
camera.StartGrabbing(pylon.GrabStrategy_LatestImageOnly) 

converter = pylon.ImageFormatConverter()
converter.OutputPixelFormat = pylon.PixelType_RGB16packed  #RGB16packed#Mono16
converter.OutputBitAlignment = pylon.OutputBitAlignment_LsbAligned
image_Array = np.empty(shape=(Height,Width,3,num_img_to_save),dtype=np.uint16) 

begin = time.time()
for i in range(num_img_to_save):
    with camera.RetrieveResult(2000) as result:
        im = converter.Convert(result)
        image = np.ndarray(buffer=im.GetBuffer(),shape= (im.GetHeight(),im.GetWidth(),3),dtype=np.uint16)
        image_Array[:,:,:,i] = image

        #To store image instead of array use below code

        #img.AttachGrabResultBuffer(result)
        #filename = frames_Path+"\%06d.tiff" % i  
        #img.Save(pylon.ImageFileFormat_Tiff, filename)
        #img.Release()

end = time.time()

camera.StopGrabbing()
camera.Close()

#uncomment this if you are saving images
#lst = os.listdir(frames_Path) 
#number_Of_Files = len(lst)

#if number_Of_Files == image_Array.shape[-1]:
#    print('Grabbing successful')
#else:
#    print('Grabbing not successful')

fps = num_img_to_save/(end-begin)
print('fps got : ',fps)

[SMA2016a]

1) if the camera parameter ResultingFrameRate reports 143fps, then the camera will work at that speed unless the grabloop is blocks by image processing. Simply measure the time taken by image processing. In case this is longer than 1/fps, you need to do the image processing outside the grab loop.

2) ResultingFrameRate

3) onebyone

[ysraja2415]

Thanks @SMA2016a for reply.

When I moved my grabresult processing code out of grab loop able to achieve the fps promised. But after grabbing I am unable to access and convert the grabresult which was stored earlier because of below error.

InvalidArgumentException: Cannot convert image. The passed source image is invalid. : InvalidArgumentException thrown (file 'imageformatconverter.cpp', line 77)

arr = []
for j in range(num_img_to_save):
    arr.append(pylon.GrabResult())
camera.StartGrabbing(pylon.GrabStrategy_OneByOne) 

begin = time.time()
for i in range(num_img_to_save):
    with camera.RetrieveResult(2000) as result:
        if result.GrabSucceeded():
            arr[i] = result
        else:
            print(result.ErrorDescription)
        result.Release()
end = time.time()

camera.StopGrabbing()
camera.Close()

fps = num_img_to_save/(end-begin)
print('fps : ',fps)

image_Array = np.empty(shape=(Height,Width,num_img_to_save),dtype=np.uint16) 
converter.OutputPixelFormat = pylon.PixelType_Mono16#RGB16packed#Mono16
converter.OutputBitAlignment = pylon.OutputBitAlignment_LsbAligned

for i in tqdm(range(num_img_to_save)):   
    im = converter.Convert(arr[i]) # The error comes for this line while converting
    image_Array[:,:,i]  = np.ndarray(buffer=im.GetBuffer(),shape= (im.GetHeight(),im.GetWidth()),dtype=np.uint16)

While converting when i try to check whether grabresult succeeded got below error

RuntimeException: No grab result data is referenced. Cannot access NULL pointer. : RuntimeException thrown (file 'grabresultptr.cpp', line 87)

image_Array = np.empty(shape=(Height,Width,num_img_to_save),dtype=np.uint16) 
converter.OutputPixelFormat = pylon.PixelType_Mono16#RGB16packed#Mono16
converter.OutputBitAlignment = pylon.OutputBitAlignment_LsbAligned

for i in tqdm(range(num_img_to_save)):

    if arr[i].GrabSucceeded():  # The error came here
        im = converter.Convert(arr[i])
        image_Array[:,:,i]  = np.ndarray(buffer=im.GetBuffer(),shape= (im.GetHeight(),im.GetWidth()),dtype=np.uint16)

[thiesmoeller]

Hi @ysraja2415,

a) the acA1300-200uc outputs Bayer RAW images in 10 bit per pixel. Which is 16bit per pixel in memory. b) What is the format, you want to put into your image_array? also Bayer 10 bit per pixel ( in uint16 ) ?

You capture completely to memory and convert later. Which is good setup, if you have enough memory available. -> you need 51210241024*2 = 1.024GByte

One thing you have to change is: the grab results are just pointers to the image memory, which are recycled during grabbing. So you have to add the real image into your array instead. And can use your final buffer directly:

for i in range(num_img_to_save):
    with camera.RetrieveResult(2000) as result:
        if result.GrabSucceeded():
            # result Array is already an uint16 array with your image Data
            image_Array[:,:,i] = result.Array
        else:
            print(result.ErrorDescription)
        result.Release()
end = time.time()

[ysraja2415]

Hi @thiesmoeller When I ran the above snippet of code observed below two things.

1. For the given camera settings my camera's resultantfps is 143 but i got around 95-100 fps with above snippet of code. For our use case we should acquire images at the maximum possible fps (here resultant fps which is based on camera settings). How to achieve this ? I am using laptop with 16gb RAM i7 11 gen with graphic card of 6Gb.
2. result.Array 2D array of dimensions (Height , width) with maximum value as 1022 implying it is a 10 bit data. Can we get 3 channel (R G B) coloured 10 bit output ? When i ran below snippet of code to get the image from result.array got checkered image as output. Is this the correct way to get image from result.Array ?

   save_path = 'image_1.png'
   image= np.array(image_Array[:,:,0]/4,dtype = np.uint8)
   cv2.imwrite(save_path,image)

   

For our usecase below is the requirement.

Acquire 10 bit information of each pixel raw information (uncompressed if image format) preferably color information with 3 channels(if not possible with our camera mono should be fine) at high fps (preferably @ camera.ResultingFrameRate.GetValue() ) display this raw image and process this raw 10 bit information in our usecase and display the processed image.

[thiesmoeller]

to better understand your use-case some more questions:

you capture at 143FPS from a camera and want to display and process in 10bit.

a) do you have a display capable of showing 143FPS?

-> if not, how do you plan to do this instead

• subsample?
• slow motion? ( display at reduced FPS from memory/disk )

b) do you have a wide dynamic range Display that can display 10bit video data?

-> if not, how do you plan to handle this instead:

• complex local tone mapping?
• gamma curve?
• bit shifting?

As currently for your camera the only way for a Basler camera to output 10bit data ( in 16bit unsigned integers ) is using BayerBG10 you have to demosaic/interpolate to get RGB data on your host.

[ysraja2415]

Hi @thiesmoeller ,

Sorry for late reply.

I want to capture at 143 fps (I mean at highest possible fps) to track the changes in pixel values to know the RPM of a machine and also process it later based on the movement of pixels between the frames using the fft .

I want to do with 10 bit because we have more (4 times) shades or colors in a 10 bit picture when compared to a 8 bit picture. So if there is any area (Region of Interest) where texture is same 8 bit data may be not be useful but 10 bit data can be useful as it can detect 4 times more colors.

a) I don't need to display @ 143 fps. I need data at high fps to plot the ffts and know the RPM of machines and then use this information to process the 10 bit information later.

Lets say if a machine's RPM is 600 => 10 Hz => Then we need to sample or acquire data at least @ 20 fps Similarly if a machine's RPM is 6000 => 100 Hz => Then we need to sample or acquire data at least @ 200 fps.

After processing as you said we will display the video in slow motion.

b) As said above 10 bit information is just for processing , once the processing done we can convert it into normal 8 bit images and display.

Currently we are capturing result.Array in the grabloop then outside grab loop using cv2.colorcvt for demoisacing and interpolation.

But even capturing result.Array consumes significant time if the ResultantFPS is 143fps for given camera setting and with only one statement image_Array[:,:,i] = result.Array in grab loop I am getting maximum 88 fps.