[HOW-TO] Fastest way to capture continuous full resolution .dng files

[akarsh-sa]

Hello,

I would like to know how to and how many raw images I can capture in .dng format (full resolution) per second.

I have modified the capture_dng.py file by removing time.sleep(2), and I have looped the line: picam2.switch_mode_and_capture_file(capture_config, "full.dng", name="raw"). By running the script, I can capture 1 image every 2 seconds on average, that is 60 images in 2 minutes of total script runtime. Is there any other way to improve the capture speed?

I wondered if I could capture raw images in .dng format every 0.25 seconds for two minutes. Please let me know if my approach is wrong in fast saving the .dng files.

My specs : Raspberry pi 4 model B (4 GB); camera module v3 NOiR

[tvoverbeek]

The major part of the 2 seconds is the conversion of the raw buffer to a dng file by the pidng module. If you want to speed it up you need to save buffers first for say 2 minutes and afterwards convert them to dng files. On my Pi3b+ with camera module v3 NoIR the conversion of a full resolution (4608x2592) takes almost 3 seconds (2975 msec).

[akarsh-sa]

Thank you for your response @tvoverbeek. I appreciate your explanation of how long it takes to convert a raw buffer to a .dng file using the pidng module.

Besides, I would like to provide some background info on the project(chemical reaction engineering related) I'm working on, and I would to ask you for your expert advice/opinion. The reason why I am trying to capture at a high speed is to capture the color changes happening on the surface of a test strip (ROI(region of interest)) for colorimetric purposes. The reaction occurring on the surface leads to the color change and the reaction occurs in the timeframe of 2 minutes. I have tried to capture a video of the test strip and extract the ROI from the video (.mp4) for further analysis, but each time some parameters would change in the ROI and I was not obtaining consistent results (Green values or Saturation values, etc), although they behave similarly, the values get shifted in terms of saturation and green values ( I'm using the same light environment to capture the changes for each experiment and the camera has been fixed in the same position). The question that I would like to ask you is :

1. I'm not sure how I can speed up the process by saving buffers for 2 minutes before converting them to .dng files. Could you possibly provide any examples or a more detailed description of how to do this?
2. Is there a method for me to rapidly capture photographs using the same camera parameters without switching between experiments in terms of color space (Green or saturation values)? To obtain the same trends when I use standard/same concentrated solution?
3. Do you think I can obtain consistent images with other formats (.png or .jpeg) from one experiment to another? For example, I know that the test strip, when dripped in the same concentration solution, develops the same color trends (plot of saturation or green color value vs. time). But I'm noticing color shifts mostly due to the changes done by camera adaptive settings based on the environment. And my goal is to obtain the same trend of color changes between the two experiments when if I use the same concentration solution.

To illustrate the color shifts I'm referring to from one trial to the next, I'll share an image below with the experimental data I found. To measure the average color development variations in ROI with the same concentration solution, I took a video of each test strip and extracted the ROI. The same lighting settings were used for all of the studies.

Thank you! and I hope I'm clear in explaining the issue.

[davidplowman]

Hi, thanks for the question. I think there could be a number of different answers depending on which route you'd like to explore.

Saving raw files quickly

This does get asked from time to time because saving DNGs is not that quick. Some users have saved the raw buffers as (for example) bare numpy arrays, and then you can read them back and convert them later. It's worth noting that the exposure time and analogue gain need to be fixed, otherwise even raw files are likely to show some variation.

Getting consistent processing

It's possible to get 100% consistent processing in a raw and even in a JPEG file.

I'm going to assume you have a fixed environment. In this case I would work out what shutter time and gain you want, and program these manually. This will disable the AGC/AEC, which may otherwise make adaptive changes as time goes on. Your raw files should now be as "consistent" as possible (though see comments below for the imx708).

For JPEG files there are a few more things to fix (these changes have no effect on the raw files). I would decide what colour gains you actually want and program these manually too. This will stop the AWB from changing the colour balance subtlely over time. Actually the imx708 (camera v3) is a bit unusual in that the AWB can affect raw files in a slightly indirect way, so I would certainly recommend fixing the colour gains even for an imx708 raw file (other cameras don't show this behaviour).

Next, in the camera tuning file you need to find the line that says "ce_enable": 1, and change it to "ce_enable": 0, and finally find the line that says "n_iter": 100, and change it to "n_iter": 0,.

Hope that helps, please get back to us if you need more information.

[tvoverbeek]

@akarsh-sa I have been experimenting with the "saving raw buffers" way @davidplowman mentioned above. If you want 2 minutes worth of 4 frames per second then you need to save 480 frames. For the imx708 (as in the camera module v3) the frame size for raw can only be the size of one of the available 3 modes: 1536x864, 2304x1296, 4600x2592. SRGGB10_CSI2P stores 4 pixels in 5 bytes. So for a single 1536x864 frame you need 1536x864x1.25 = 1,658,880 bytes. 480 frames need 796,262,400 bytes. Moreover for each frame the metadata also has to be stored, since the save dng needs both. Conclusion: only a Pi4b with 2GB might work, probably best using a 4GB or 8GB model. For 4600x2592 it is up to 14MB per frame. 480 frames is already 7GB. Therefore I have used the smallest format (1536*864). Here is the code for acquiring and storing 480 frames at 4 fps:

#!/usr/bin/python3

import pickle
import time
from picamera2 import Picamera2, Preview

picam2 = Picamera2()
picam2.start_preview(Preview.NULL)

# Specifying a small size in capture_config forces the smallest mode
capture_config = picam2.create_still_configuration(raw={"size": (640, 480)}, buffer_count=2)
picam2.configure(capture_config)
print(capture_config["raw"])

picam2.set_controls({"FrameRate": 4})

# Add other controls to force constant exposures here

time.sleep(2)

buffers = []
metadatas = []
starttime = time.monotonic()
while (time.monotonic() -  starttime) < 120.0:
    request = picam2.capture_request()
    buffers.append(request.make_buffer("raw"))
    metadatas.append(request.get_metadata())
    request.release()
endtime = time.monotonic()

print('Capture duration = ', endtime - starttime)
print(len(buffers))
print(type(buffers[0]))
print(type(buffers[0][0]))
print(len(metadatas))
print(metadatas[0])

starttime = time.monotonic()
# The dng conversion needs the used config, frames and metadatas
savedata=[capture_config["raw"], buffers, metadatas]
with open('testbuffers','wb') as f:
    pickle.dump(savedata, f)
endtime = time.monotonic()
print('pickletime = ', endtime-starttime)

My testbuffers file has size 798156718. for 481 frames. Hre is the code to produce 481 dng files:

#!/usr/bin/python3

import pickle
from picamera2 import Picamera2

with open('testbuffers','rb') as f:
    data=pickle.load(f)

print(len(data))
config = data[0]
print(config)
buffers = data[1]
print(len(buffers[0]))
metadatas = data[2]
print(metadatas[0])

picam2 = Picamera2()
for i in range(len(buffers)):
   filename = 'full{0:03d}.dng'.format(i)
   print(filename)
   picam2.helpers.save_dng(buffers[i], metadatas[i], config, filename)

Each dng file has size 1659640

I have not tried to save the buffers directly to disk instead of to memory. Might be possible with a fast SSD.

[davidplowman]

Is there anything else to investigate here? If there are no further updates I'll consider closing this issue shortly. Thanks!