ESP32 Arduino 2.0.2 - multiple fb_count problem

[jameszah]

There seems to be a problem with multiple fb_count buffers in 2.0.2. It appears similar to this issue https://github.com/espressif/esp32-camera/issues/309

The problem is that when you init() the camera it fills the buffers with 4 pictures (for fb_count = 4), and then the first 4 calls to esp_camera_fb_get() will deliver those old original pictures, and it will store a new picture. And then the new picture will sit there until 4 more calls to fb_get(). So you are always 4 pictures behind. If you are waiting for 1 minute for an event, and then you call esp_camera_fb_get() for a current picture, you will get a picture of whatever was happening a minute ago.

The easy work-around is to take 4 pictures to clear out the old stuff, but you are still getting the 4th oldest picture with a call to fb_get().

Switching to fb_count = 2, improves things so you get the continuous camera function, but you still have to discard 1 picture to make sure you have something recent after the camera has not be used for a while.

If you run the code below while taking pictures of a stopwatch, you will see the effect -- the pictures are stored on the sd card with a sequence number, and the time hhmmss of the picture - you can see the burst of 10 pictures that include 4 old and 6 new marked within the same second or two.

• After the init() there is a delay before the first group of 10 pictures - 4 from the time of the init(), and 6 current.
• then 4 calls to fb_return() does nothing
• then 10 more pictures gives you the last 4 from the previous batch, and 6 new
• then the 4 calls to fb_get() and return() will clear out the old pictures, and give you 10 new (4 + 10 fb_get's )
• but the last bit of code which takes pictures at various intervals, you will first get the last 4 of that previous group of 14, and you will not see all the delays of 5 or 7 seconds

So maybe just use fb_count = 2, do 2 fb_gets if you are doing any delay between fb_get, and copy the picture out of camera storage to your own psram storage if you are saving it for later use.

This is Arduino 1.8.19 with esp32-arduino 2.0.2 for an ai-thinker esp32-cam module with an sd card and 2640 camera.

"C:\\ArduinoPortable\\arduino-1.8.19\\portable\\packages\\esp32\\hardware\\esp32\\2.0.2/tools/gen_esp32part.exe" -q "C:\\Users\\James\\AppData\\Local\\Temp\\arduino_build_63660/partitions.csv" "C:\\Users\\James\\AppData\\Local\\Temp\\arduino_build_63660/cam_test.1.ino.partitions.bin"
Using library SD_MMC at version 2.0.0 in folder: C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\hardware\esp32\2.0.2\libraries\SD_MMC 
Using library FS at version 2.0.0 in folder: C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\hardware\esp32\2.0.2\libraries\FS 
"C:\\ArduinoPortable\\arduino-1.8.19\\portable\\packages\\esp32\\tools\\xtensa-esp32-elf-gcc\\gcc8_4_0-esp-2021r2/bin/xtensa-esp32-elf-size" -A "C:\\Users\\James\\AppData\\Local\\Temp\\arduino_build_63660/cam_test.1.ino.elf"
Sketch uses 394925 bytes (12%) of program storage space. Maximum is 3145728 bytes.
Global variables use 22760 bytes (6%) of dynamic memory, leaving 304920 bytes for local variables. Maximum is 327680 bytes.

#include <stdio.h>
// Time
#include "time.h"

#include "esp_camera.h"

int start;

// MicroSD
#include "driver/sdmmc_host.h"
#include "driver/sdmmc_defs.h"
#include "sdmmc_cmd.h"
#include "esp_vfs_fat.h"
#include <SD_MMC.h>

struct tm timeinfo;
time_t now;
char strftime_buf[64];

// CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM     32
#define RESET_GPIO_NUM    -1
#define XCLK_GPIO_NUM      0
#define SIOD_GPIO_NUM     26
#define SIOC_GPIO_NUM     27
#define Y9_GPIO_NUM       35
#define Y8_GPIO_NUM       34
#define Y7_GPIO_NUM       39
#define Y6_GPIO_NUM       36
#define Y5_GPIO_NUM       21
#define Y4_GPIO_NUM       19
#define Y3_GPIO_NUM       18
#define Y2_GPIO_NUM        5
#define VSYNC_GPIO_NUM    25
#define HREF_GPIO_NUM     23
#define PCLK_GPIO_NUM     22

static esp_err_t init_sdcard()
{

  esp_err_t ret = ESP_FAIL;
  sdmmc_host_t host = SDMMC_HOST_DEFAULT();
  host.flags = SDMMC_HOST_FLAG_1BIT;                       // using 1 bit mode
  host.max_freq_khz = SDMMC_FREQ_HIGHSPEED;
  int diskspeed = host.max_freq_khz;
  sdmmc_slot_config_t slot_config = SDMMC_SLOT_CONFIG_DEFAULT();
  slot_config.width = 1;                                   // using 1 bit mode
  esp_vfs_fat_sdmmc_mount_config_t mount_config = {
    .format_if_mount_failed = false,
    .max_files = 8,
  };

  sdmmc_card_t *card;

  ret = esp_vfs_fat_sdmmc_mount("/sdcard", &host, &slot_config, &mount_config, &card);

  if (ret == ESP_OK) {
    Serial.println("SD card mount successfully!");
  }  else  {
    Serial.printf("Failed to mount SD card VFAT filesystem. Error: %s", esp_err_to_name(ret));
  }

  sdmmc_card_print_info(stdout, card);

  return ret;

}

static void save_photo_dated(int i, camera_fb_t *fb) {

  time(&now);
  localtime_r(&now, &timeinfo);
  strftime(strftime_buf, sizeof(strftime_buf), "%H.%M.%S", &timeinfo);
  char jfname[130];
  sprintf(jfname, "/sdcard/pic_%d_%s.jpg",  i, strftime_buf);

  Serial.printf("save %d @ %d, len %d, %s\n", i, millis() - start, fb->len, jfname);

  FILE *file = fopen(jfname, "w");
  if (file != NULL)  {
    size_t err = fwrite(fb->buf, 1, fb->len, file);
    Serial.printf("File saved: %s\n", jfname);
  }  else  {
    Serial.printf("Could not open file: %s\n\n ", jfname);
  }
  fclose(file);
}

static void config_camera() {

  camera_config_t config;

  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;

  config.frame_size = FRAMESIZE_HD;

  config.jpeg_quality = 6;
  config.fb_count = 4;

  // camera init
  int cam_err = esp_camera_init(&config);
  if (cam_err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", cam_err);
  }

  delay(100);
}

void setup() {

  Serial.begin(115200);
  Serial.println("\n\n---");

  config_camera();

  int card_err = init_sdcard();
  if (card_err != ESP_OK) {
    Serial.printf("SD Card init failed with error 0x%x", card_err);
  }

  delay(4000);

  camera_fb_t * fb;
  for (int j = 10; j < 20; j++) {
    fb = esp_camera_fb_get();
    save_photo_dated(j, fb);
    esp_camera_fb_return(fb);
  }

  delay (6000);

  esp_camera_fb_return(fb);
  esp_camera_fb_return(fb);
  esp_camera_fb_return(fb);
  esp_camera_fb_return(fb);

  for (int j = 20; j < 30; j++) {
    fb = esp_camera_fb_get();
    save_photo_dated(j, fb);
    esp_camera_fb_return(fb);
  }

  delay (10000);

  fb = esp_camera_fb_get(); esp_camera_fb_return(fb);
  fb = esp_camera_fb_get(); esp_camera_fb_return(fb);
  fb = esp_camera_fb_get(); esp_camera_fb_return(fb);
  fb = esp_camera_fb_get(); esp_camera_fb_return(fb);

  for (int j = 30; j < 40; j++) {
    fb = esp_camera_fb_get();
    save_photo_dated(j, fb);
    esp_camera_fb_return(fb);
  }

  delay(5000);

  Serial.printf("starting %d\n", millis());
  start = millis();

  delay(5000);

  camera_fb_t * fb1 = esp_camera_fb_get();
  Serial.printf("take 1 @ %d,  len %d\n", millis() - start, fb1->len);

  delay(5000);

  camera_fb_t * fb2 = esp_camera_fb_get();
  Serial.printf("take 2 @ %d, len %d\n", millis() - start, fb2->len);

  delay(5000);

  camera_fb_t * fb3 = esp_camera_fb_get();
  Serial.printf("take 3 @ %d, len %d\n", millis() - start, fb3->len);

  delay(2000);

  Serial.printf("save 1 @ %d, len %d\n", millis() - start, fb1->len);
  save_photo_dated(51, fb1);
  esp_camera_fb_return(fb1);

  delay(2000);

  Serial.printf("save 2 @ %d, len %d\n", millis() - start, fb2->len);
  save_photo_dated(52, fb2);
  esp_camera_fb_return(fb2);

  delay(2000);

  Serial.printf("save 3 @ %d, len %d\n", millis() - start, fb3->len);
  save_photo_dated(53, fb3);
  esp_camera_fb_return(fb3);

  delay(5000);

  camera_fb_t * fb4 = esp_camera_fb_get();
  Serial.printf("take 4 @ %d, len %d\n", millis(), fb4->len);
  Serial.printf("save 4 @ %d, len %d\n", millis() - start, fb4->len);
  save_photo_dated(54, fb4);
  esp_camera_fb_return(fb4);

  delay(5000);

  camera_fb_t * fb5 = esp_camera_fb_get();
  Serial.printf("take 5 @ %d, len %d\n", millis(), fb5->len);
  Serial.printf("save 5 @ %d, len %d\n", millis() - start, fb5->len);
  save_photo_dated(55, fb5);
  esp_camera_fb_return(fb5);

  delay(5000);

  camera_fb_t * fb6 = esp_camera_fb_get();
  Serial.printf("take 6 @ %d, len %d\n", millis(), fb6->len);
  Serial.printf("save 6 @ %d, len %d\n", millis() - start, fb6->len);
  save_photo_dated(56, fb6);
  esp_camera_fb_return(fb6);

  delay(7000);

  camera_fb_t * fb7 = esp_camera_fb_get();
  Serial.printf("take 7 @ %d, len %d\n", millis(), fb7->len);
  Serial.printf("save 7 @ %d, len %d\n", millis() - start, fb7->len);
  save_photo_dated(57, fb7);
  esp_camera_fb_return(fb7);

  Serial.println("done");
}

void loop() {

}

[jameszah]

Solution over here; https://github.com/espressif/esp32-camera/issues/363#issuecomment-1042667693

[jameszah]

Looks like it was a "feature, not a bug", introduced May 21, 2021, but didn't get to the Arduino world until esp32-arduino 2, late in 2021, and not much noticed until 2022.

https://github.com/espressif/esp32-camera/commit/8eb032a94e518257d6bdbefc6d4d77f64cbf6b77

/**
 * @brief Configuration structure for camera initialization
 */
typedef enum {
    CAMERA_GRAB_WHEN_EMPTY,         /*!< Fills buffers when they are empty. Less resources but first 'fb_count' frames might be old */
    CAMERA_GRAB_LATEST              /*!< Except when 1 frame buffer is used, queue will always contain the last 'fb_count' frames */
} camera_grab_mode_t;

https://github.com/espressif/esp32-camera/blob/86a4951f507e86034725b04f2428c98c49496dfe/driver/include/esp_camera.h#L42

It defaults to filling all buffers with pictures (fb_count), and then giving them to you when you call fb_get. If you want the picture when you call fb_get, you have to change the parameter to CAMERA_GRAB_LATEST.

[tcpipchip]

and on Arduino, no news ?

i had to make to two pictures to validade a updated image...

[tcpipchip]

config.fb_count = 1; ?

[me-no-dev]

@tcpipchip it already is in Arduino

[tcpipchip]

Yes, thanks

[tcpipchip]

its working, i can send a image to aws IoT, send to lambada, call the recokgnizer to extract text and send back to aws IoT

[jameszah]

quick update:

The CAMERA_GRAB_LATEST does not actually give you the latest image when you call fb_get. If fb_count is 4, you still have to empty the queue of 3 old images. The images are only 80 ms (720p for ov2640) or 40 ms (vga on ov2640) old, so they are quite new, but still not the latest.

The big problem is CAMERA_GRAB_WHEN_EMPTY, when they could be hours/days old if your camera is sitting there waiting for activity.

You can check the time that comes with the image if it is critical https://github.com/espressif/esp32-camera/blob/f0bb42917cddcfba2c32c2e2fb2875b4fea11b7a/driver/include/esp_camera.h#L169

if fb_count > 1, you have to get and release all old jpegs to get the current image.

If CAMERA_GRAB_WHEN_EMPTY, then you could have images that are a week old (good for battery and keep camera/esp32 cool)

If CAMERA_GRAB_LATEST, then the camera and esp32 are running full speed and your oldest image is 3 x 40 ms = 120 ms (for fb_count is 3 and camera set to vga) so not that noticeable. (good for fast efficient video - but camera and esp32 might get hot)