[Nicla Vision] Camera example not working properly

[fabik111]

Hi, I'm testing the Nicla Vision board with the example CameraCaptureRawBytes and with its companion Processing sketch CameraRawBytesVisualizer (both coming from the Camera lib of the ArduinoCore-mbed).

The image drawn on my PC by CameraRawBytesVisualizer sketch is not properly rendered (see the image below).

I didn't find the root cause. If there is a problem with the Arduino board or with the Processing sketch or with the COM port...

Here the code of the Processing sketch from the example CameraRawBytesVisualizer:

/*
  This sketch reads a raw Stream of RGB565 pixels
  from the Serial port and displays the frame on
  the window.
  Use with the Examples -> CameraCaptureRawBytes Arduino sketch.
  This example code is in the public domain.
*/

import processing.serial.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

Serial myPort;

// must match resolution used in the Arduino sketch
final int cameraWidth = 320;
final int cameraHeight = 240;

// Must match the image mode in the Arduino sketch
final boolean useGrayScale = false;

// Must match the baud rate in the Arduino sketch
final int baudRate = 921600;

final int cameraBytesPerPixel = useGrayScale ? 1 : 2;
final int cameraPixelCount = cameraWidth * cameraHeight;
final int bytesPerFrame = cameraPixelCount * cameraBytesPerPixel;
final int timeout =  int((bytesPerFrame / float(baudRate / 10)) * 1000 * 2); // Twice the transfer rate

PImage myImage;
byte[] frameBuffer = new byte[bytesPerFrame];
int lastUpdate = 0;
boolean shouldRedraw = false;

void setup() {
  size(640, 480);  

  // If you have only ONE serial port active you may use this:
  //myPort = new Serial(this, Serial.list()[0], baudRate);          // if you have only ONE serial port active

  // If you know the serial port name
  myPort = new Serial(this, "COM7", baudRate);                    // Windows
  //myPort = new Serial(this, "/dev/ttyACM0", baudRate);            // Linux
  //myPort = new Serial(this, "/dev/cu.usbmodem14301", baudRate);     // Mac

  // wait for a full frame of bytes
  myPort.buffer(bytesPerFrame);  

  myImage = createImage(cameraWidth, cameraHeight, ALPHA);

  // Let the Arduino sketch know we're ready to receive data
  myPort.write(1);
}

void draw() {
  // Time out after a few seconds and ask for new data
  if(millis() - lastUpdate > timeout) {
    println("Connection timed out.");    
    myPort.clear();
    myPort.write(1);
  }

  if(shouldRedraw){    
    PImage img = myImage.copy();
    img.resize(640, 480);
    image(img, 0, 0);
    shouldRedraw = false;
  }
}

int[] convertRGB565ToRGB888(short pixelValue){  
  //RGB565
  int r = (pixelValue >> (6+5)) & 0x01F;
  int g = (pixelValue >> 5) & 0x03F;
  int b = (pixelValue) & 0x01F;
  //RGB888 - amplify
  r <<= 3;
  g <<= 2;
  b <<= 3; 
  return new int[]{r,g,b};
}

void serialEvent(Serial myPort) {  
  lastUpdate = millis();

  // read the received bytes
  myPort.readBytes(frameBuffer);

  // Access raw bytes via byte buffer  
  ByteBuffer bb = ByteBuffer.wrap(frameBuffer);

  // Ensure proper endianness of the data for > 8 bit values.
  // The 1 byte bb.get() function will always return the bytes in the correct order.
  bb.order(ByteOrder.BIG_ENDIAN);

  int i = 0;

  while (bb.hasRemaining()) {
    if(useGrayScale){
      // read 8-bit pixel data
      byte pixelValue = bb.get();

      // set pixel color
      myImage.pixels[i++] = color(Byte.toUnsignedInt(pixelValue));
    } else {
      // read 16-bit pixel data
      int[] rgbValues = convertRGB565ToRGB888(bb.getShort());

      // set pixel RGB color
      myImage.pixels[i++] = color(rgbValues[0], rgbValues[1], rgbValues[2]);
    }       
  }

  myImage.updatePixels();

  // Ensures that the new image data is drawn in the next draw loop
  shouldRedraw = true;

  // Let the Arduino sketch know we received all pixels
  // and are ready for the next frame
  myPort.write(1);
}

The Arduino sketch code from the example CameraCaptureRawBytes:

#include "camera.h"

#ifdef ARDUINO_NICLA_VISION
  #include "gc2145.h"
  GC2145 galaxyCore;
  Camera cam(galaxyCore);
  #define IMAGE_MODE CAMERA_RGB565
#else
  #include "himax.h"
  HM01B0 himax;
  Camera cam(himax);
  #define IMAGE_MODE CAMERA_GRAYSCALE
#endif

/*
Other buffer instantiation options:
  FrameBuffer fb(0x30000000);
  FrameBuffer fb(320,240,2);
*/
FrameBuffer fb;

unsigned long lastUpdate = 0;

void blinkLED(uint32_t count = 0xFFFFFFFF)
{
  pinMode(LED_BUILTIN, OUTPUT);
  while (count--) {
    digitalWrite(LED_BUILTIN, LOW);  // turn the LED on (HIGH is the voltage level)
    delay(50);                       // wait for a second
    digitalWrite(LED_BUILTIN, HIGH); // turn the LED off by making the voltage LOW
    delay(50);                       // wait for a second
  }
}

void setup() {
  // Init the cam QVGA, 30FPS
  if (!cam.begin(CAMERA_R320x240, IMAGE_MODE, 30)) {
    blinkLED();
  }

  blinkLED(5);
}

void loop() {
  if(!Serial) {    
    Serial.begin(921600);
    while(!Serial);
  }

  // Time out after 2 seconds and send new data
  bool timeoutDetected = millis() - lastUpdate > 2000;

  // Wait for sync byte.
  if(!timeoutDetected && Serial.read() != 1) return;  

  lastUpdate = millis();

  // Grab frame and write to serial
  if (cam.grabFrame(fb, 3000) == 0) {
    Serial.write(fb.getBuffer(), cam.frameSize());
  } else {
    blinkLED(20);
  }
}

My env:

• PC: Win 11
• Arduino IDE: 1.8.19
• Processing IDE: 4.0.1
• Arduino Core: Arduino MbedOS Nicla Boards v 3.1.1

[facchinm]

Hi @fabik111 , the problem is thye same we already encountered with Vision shield on Windows https://github.com/arduino/ArduinoCore-mbed/issues/112#issuecomment-751781224 . https://github.com/arduino/ArduinoCore-mbed/pull/143 improved the situation but probably didn't fix it altogether ( your Processing sketch looks patched). @sebromero may you take a look?

[fabik111]

Thank you @facchinm, after your tips I runned the Processing sketch on an Ubuntu PC and works well, the image is drawn properly. The problem could be only on Windows PC!