Intel RealSense for Processing

Intel RealSense 2 support for the Processing framework.

Introduction

Intel RealSense for Procesing is a port of the Intel RealSense library for processing. With this library it is possible to use the Intel RealSense T200 / D400 / D500 camera series within processing. The idea is not to expose the full API into Processing, however a simple and convenient way to work with RealSense devices. For full API support switching over to the underlying java wrapper is recommended.

Supported Intel RealSense Version: 2.53.1

Important ⚠️

• If you were using the old API (PreReleases, dated Feb 2019) and do not want to update your sketch, download the 1.1.0 library from the releases and install it manually into your processing library folder.
• It is not recommended to kill a sketch in processing, without closing the camera.
• Currently the library is still under development.
• Linux (x86 / x64 / armhf / arm64), MacOS (x64) and Windows (x86 / x64) binaries are already bundled into the jar file.

Supported Configurations

Here are some configurations I have tested and which are working with the Intel RealSense D435. Please make sure you are using a USB 3.0 or 3.1 cable!

Installation

There are multiple ways on how to install the library for this repository into your project.

Contribution Manager

Use the contribution manager inside Processing to directly install the library into your local Processing instance.

Gradle / Maven

Include the library directly into your gradle / maven build by using jitpack.

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    implementation 'com.github.cansik:realsense-processing:2.5.0'
}

Manual

Download the latest build and extract the files into your processing library folder.

Example

Here are some examples which show how to use the library. You will find more examples here. (The examples have been tested with a RealSense D430.)

Camera

To use a RealSense camera within processing, you have to create a new instance of a RealSenseCamera. This object will give you all the possibilities of the API.

import ch.bildspur.realsense.*;

RealSenseCamera camera = new RealSenseCamera(this);

void setup() {
    // check if a camera is available
    boolean a = camera.isDeviceAvailable();

    // check how many cameras are available
    int c = camera.getDeviceCount();
}

To start a specific camera device (or multiple of them), check out the Multi Camera Color Stream example. To control an undefined amount of cameras, check out the Advanced Device Handling example.

Streams

RealSense cameras usually are equiped with multiple Sensors. Mainly video but also depth and position sensors. To use the data streams of these sensors, you have to enable them before starting the camera. It is possible to use the default values (640x480 30 FPS) or set your own settings. A complete list of valid settings can be found in the RealSense Viewer app.

After enabling the streams, you have to call the method readFrames() every time you are looking for new frames from the camera. If you do not call this method, your streams will always be black or not updated.

This example activates color and infrared streams and reads their frame data. The frames provided by videostreams are in the PImage RGB format.

void setup()
{
    size(1280, 480);

    camera.enableColorStream(640, 480, 30);
    camera.enableIRStream(640, 480, 30);
    camera.start();
}

void draw()
{
    background(0);

    // read frames
    camera.readFrames();

    // show images
    image(camera.getColorImage(), 0, 0);
    image(camera.getIRImage(), 640, 0);
}

Infrared

It is important to notice that the D415 and D430 cameras both support multiple infrared streams. To read both of them it is possible to tell the camera, which one to enable and to get.

import ch.bildspur.realsense.type.*;

void setup() {
    //...
    camera.enableIRStream(640, 480, 30, IRStream.Second);
}

void draw() {
    //...
    image(camera.getIRImage(IRStream.Second), 0, 0);
}

Depth

The depth stream of a camera is not pixel based. It usually comes as a 16bit raw byte streams which can be interpreted as depth data. To view this data it is possible to enable the Colorizer filter. This filter colorizes the depth data by using a color scheme. It is even possible to change the scheme to eight different presets.

import ch.bildspur.realsense.*;
import ch.bildspur.realsense.type.*;

RealSenseCamera camera = new RealSenseCamera(this);

void setup()
{
    size(640, 480);

    camera.enableDepthStream(640, 480);
    camera.enableColorizer(ColorScheme.Cold);

    camera.start();
}

void draw()
{
    background(0);

    camera.readFrames();
    image(camera.getDepthImage(), 0, 0);
}

Measure Distance

It is possible to measure distance on a depth frame by using getDistance(int x, int y). This will return you a float which represents the distance from the camera to the selected pixel in meters.

void draw {
    //...
    float distance = camera.getDistance(mouseX, mouseY)
}

It is important to notice that usually depth and color streams are not aligned, which makes it impossible to measure depth on a color image. For this problem you will have to align the two streams.

Projected Point

To project a point into the camera space by using the depth intrinsics, it is possible to use the method getProjectedPoint which returns a PVector containig the coordinates.

void draw {
    //...
    PVector vertex = camera.getProjectedPoint(mouseX, mouseY)
}

For more information have a look at the ProjectedPoint example.

Depth Data

To work with the raw depth data it is possible to enable the depth stream without the colorizer filter and start reading the depth data by using getDepthData(). This returns a 2-dimensional array of short with the Y / X order.

short[][] data = camera.getDepthData();

for (int y = 0; y < height; y++) {
    for (int x = 0; x < width; x++) {
        int intensity = data[y][x];
    }
}  

Alignment

To align all the incoming frames to one specific (by default depth to color frame), it is possible to enable the alignment as a preprocessor.

// enable color & depth stream
camera.enableColorStream();
camera.enableDepthStream();

// align the depth to the color stream
camera.enableAlign();

camera.start();

Filters

It is possible to use all the filters offered by the RealSense API inside processing. Just add the filter by using its add method. Some of the filters offer you to set the configuration settings while adding them. All of them support calling a default constructor (for example addThresholdFilter()) to use the default configuration.

// include the following package for the types
import ch.bildspur.realsense.type.*;

// list of all supported filters
camera.addThresholdFilter(0.0f, 1.0f);
camera.addSpatialFilter(2, 0.5f, 20, 0);
camera.addDecimationFilter(2);
camera.addDisparityTransform(true);
camera.addHoleFillingFilter(HoleFillingType.FarestFromAround);
camera.addTemporalFilter(0.4f, 20, PersistencyIndex.ValidIn2_Last4);

// The following filters have not been tested yet:
camera.addUnitsTransform();
camera.addZeroOrderInvalidationFilter();

Filter Options

To change the initial sensor options it is possible to get the filter block from the add method and use the methods provided there. Here an example on how to do use the threshold filter.

// include the processing package
import ch.bildspur.realsense.processing.*;

// in setup
RSThresholdFilter thresholdFilter = camera.addThresholdFilter();

// in draw
thresholdFilter.setMinDistance(5.0);
thresholdFilter.setMaxDistance(8.0);

Check out the ControlThresholdFilter example as well.

Sensor Options

A RealSense camera usually contains multiple sensors, each with it's unique options and settings. Currently supported are only the Depth and RGB sensor. Here is an example on how to set the Enable Auto Exposure option on the RGB sensor.

import org.intel.rs.types.Option;

camera.start();
camera.getRGBSensor().setOption(Option.EnableAutoExposure, 1.0f);

RealSense options are always of type float, ranging from min to max and do have a default value. Please be aware, that setting sensor options is only possible after the camera has been started!

Configuration

It is possible to load a predefined JSON file which contains a custom configuration. These configurations can be created in the RealSense Viewer app provided by Intel. To apply a JSON configuration, the camera has to be running already:

// load json config from file
String jsonConfig = String.join("\n", loadStrings("RawStereoConfig.json"));

// enable an example stream and start camera
camera.enableColorStream();
camera.start();

// load a json cofiguration as a string
camera.setJsonConfiguration(jsonConfig);

Advanced

For more advanced topics, the wrapper allows you to use the underlaying Java API through following getter methods.

// getters for interacting with the java API
Context context = camera.getContext();
Config config = camera.getConfig();
Pipeline pipeline = camera.getPipeline();
PipelineProfile profile = camera.getPipelineProfile();
FrameList frames = camera.getFrames();

Also check out the following example, which uses this API getters to display a pointcloud.

• Pointcloud Example (Advanced API)

FAQ

We try to gather the most frequent questions and answer them here, so we do not have to answer them in every issue.

The method start(Device) in the type RealSenseCamera is not applicable for the arguments (int, int, int, boolean, boolean)

You are still using the deprecated API. Please update your code to the 2.0 API structure or install the deprecated API as described in the section Important.

The image from the RealSense looks distorted and glitchy.

When shutting down the camera without using the stop() method, the camera can fall into a bricked state. Just plug out the camera and plug it back in to reset it.

The camera directly starts with an error that the device could not have been opened.

Either the camera is already used in another application (RealSense-Viewer?) or it is in a bricked state. Just plug out the camera and plug it back in to reset it.

Build

To build the library yourself just use the predefined gradle command. The zipped processing library will be in the release folder.

# windows
gradlew.bat releaseProcessingLib

# mac / unix
./gradlew releaseProcessingLib

About

The processing library is maintained by cansik and based on the Intel RealSense Java wrapper.