disable_stream() still does not work

[someilay]

System Info
Camera Model	D415
Firmware Version	5.13.0.55
Operating System & Version	Ubuntu 22.04.5 LTS
Kernel Version (Linux Only)	6.8.0-45-generic
Platform	PC
SDK Version	2.56.1
Language	C++
Segment	CV

---

Issue Description

Hello everyone. I met an problem with using librealsense. For my purposes I needed to collect some number of depth frames for post-processing and one color frame. I had four D415 cameras. I decided to take a depth frames first and then collect the color one. So, for optimization it is good to enable only depth stream first. However, I could not disable color stream with disable_stream, these frames still had been collected.

The code:

rs2::context ctx;
rs2::config cfg;
cfg.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, depthFrameRate);
cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_YUYV, colorFrameRate);

std::vector<Device> devices;
for (auto serial : serials) {
    devices.emplace_back(enable_device(serial, cfg, ctx, false));
    cout << serial << " is successfully enabled!\n";
}

...

unordered_map<string, vector<rs2::depth_frame>> depthFrames;
for (auto device : devices) {
    enableEmitter(device);
    auto enableTimeMs = (double)duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
    int framesCollected = 0;
    rs2::frameset frameset;
    rs2::depth_frame* firstDepthFrame = nullptr;
    rs2::depth_frame* prevDepthFrame = nullptr;
    depthFrames[device.serial] = vector<rs2::depth_frame>();
    cfg.disable_stream(RS2_STREAM_COLOR);

    auto sensor = device.profile.get_device().first<rs2::color_sensor>();
    for (auto stream: sensor.get_active_streams()) {
        cfg.disable_stream(RS2_STREAM_COLOR, stream.stream_index()); // Disabling color stream
    }

    while (framesCollected < numFramesForAverage) {
        try {
            frameset = device.pipeline.wait_for_frames(5000);
        }
        catch (std::runtime_error& e) {
            cout << "Frames does not received for 5000 ms for " << device.serial << "\n";
            continue;
        }

        auto depthFrame = frameset.get_depth_frame();
        auto colorFrame = frameset.get_color_frame(); // <-- still collected

        if (depthFrame.get_timestamp() < enableTimeMs + 130) {
            continue;
        }

        if (prevDepthFrame != nullptr && prevDepthFrame->get_frame_number() == depthFrame.get_frame_number()) {
            continue;
        }

        if (firstDepthFrame == nullptr) {
            firstDepthFrame = new rs2::depth_frame(depthFrame);
        }

        if (prevDepthFrame != nullptr) {
            delete prevDepthFrame;
        }
        prevDepthFrame = new rs2::depth_frame(depthFrame);
        depthFrames[device.serial].push_back(depthFrame);
        framesCollected += 1;
    }

    disableEmitter(device);
    delete firstDepthFrame;
    delete prevDepthFrame;
}

For reference Device is

typedef struct {
    rs2::pipeline pipeline;
    rs2::pipeline_profile profile;
    string serial;
} Device;

enable_device() is

Device enable_device(string& serial, rs2::config& cfg, rs2::context& ctx, bool enableEmitter) {
    rs2::pipeline pipeline(ctx);
    cfg.enable_device(serial);
    auto profile = pipeline.start(cfg);

    auto sensor = profile.get_device().first<rs2::depth_sensor>();
    if (sensor.supports(RS2_OPTION_EMITTER_ENABLED)) {
        sensor.set_option(RS2_OPTION_EMITTER_ENABLED, enableEmitter ? 1 : 0);
    }

    return { pipeline, profile, serial };
}

Commenting line with cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_YUYV, colorFrameRate); prevents collecting color frames. However, I want to it to be enabled. I tried to find the solution in the official documentation but I failed. Also I saw this issue but provided solution uses low API calls.

I will be glad to any suggestion.

[MartyG-RealSense]

Hi @someilay instead of using disable_stream(), an alternative approach that you could take is to start the pipeline with a cfg instruction for depth only. Later, stop the pipeline after the depth capture, set a new cfg configuration for the color stream only and then restart the pipeline. Something like this:

pipeline.stop();

cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_YUYV, colorFrameRate);

 auto profile = pipeline.start(cfg);

When the pipeline is restarted after the stop, the new color stream only configuration will be applied.

[someilay]

@MartyG-RealSense Thanks for your quick response. I tried your solution. It works well for disabling color stream, but I cannot enable it back. My code looks now like that

// Enable emitter

device.pipeline.stop();
device.cfg.disable_stream(RS2_STREAM_COLOR);
device.profile = device.pipeline.start(device.cfg);

// Collect depth frames

device.pipeline.stop();
device.cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_YUYV, colorFrameRate);
device.profile = device.pipeline.start(device.cfg);

try {
    frameset = device.pipeline.wait_for_frames(5000);
    colorFrames[device.serial] = new rs2::video_frame(frameset.get_color_frame());
}
catch (std::runtime_error& e) {
    cout << "Frames does not received for 5000 ms for " << device.serial << "\n"; // <-- getting this error
}

[MartyG-RealSense]

You should not need to use a disable_stream instruction. When the pipeline is stopped, all active streams are stopped automatically.

When cfg is used before a pipe start line, only the stream specified in the cfg line will be started.

[someilay]

Ok. If I understand correct, I should recreate rs2::config each time when I want to change active stream. Right?

[MartyG-RealSense]

You only need to call rs2::config once in the script. When you need to change the active stream, just change the cfg line after the pipeline stop command and then start the pipeline again to apply the change.

[someilay]

Without passing rs2:config as argument?

[MartyG-RealSense]

In a C++ script, rs2::config will typically be near the start of the script. For example:

rs2::pipeline pipe;
rs2::config cfg;

You only need to define it once at the beginning.

[someilay]

But it does not work. I tried it here. I created for each camera separate rs2::pipeline and rs2::config.

[MartyG-RealSense]

Ah yes, you are using multiple cameras. Thanks for the reminder.

https://github.com/IntelRealSense/librealsense/issues/1735#issuecomment-390840337 has a Python example of setting separate configs for each camera that you may be able to adapt for C++.

[someilay]

Thanks. However, I have already saw it, and it is not suitable for me( I want to collect multiple depth frames and one color frame. Of course, I can achieve it without disabling and enabling different streams. But my project is on the stage where any time delay or USB bandwidth over consumption are critical. So, I am forced to find some optimization tricks.

[someilay]

May be you have any suggestions?

[MartyG-RealSense]

If you want to reduce the processing demand without changing the depth resolution, you could use the Decimation post-processing filter to 'downsample' the depth image to reduce the depth scene complexity.

https://dev.intelrealsense.com/docs/post-processing-filters#decimation-filter

If you do not want to sacrifice depth image quality though then what might improve performance for you is to use the Keep() instruction to store frames in the computer's memory.

https://github.com/IntelRealSense/librealsense/issues/1942#issuecomment-400031901 has an example of a C++ script that implements both Keep() and the Decimation filter.

[someilay]

you could use the Decimation post-processing filter to 'downsample' the depth image to reduce the depth scene complexity.

Already used

what might improve performance for you is to use the Keep()

No problems with storing frames

[someilay]

Let's back to the issue topic. I still cannot achieve disabling/enabling color stream. Currently my progress is on this stage

[MartyG-RealSense]

With four simultaneously active cameras on the same computer, there will be a limit to how much you can improve performance unless you use a computer with a higher hardware specification (e.g a more powerful CPU), because each individual camera is consuming a portion of the computer's available resources.

It might be best to go back to your original plan of collecting the depth and color frames at the same time.

https://github.com/IntelRealSense/librealsense/issues/2219#issuecomment-412899468 has a C++ script for automatically retrieving an RGB color frame from all attached cameras simultaneously.

[someilay]

So, disabling unnecessary stream with lower FPS does not improve the overall performance. Did I understand you right?

[MartyG-RealSense]

There should be a minor improvement in performance if an unnecessary stream is disabled. Depth streams tend to impose a higher processing burden than RGB streams though, so you may not be gaining much performance by disabling color.

The easiest way to reduce processing burden is to use a lower stream resolution or a lower FPS speed. Either way will reduce the total USB data bandwidth being consumed by the four cameras.

Charts at the link below provide estimates of the USB bandwidth that four cameras will use at different resolutions and streaming modes.

https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#2-multi-camera-considerations

[someilay]

Thanks for nice hint. Nevertheless, can we consider my case as bug? I still want to test this hypothesis by myself

[MartyG-RealSense]

To consider the possibility of a bug, we would need a report from another user experiencing the same issue in order to confirm that it is not an issue that is only occuring on one particular computer. I would be happy to leave this case open to see if anyone else comments about a similar experience.

[someilay]

Ok, I don't mind

[MartyG-RealSense]

Okay, I have added an 'Enhancement' tag to this case as a reminder for it to be kept open.