Bag files and Matlab

[TDVL]

Hello. I worked yesterday with RealSense Viewer in Windows and Matlab. I have extracted RGB, Infrared and depth images from .bag file using Matlab. But how can I extract the XYZ coordinates of the scene and objects from .bag file? I can't find this message in .bag file. Where are stored these data? Also, how can I display the point cloud from the .bag file using Matlab? Thank you in advance for help.

[MartyG-RealSense]

In a discussion on the Intel Support site about getting the points from a bag file in MATLAB, Intel support agent Alexandra posted a script.

https://forums.intel.com/s/question/0D70P0000069ICXSA2

[lramati]

Also, if you didn't save a pointcloud object into the bag file while recording, you'll need to construct it from the depth (and optionally color) frames. The Matlab API is almost identical to the C++ and Python APIs, so the examples from those languages should help you understand how to do this with the pointcloud filter

[TDVL]

Marty and Lior, thank you for the qiuck response. I stored the recording in *.bag file with the Viewer. Later, I used functions: bag, select, readMessages to access RGB, infrared and depth data in Matlab. Now, I don't know yet how to access the array with XYZ data and how can I access to the point cloud (for example with pcshow in Matlab)? These are my questions. I am a new RealSense user and I am not very familiar with ROS.

[MartyG-RealSense]

I do not have a script for MATLAB point clouds based on a bag, but a community member called VHess did contribute a streaming example that may give useful pointers.

https://forums.intel.com/s/question/0D70P000006ANoxSAG

[lramati]

It sounds like you're using Matlab functions for accessing general bag files. If you open it with the SDK instead, you'll get them as a simulated camera, allowing you to use all of the SDKs functionality on the pre-recorded data.

You can quickly modify the live pointcloud example by replacing line 8 with

cfg = realsense.config()
cfg.enable_device_from_file('file.bag')
profile = pipe.start(cfg)

[TDVL]

And what about xyz array? Can I access, extract the array from Matlab command window? I need it for later image processing. I use the SDK Viewer only for recording.

[lramati]

The pointcloud example (found here) has a call to pcshow commented out on line 50. Is this what you mean? If you need help setting up the SDK's Matlab wrapper, you can check the wrapper's readme

[TDVL]

I checked the example. But, can I in *.bag file simply access the XYZ array which contains the coordinates from Matlab?

[lramati]

As far as I know, bag files recorded from the viewer do not contain XYZ data in that format

[TDVL]

OK, thank you Lior. Can I record data with SDK Viewer so that the data contain XYZ?

[lramati]

I don't know that you can save pointclouds to a bag from the Viewer, but if you record the depth stream into a bag, you'll be able to use the code in the example from before to generate the XYZs. If you also record the color stream into the same bag, you'll be able to generate UVs as well to color the pointcloud

[TDVL]

OK. What about 'ply' format? I think this format should contain the XYZ.

[lramati]

It will, but last I checked the viewer didn't have a way to save PLYs for every frame, and could only be triggered manually on individual frames

[lramati]

You might be interested in the rs-convert command line tool, which can take a bag with depth frames as input and output a PLY for each one

[TDVL]

And how can I record than to have all the information (XYZ, RGB)?

[TDVL]

I need an 640x480x3 image and it's xyz coordinates

[lramati]

I would record a standard depth+color bag from the viewer, and then use the SDK's pointcloud class to obtain XYZ+UV (color of point at x,y,z = points(i) with texture coordinates u,v = tex_coords(i) is color(u,v))

[TDVL]

Is there a Matlab version of this class?

[lramati]

There is, in the Matlab wrapper I linked above. As the readme states, you can install it either via the SDK's installer for Windows, or it can be compiled from source for any platform

[TDVL]

OK, so I need to record a bag file and then use one of the function from Matlab folder after SDK installation? Are there any specific settings? Thank you for your guiding.

[lramati]

Just make sure you're saving the depth stream in the bag. The pointcloud_example I linked above should work as-is, but uses a live stream from a connected camera. The 3 line modification I shared earlier will update it to read from the specified bag file instead

[TDVL]

OK, then I will save the bag file. After that, with your modifications in pointcloud_example function I can extract the RGB and XYZ arrays and I can also check the point cloud with pcshow. Am I correct? I hope yes.

[lramati]

Yes. You'll need to uncomment line #21 to get the color frame, and uncomment line #26 to generate texture coordinates though. You should be able to understand how to access the texture coordinates just from the example and pointcloud.m, but I don't know enough about Matlab to tell how how to display a pointcloud with colored points

[TDVL]

Hello Lior, I tried with your modification, but it didn't work. I think there is a problem with lines #3 and #5, with the initial setup.

[lramati]

What error are you getting? Did you follow the instructions in the readme? There's a manual step you need to do after installing the wrapper from the installer so that Matlab can find the library

[TDVL]

Hello Lior, I used the windows installer and than I copied the Matlab folder to another folder where the bag file was. The error is: Undefined variable "realsense" or class "realsense.pipeline". I think there should be some other settings too.

[lramati]

You need to copy them still inside a folder called +realsense. This is due to how Matlab defines packages.

[TDVL]

I have copied everything from +realsense folder to another folder with bag file. Is this not correct?

[lramati]

No, you need to copy the entire +realsense folder. The folder's name is important to how Matlab works

[TDVL]

I have copied the entire folder, but the new folder's name was not +realsense

[TDVL]

I thing, the bag file should be inside the +realsense folder?

[lramati]

So long as you pass the right argument to config.enable_device_from_file(), the bag's location shouldn't matter

[TDVL]

Now everything is in +realsense folder, the bag file and the Matlab files. I changed the line with you 3 lines. The error is: realsense.pointcloud_example Error using librealsense_mex Failed to resolve request. Request to enable_device_from_file("20190212_082009.bag") was invalid, Reason: Failed to create ros reader: Error opening file: 20190212_082009.bag

Error in realsense.pipeline/start (line 30) out = realsense.librealsense_mex('rs2::pipeline', 'start', this.objectHandle, config.objectHandle);

Error in realsense.pointcloud_example (line 11) profile = pipe.start(cfg);

[TDVL]

This your code with my bag file: cfg = realsense.config(); cfg.enable_device_from_file('20190212_082009.bag'); profile = pipe.start(cfg);

[lramati]

It looks like it can't find the bag file. Try passing the full path (C:...\0000000000.bag)

[TDVL]

Something is wrong, again the error:

realsense.pointcloud_example Error using librealsense_mex Failed to resolve request. Request to enable_device_from_file("D:\Work\Measurements+relasense\20190212_082009.bag") was invalid, Reason: Failed to create ros reader: Error opening file: D:\Work\Measurements+relasense\20190212_082009.bag

Error in realsense.pipeline/start (line 30) out = realsense.librealsense_mex('rs2::pipeline', 'start', this.objectHandle, config.objectHandle);

Error in realsense.pointcloud_example (line 11) profile = pipe.start(cfg);

[TDVL]

This is the code part now:

%profile = pipe.start(); cfg = realsense.config(); cfg.enable_device_from_file('D:\Work\Measurements+relasense\20190212_082009.bag'); profile = pipe.start(cfg);

[lramati]

Looks like you've got a typo in your path there. Measurements+realsense should probably be Measurements+realsense ?

[TDVL]

Now it is working (still), I got two empty figures and this is again the error message (but it is still busy):

realsense.pointcloud_example realsense.pointcloud_example Error using librealsense_mex Failed to resolve request. Request to enable_device_from_file("D:\Work\Measurements+relasense\20190212_082009.bag") was invalid, Reason: Failed to create ros reader: Error opening file: D:\Work\Measurements+relasense\20190212_082009.bag

Error in realsense.pipeline/start (line 30) out = realsense.librealsense_mex('rs2::pipeline', 'start', this.objectHandle, config.objectHandle);

Error in realsense.pointcloud_example (line 11) profile = pipe.start(cfg);

[TDVL]

I am sure that the Matlab script is stucked, because it is still working.

[TDVL]

It stopped now, the error message is: Undefined function or variable 'Error'.

[TDVL]

Hello Lior, I changed the function a little, it is working now. Now, I will check that the function works well or not? Is there a Matlab script that extract the RGB, INFRA, DEPTH IMAGE and the XYZ at once?

[lramati]

You would need to record infrared from the viewer in order for it to be readable from the bag, but assuming you did that, pipeline.wait_for_frames() returns a realsense.frameset object containing all your frames. It's API should be rather self explanatory, so you shouldn't have a problem extracting each individual frame and putting them in a list if that's what you mean. Just remember that because of how the backend works, if you want to operate on a significant number of framesets at once, you'll need to call the keep() function on the frame objects so the backend can allocate more memory for additional frames

[TDVL]

Hi Lior, Thank you for the suggestions. I've recorded the RGB, infra and depth data in the bag file. I have many frames and I can extract the RGB, infra data for each frame separatelly, but the pointcloud_example calculates differently the XYZ data, I can't extract the coordinates for each frame separatelly, or I can (then I don't know how). Can I extract XYZ for each frame? The pipeline.wait_for_frames() is in pipeline.m file, am I correct, or it is somewhere else? I would like to extract the RGB, XYZ, INFRA and Depth with one function or script. Can I do this with the Matlab wrapper?

[TDVL]

Hello Lior, I tested the results for the obtained vertices, but they are not correct. Something is wrong. I have few frames, but I have only one array (307200x3) with xyz values (not for all frames separatelly).

[TDVL]

Hello Lior, I tested the pointcloud_example function and I got a set of parameters and variables as result. One of them is pointcloud 1x1, but I can not open that variable to see the content. I assume that this variable should contain the RGB and XYZ arrays, like when I read in a PLY file. What can be the problem? Thank you in advance.

[lramati]

The pointcloud object has functions on it you can use to access the actual xyzuv values

[TDVL]

Dear Lior, I know that, but I can not access the pointcloud object and I don't know why. I made a screenshot of the Matlab window and I saved the help message. I attached them, you can see them. After the execution of the code (the way you guide me to change the pointcloud_example function) I got the vertices and a pointcloud named object. I think this object should store the RGB and XYZ data of the point cloud, but I can't open it. Can you help me with this? Thank you. im2.pdf

[RealSenseCustomerSupport]

---

Hi TDVL,

Wonder if you have any update on this one? Do you still need help? If so, would you please share out your code and clarify issues again?

Thanks!

[TDVL]

Hello, I have extracted the point cloud using the pointcloud_example.m function, but I can not extract the point clouds for each frame (like my friend in ROS). Is this possible with Matlab? Thank you for the response. Best regards,