kinect data logging aka recording and playback?

[ahundt]

Is it possible to record kinect data to a file and play it back?

I'm calling this data logging but don't want to confuse it with the status/error logging components of the library I found in my searching for support.

If it doesn't exist, consider this a request for it! :+1:

[xlz]

ROS can do this. I can't think of a use case of playback where there is no ROS.

Record and export to PCL would be a useful addition to examples.

[svddries]

I also think this would be very valuable, even when not using ROS or pcl (I'm currently working on a kinect project in which neither of them are used).

I guess the easiest way would be to store the raw rgb data packages obtained from the device (which are nicely JPG compressed) and either store the depth values for the depth stream directly, or compress them using PNG (which seems to give good compression results, in my experience).

[davudadiguezel]

It is possibe. You can use NiViewer that comes with openni2 to record. You'll get a .oni file that you can playback. But if you want to do skeleton detection with Nite2, you'll have to overcome some bugs in Nite2 and I don't know, if somebody has ever did this on Linux. If there is somebody please let me know. But recording and playback is with openni2 no problem.

[ahundt]

I don't need skeleton detection, just have to log the data stream because not everyone in my group has usb3 support, plus it would be able to test new code away from the physical rig just using a log of the data. I'm not using ROS and hope to remain cross platform and minimize overhead so I don't think I should necessarily start using it either, unfortunately.

[xlz]

Raw JPEG images are 20MB/s. "Raw" 32-bit float depth images are 25MB/s. Do you think you need compression?

[ahundt]

Hmm, at that data rate it could be tough to log to a non-ssd without compression so it would probably be a good option.

[ttsesm]

@ahundt did you manage to store raw data?

[ahundt]

@TheodoreT Right now I'm just saving single frame point clouds out with this: https://github.com/ahundt/libfreenect2pclgrabber

A real implementation would likely still prove useful.

[ttsesm]

@ahundt thanks for the feedback, I see. Actually, at the moment I am saving individual frames as well. I am using opencv to convert libfreenect2::Frames into opencv cv::Mat

 listener.waitForNewFrame(frames);
libfreenect2::Frame *rgb = frames[libfreenect2::Frame::Color];
 libfreenect2::Frame *ir = frames[libfreenect2::Frame::Ir];
libfreenect2::Frame *depth = frames[libfreenect2::Frame::Depth];

cv::Mat(rgb->height, rgb->width, CV_8UC4, rgb->data).copyTo(rgbMat);
cv::Mat(ir->height, ir->width, CV_32FC1, ir->data).copyTo(irMat);
cv::Mat(depth->height, depth->width, CV_32FC1, depth->data).copyTo(depthMat);

//! [registration]
registration->apply(rgb, depth, &undistorted, &registered, true, &registeredInv);
//! [registration]

cv::Mat(undistorted.height, undistorted.width, CV_32FC1, undistorted.data).copyTo(undistortedMat);
cv::Mat(registered.height, registered.width, CV_8UC4, registered.data).copyTo(registeredMat);
cv::Mat(registeredInv.height, registeredInv.width, CV_32FC1, registeredInv.data).copyTo(registeredInvMat);

and then either to save as CV_16UC1 .png(for the depth images, CV_8UC4 for the rgb and registered images) files or as CV_32FC1 .exr files.

Mat registeredInv;
registeredInvMat.convertTo(registeredInv, CV_16UC1);
imwrite(filepath + "registeredInv/" + filename +".png", registeredInv);

or

imwrite(filepath + "registeredInv/" + filename +".exr", registeredInvMat);

However, though I prefer the 16bit .png files due to the file size I am facing the problem I am describing at the #509 with the bigdepth image and the white background if I try to restore it.

---

Also I noticed in your source code from the link above in the depth grabber function:

cv::Mat getDepth(){
        listener_.waitForNewFrame(frames_);
        libfreenect2::Frame * depth = frames_[libfreenect2::Frame::Depth];
        cv::Mat tmp(depth->height, depth->width, CV_8UC4, depth->data);
        cv::Mat r;
        if (mirror_ == true) {cv::flip(tmp,r,1);}
        else {r = tmp.clone();}

        listener_.release(frames_);
        return std::move(r);
    }

you are transforming the libfreenect2::Frame to a CV_8UC4 cv::Mat, shouldn't that be a CV_32FC1 cv::Mat?

[ahundt]

you are transforming the libfreenect2::Frame to a CV_8UC4 cv::Mat, shouldn't that be a CV_32FC1 cv::Mat?

You may be right, I guess the depth is 32 bit float values? Since I'm just flipping it and nothing else 4 bytes and a 32bit float are the same size so I guess that would explain why it works even if I'm using the wrong type... I should fix that, thanks :-)

[xlz]

Tentative specification for such a recording tool.

• JPEG re-compression. Raw JPEG images are 20MB/s. Maybe use TurboJPEG to save scale it down to half and then save it with lower quality.
• Depth compression. "Raw" 32-bit float depth images are 25MB/s. Maybe use a 16-bit fixed point number to represent that (with 1mm precision, the range of 65m is more than enough).
• Use format compatible with PCL. The most efficient one is PCLZF, which can be directly converted to pointclouds with no further registration. Storing full point clouds is very inefficient.

May implement it like this:

kinect2_record [-raw | -pclzf] [cl | gl] [serial]

-raw: unsynchronized, raw jpeg, 16-bit depth, 16-bit ir, 40MB/s
-pclzf: PCLZF format with registered color (512x424) and undistorted depth
[cl | gl] [serial]: the same arguments of Protonect.cpp

The -raw format saves frames in a directory:

kinect2_saves_${serial}_${starting_timestamp}/
camera.txt
frame_$epochsecond.microsecond_color_1920x1080.jpg
frame_$epochsecond.microsecond_depth_512x424.u16
frame_$epochsecond.microsecond_ir_512x424.u16

-pclzf:

kinect2_saves_${serial}_${starting_timestamp}/
frame_$epochsecond.microsecond.xml
frame_$epochsecond.microsecond_depth.pclzf
frame_$epochsecond.microsecond_rgb.pclzf

[ahundt]

Will the overhead of opening and closing many files a second be a problem? What about an optional dependency on a library designed for this sort of thing like flatbuffers or hdf5?

[xlz]

I don't think opening and closing files is the bottleneck.

[floe]

BTW, are the JPEG images encoded in progressive mode? Then you could probably just discard some percentage of data (perhaps ~ 50%?) at the tail end of each image and still get decent quality.

16-bit integer for depth data seems entirely reasonable. Maybe consider an additional 8-bit PNG or JPEG for IR data, though?

[xlz]

The JPEG images are baseline mode.

I don't want libpng dependency, so I'll save IR data as raw 16-bit.

[floe]

I'm not a JPEG expert, but it is apparently possible to convert baseline to progressive without re-encoding (at least jpegtran can do that). Then you can discard an adjustable percentage of coefficients while avoiding extra CPU load.

[ahundt]

@xlz beyond overhead also consider the issue of forwards/backwards compatibility and versioning. That will either have to be manually maintained by the project or offloaded to a tool that has already dealt with the issue. For example, versioning and serialization is well supported by flatbuffers. At a minimum if you go with a directory of files I suggest writing out a "header" or "index" file that includes a version number and a way to quickly organize and look up data.

[xlz]

@ahundt camera.txt and frame_$epochsecond.microsecond.xml should suffice for a metadata file.

[ahundt]

Great, camera.txt and a frame XML would suffice particularly if there is a version number at the beginning.

What about a "raw" format that just writes the raw incoming/outgoing kinect usb byte stream to a file? wireshark's usb pcap might be a way to achieve this! Then the data dump to a folder you describe would be perfect for analysis, and pcap files could be used for raw recording if they could be utilized by libfreenect2 when wireshark plays them back.

In addition to kinect2_record, were you imagining an API to interact programmatically with recording and playback functionality?

[xlz]

libfreenect2 is not low level enough to capture USB byte stream.

• API for recording: already exists
• API for playback: PCL API if pclzf format; The following if raw format:

FILE *f=fopen("frame_$epochsecond.microsecond_depth_512x424.u16", "r");
unsigned short *depth = malloc(512*424*sizeof(unsigned short));
fread(depth, sizeof(*depth), 512*424, f);
fclose(f);

[andreydung]

@xlz is this resolved yet? Could you please tell me how to use this?

I went though the source code and could not find linect2_record