CBOR compression summary

mvollrath commented 5 years ago

PRs

Background

Previous to these changes, there are two options for encoding data between rosbridge and clients: JSON and JSON inside a base64-encoded PNG image.

Both of these options are text-based. Human-readable messages are great, but when the message data is thousands of integer values, the value of human readability diminishes and decoding speed becomes a bigger concern. When visualizing messages containing large blobs of binary data such as PointCloud2 or OccupancyGrid, these existing methods don't scale.

With the goal of increasing both the capacity and speed of rosbridge/roslibjs encoding and decoding, I started looking for a binary encoding format that could a) comply to the rosbridge protocol and b) quickly decode numeric arrays as views of a JavaScript ArrayBuffer.

CBOR is a perfect fit. It implements a superset of JSON, adds byte arrays, and is extensible with tags. With the addition of the draft typed array tags, it meets all of the requirements. CBOR libraries are available for most languages and I found fast implementations for Python and JavaScript.

Benchmarks

Before implementing CBOR compression I wanted to make sure it was actually faster than JSON or PNG compression. I set up a Node.js application to simulate encoding and decoding some ubiquitous ROS message types containing binary data: sensor_msgs/Image and sensor_msgs/PointCloud2.

640x480 RBG Image:

CBOR encode: 1.523ms
JSON encode: 269.129ms
CBOR size: 921711 bytes
JSON size: 11474451 bytes
CBOR decode: 0.672ms
JSON decode: 95.300ms

PointCloud2 with 1928 points @ 32 point_step:

CBOR encode: 0.201ms
JSON encode: 12.775ms
CBOR size: 62001 bytes
JSON size: 703239 bytes
CBOR decode: 0.174ms
JSON decode: 6.152ms

For sake of completeness, I also tested geometry_msgs/PoseStamped where JSON is faster, as expected:

CBOR encode: 0.172ms
JSON encode: 0.017ms
CBOR size: 179 bytes
JSON size: 232 bytes
CBOR decode: 0.125ms
JSON decode: 0.019ms

After my proof-of-concept integration with rosbridge and roslibjs, I ran some more tests on a real application to verify that it would increase frame rate in a "real world" setting where message decoding competes with rendering for JavaScript time. I compared the average decoding time for the first 100 messages of a 5,000 point PointCloud2:

JSON: 0.83ms
CBOR: 0.1ms
PNG: 45.37ms

Note that PNG didn't actually spend 45ms on the main thread because it decodes asynchronously, this was the "wall time" from message receipt to the handler being run.

A 80,000 point PointCloud2 demonstrates how much better binary encoding can scale:

JSON: 10.99ms
CBOR: 0.11ms
PNG: 264.48ms

Finally, I ran a "stress test" to see how a ROS3DJS PointCloud2 compares to rviz. I made a small patch to ROS3DJS to enable CBOR encoding with fast buffer-flipping. I was able to view a 250,000 point cloud with no frames dropped. The same application with JSON or PNG decoding dropped at least one frame per message at this scale. When I increased the cloud density to 500,000 points there was a visible frame drop with CBOR. I noticed a similar frame drop in rviz on the same cloud.

To summarize, when CBOR compression is used, ROS3DJS has the same capacity for rendering point clouds as rviz.

Web Socket Compression

To make CBOR message wire size as small as possible, I made a PR to enable per-message deflate algorithm on the web socket server. This should result in smaller messages than PNG compression where wire size is a concern (not connecting to localhost).

This compression technique is better than PNG compression because it is handled by the browser before the message makes it to the JavaScript main thread.

Implementation

CBOR works best when transmitted as a binary web socket message, so I took some liberty with the protocol. Instead of sending a JSON message with a base64 string of CBOR data, the server sends an unwrapped binary protocol message.

On the client side, this message is received as an ArrayBuffer and the CBOR decoder is run on it directly to produce an object, just like the JSON decoder. From there, everything works the same.

I produced some code to support typed array tags:

On the server side, cbor_conversion.py extracts CBOR-friendly values from a ROS message including tagged typed arrays.
On the client side, cborTypedArrayTags.js defines a "tagger" for the decoder to efficiently unpack tagged arrays.

Backwards Compatibility

Since CBOR is completely opt-in (the server will never send a CBOR binary message unless the client requests it), clients without CBOR decoding support are safe from newer server versions.

If the client requests CBOR but it is not available on the server side, the server will send JSON messages instead.

When to use CBOR

CBOR is not ideal for all messages and transports, but it's safe to say that it's the best choice for any message containing a large amount of binary data when using roslibjs in a browser.

JSON is still faster and better for small messages such as transforms, or messages containing mostly strings.

PNG is best in some specific edge cases. It is still potentially the smallest wire size where transport compression is not available, depending on the data. When the data is mostly redundant, e.g. an array full of zeros, PNG messages can be smaller and decode faster than CBOR, because the deflate algorithm is efficient in this case, but when the data is more random, PNG will often have a larger wire size and longer decoding time than regular JSON compression. This makes it a poor choice for PointCloud2 and situationally useful for Image or OccupancyGrid.

mvollrath commented 5 years ago

All merged :+1:

Weimmer commented 4 years ago

Hi @mvollrath ,

this is the exactly problem what i met, i use rosbridge send the point cloud to Unity and it is JSON, which is too slow and lagging, i am new to ROS , as i understood, if i want to use CBOR, i have to write a subscriber in ROS side to recive the point cloud data as JSON and a publisher to republish the data with CBOR, and wirte another subscriber in Unity side to recive the CBOR data. is it right to use the CBOR like this?

thanks in advance

Wei

mvollrath commented 4 years ago

Hi Wei,

To get PointCloud2 data from ROS to Unity over rosbridge using CBOR you shouldn't need to write any ROS subscribers, rosbridge will do the CBOR encoding for you.

From Unity you will connect to the rosbridge socket and send a "subscribe" message with the "compression" field set to "cbor".

Whenever a point cloud message comes into Unity over the socket, it will be a CBOR-encoded binary "publish" message.

Weimmer commented 4 years ago

Hi @mvollrath thanks for the answer. i use the ROSbridgeLib in Unity to connect with ROS via websocket, but i don't know how to send a "subscribe" message with the "compression" field set to "cbor". what now i understand is, i add the additional "compression: "cbor"" in my subscriber in Unity, and the publicher in ROS will send the message in binary, namely i will recive the point cloud data in binary in Unity.

mvollrath commented 4 years ago

ROSBridgeLib does not support CBOR decoding. You have a few options:

Get ROSBridgeLib to support CBOR decoding, either by creating an issue on their tracker and getting help or adding it yourself.
You might also try ros-sharp, which also doesn't yet support CBOR decoding but has an issue tracker.
Instead of using ROSBridgeLib, use a websocket library and send/receive the protocol messages yourself, decoding either JSON or CBOR as it comes in.

Sorry about this, CBOR encoding on the server side is a new feature and the client libraries haven't caught up yet.

Weimmer commented 4 years ago

@mvollrath Thanks for the detailed answer. i am new to programming and computer science, but i will try it:)

Weimmer commented 4 years ago

Hi @mvollrath , i am now getting start to deal with the CBOR, and trying to add a compression in ROSBridgeLib. Could you tell me if i should write some code in ROS side in oder to compress the message? i have tried to send message {"op": "subscribe", "topic": "/zed/zed_node/point_cloud/cloud_registered", "type": "sensor_msgs/PointCloud2","compression": "cbor"} but it seems nothing changed, maybe i did something wrong. Thanks in advance

mvollrath commented 4 years ago

@eisenWu that looks correct, make sure you're running at least rosbridge_suite version 0.11.3. You should get binary messages. Compare to "compression": "json" which should yield text messages.

Weimmer commented 4 years ago

Hi @mvollrath , today i did the test, and i found that when i want to get the byte[] Message from the function OnMessage in websocket-sharp, it contains a RawData. when i debug.log the RawData it is a byte array. Now i am confused, even when i deleted the "compression": "cbor" it was aways byte array Message, and i have no idea if the CBOR works or not, how can i detect is the Message compressed?

mvollrath commented 4 years ago

That RawData is always going to be a byte string, when you expect JSON try decoding it with UTF-8 into a string.

e.g. string text = Encoding.UTF8.GetString(msg.RawData)

Weimmer commented 4 years ago

Hi @mvollrath , thanks for your answer :) I think i have made a mistake, now once when i send {"op": "subscribe", "topic": "/zed/zed_node/point_cloud/cloud_registered", "type": "sensor_msgs/PointCloud2","compression": "cbor"} to ROS, the window shows the infomation: Client disconnected. i use the ROS version: melodic, but have no idea how to see which version of rosbridge_suite i am using. I do not have the decode of CBOR in my code yet, would that be the reson why the Client just disconnected?

mvollrath commented 4 years ago

That is possible, maybe something to do with how C# Encoding handles illegal UTF-8 codes which may be present in a large blob of point cloud data. Hopefully you can get some helpful log output from the client. Unity prints traces in the player log if not in the editor log.

If your version of rosbridge_suite didn't support CBOR it would just send JSON.

zdzhaoyong commented 3 years ago

ROSBridgeLib does not support CBOR decoding. You have a few options:

Get ROSBridgeLib to support CBOR decoding, either by creating an issue on their tracker and getting help or adding it yourself.

You might also try ros-sharp, which also doesn't yet support CBOR decoding but has an issue tracker.

Instead of using ROSBridgeLib, use a websocket library and send/receive the protocol messages yourself, decoding either JSON or CBOR as it comes in.

Sorry about this, CBOR encoding on the server side is a new feature and the client libraries haven't caught up yet.

I was confusing why the rosbridge server only send CBOR but not receive it. The CBOR decoding should not a difficult job.

zdzhaoyong commented 3 years ago

I just wrote a C++11 based rosbridge client where CBOR (de)serializer supported. See. https://github.com/zdzhaoyong/svar_rosbridge By using Svar, it can be used by both C++ and Python. Hope the rosbridge_server also supports CBOR decoding soon!

wyq730 commented 2 years ago

@mvollrath

Thanks for this doc. That's a great introduction!

I'm wondering, however, how is CBOR compared with some common bytes-compatible serialization schemes like bson or msgpack? I did a simple benchmark and it seems that CBOR didn't do better than them, so I would like to know if there is additional rationale behind which I didn't take into consideration. Thanks!

mvollrath commented 2 years ago

The thing I was looking for was performance at homogenous typed arrays, e.g. large arrays of int16 being transformed into JavaScript typed arrays. There's a big difference between a CBOR array and a tagged typed array; instead of declaring the type of each array element you just put a tag in front and pack the bytes as densely as possible. Decoding the bytes is very efficient in most languages incl. JavaScript (typed arrays from byte strings is superfast) and Python (import struct and unpack is superfast). These kinds of homogenous arrays are used in ROS messages like point clouds and occupancy grids and such, and these are the kinds of messages CBOR encoding was intended for. JSON is still faster in typical browsers for small messages or messages with many keys because it has native implementation.

Otherwise:

BSON was never designed to be a wire protocol and all the libs I tried were some combination of slow, inefficient, or cumbersome.
Msgpack is what CBOR is based on, but didn't seem to have the same kind of tagging scheme to efficiently deal with packed arrays.
CBOR is an IETF standard, unlike the other two which are proprietary protocols.

Hytac commented 1 year ago

Hi, i'm having the following error when using cbor for OccupancyGrid

[rosbridge_websocket-1] [INFO] [1683543613.742826618] [rosbridge_websocket]: [Client 1df4abbf-5909-46b4-9923-855a7d997bec] Subscribed to /map_manager/occupancy_grid [rosbridge_websocket-1] [ERROR] [1683543614.276534924] [rosbridge_websocket]: Exception calling subscribe callback: 'OccupancyGrid' object has no attribute '_slot_types' Any idea?

jesusramondovale commented 11 months ago

Hi, i'm having the following error when using cbor for OccupancyGrid

[rosbridge_websocket-1] [INFO] [1683543613.742826618] [rosbridge_websocket]: [Client 1df4abbf-5909-46b4-9923-855a7d997bec] Subscribed to /map_manager/occupancy_grid [rosbridge_websocket-1] [ERROR] [1683543614.276534924] [rosbridge_websocket]: Exception calling subscribe callback: 'OccupancyGrid' object has no attribute '_slot_types' Any idea?

I'm having exactly the same issue

RobotWebTools / rosbridge_suite