Streaming from nats Jetstream into the rerun web gui

[gedw99]

I have a multi modal sensor fusion project where we need to train a convolution neural network .

the data is streamed in real time using nats Jetstream from various platforms like cars and trucks and airplanes.

I have never used rerun , and discovered the project via open pilot / comm.ai.

Nats Jetstream is 100% open source and has has golang, rust and other language sdk, but it can also stream using web sockets .

it is sort of like bgp anycast in that a client connects directly to the nearest geo physical server and then starts streaming data up and down. There is no dns or other stuff in the way.

I am hoping that someone could point me to the sone code and example data where I can see the data frames that rerun gui needs ? I read that the viewer can also take a file and the plugins fingerprint it to work out which plugin will handle the decoding of the “ file type “ so over web sockets I would like help on knowing that data feed structure ..

there is no golang sdk st the moment I assume ?

[jleibs]

No goland bindings yet, only rust, c++, and python.

Although Rerun does have websocket support, the viewer only knows how to receive rerun-encoded data over the websockets.

Most users working with similar structure (such as ROS graphcs) end up implementing a separate "bridge" application that reads data in the native-format and then uses the Rerun logging-API to send the transformed data to rerun.

Here's an example of a C++ app that does this for ROS data: https://github.com/rerun-io/cpp-example-ros-bridge/blob/main/rerun_bridge/src/rerun_bridge/visualizer_node.cpp

Our doc-pages for the respective types: https://rerun.io/docs/reference/types/archetypes all have examples of using the logging APIs in the different languages to send data to Rerun.

[gedw99]

Thanks . That confirms what I saw in the docs. It was not sure what does what.

There is a similar project called cogent written in golang btw. It compiles to WASM and gives similar things to your gui. Its video capabilities are not as good though .

My project needs to take video feeds and data feeds and then does convulsion nets to transform the videos onto a 3d model.

I was hoping rerun helped there and I could tap into it .I have a ton of code doing the transformation but hoped I could leverage rerun to help with the convolution nets .

[gedw99]

I might be able to use to interface in

https://github.com/cogentcore/core/tree/d84b6840cf4b7881ea3c1fb6bdcef592e16eabfa/tensor

it’s only data frames.

cogent is similar to rerun. GUI is WASM. GUI config it fully controlled. But it’s got a long way to go