Feature request: remote access to RocketLogger data through some streaming, machine-readable manner

[gemarcano]

Currently, as far as I can tell, RocketLogger data can be accessed through:

1. Web interface (exposed by the nodejs webserver)
2. Interactive mode (exposed through ncurses)
3. Output file (saved to the filesystem)

I don't know of a way to read live data from the RocketLogger and ingest it on a remote computer. For my particular use case, we want to act on data we're sensing from the RocketLogger, and the control software is on the computer that's connected to the RocketLogger. Does functionality like this already exist and I just missed it?

[jmadden173]

I would be interested in this as well

[lsigrist]

@gemarcano thanks for your question regarding use case that we have not considered so far. Currently, only the three options you listed are provided as public (and documented) interfaces to the measurement data.

However, as of release 2.0 there is an internal (undocumented) interface for streaming measurement data blocks from the RocketLogger measurement binary to the nodejs server. This ZeroMQ based interface follows the publisher-subscriber pattern, where the RocketLogger binary publishes a multi-part message for every measurement data block (at the rate configurable via the --update argument).

In a nutshell, the ZeroMQ data message structure looks a follows:

1. JSON formatted metadata: measurement data rate, and the sampled channel info (name, scale, unit, bit offset for binary channels)
2. data block timestamps: the same that are stored in the data file
3. channel data as signed 32-bit integers: there is an individual message part for each channel, the order is as specified in the metadata
4. last message part: binary channel data aggregated into 32-bit values, same as in the data file format

The use of the interface is probably best illustrated by the following Python script:

import json
import numpy as np
import zmq

# ZeroMQ socket identifier for status publishing
STATUS_SOCKET = "tcp://127.0.0.1:8276"

# ZeroMQ socket identifier for data publishing
DATA_SOCKET = "tcp://127.0.0.1:8277"

# timestamp block data format
DT_TIMESTAMP = np.dtype(
    [
        ("realtime_sec", "<M8[s]"),
        ("realtime_ns", "<m8[ns]"),
        ("monotonic_sec", "<M8[s]"),
        ("monotonic_ns", "<m8[ns]"),
    ]
)

context = zmq.Context()

print("subscribe to RocketLogger data socket")
socket = context.socket(zmq.SUB)
socket.connect(DATA_SOCKET)
socket.subscribe("")

while True:
    message = socket.recv_multipart()

    # 1. JSON channel metadata
    meta = json.loads(message[0])
    print(f"data received: {meta}")

    # 2. data block timestamps
    time = np.frombuffer(message[1], dtype=DT_TIMESTAMP)
    print(f"time: {time}")

    has_binary_channel = False
    channel_index = 2
    # 3. non-digital channels in order of channel metadata
    for channel in meta["channels"]:
        if channel["unit"] == "binary":
            has_binary_channel = True
            continue

        data = np.frombuffer(message[channel_index], dtype="<i4")
        print(channel["name"], data)

        channel_index += 1

    # 4. binary channel bitmaps (if any listed in metadata)
    if has_binary_channel:
        digital = np.frombuffer(message[channel_index], dtype="<u4")
        print(digital)

I tested this to work with release 2.0.2 when executing on the RocketLogger system. Remote access of the interface is not supported, as only local connections are accepted. But if you're not afraid of recompiling the rocketlogger binary: with the following patch, connection from any remote host can be accepted:

diff --git a/software/rocketlogger/rl.h b/software/rocketlogger/rl.h
index b88f347..a1b234d 100644
--- a/software/rocketlogger/rl.h
+++ b/software/rocketlogger/rl.h
@@ -118,9 +118,9 @@
 #define RL_SHMEM_PERMISSIONS 0666

 /// ZeroMQ socket identifier for status publishing
-#define RL_ZMQ_STATUS_SOCKET "tcp://127.0.0.1:8276"
+#define RL_ZMQ_STATUS_SOCKET "tcp://*:8276"
 /// ZeroMQ socket identifier for data publishing status
-#define RL_ZMQ_DATA_SOCKET "tcp://127.0.0.1:8277"
+#define RL_ZMQ_DATA_SOCKET "tcp://*:8277"

 /**
  * RocketLogger data aggregation modes.

[lsigrist]

What I skipped in the reply above:

The 2nd half of the connection to the nodejs server is the measurement status interface. On this interface, JSON formatted measurement status info is published on every data block update (measurement config, sample count, etc.).

import json
import zmq

# ZeroMQ socket identifier for status publishing
STATUS_SOCKET = "tcp://127.0.0.1:8276"

context = zmq.Context()

print("subscribe to RocketLogger status socket")
socket = context.socket(zmq.SUB)
socket.connect(STATUS_SOCKET)
socket.subscribe("")

while True:
    message = socket.recv()
    status = json.loads(message)
    print(f"status received: {status}")

[lsigrist]

This feature is now officially documented for release 2.1.0 (#99).