infinyon / fluvio

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.
https://www.fluvio.io/
Apache License 2.0
3.88k stars 491 forks source link

Question: Is fluvio suitable for sending binary messages? #1963

Closed paucarre closed 2 years ago

paucarre commented 2 years ago

As far as I can see in the fluvio api the messages sent are string. I was wondering if fluvio is designed in a way where binary (e.g. byte arrays or numerical values) data can be sent without serializing to string type.

nicholastmosher commented 2 years ago

Hi @paucarre, yes Fluvio is designed to work with binary payloads. With the Rust API, you may pass any type to producer.send() that can convert to a byte array (i.e., it implements Into<Vec<u8>>). Here are some examples of binary values you can send with the Rust client library:

use fluvio::RecordKey;

producer.send(RecordKey::NULL, b"hello").await?; // No key, value: &[u8]
producer.send("00001", vec![104, 101, 108, 108, 111]).await?; // Key: &str, value: Vec<u8>

// Send an arbitrary file
let file_data: Vec<u8> = std::fs::read("./file.bin")?;
producer.send("file.bin", file_data).await?;

You can also use the Fluvio CLI to send binary files by using the --raw flag on fluvio produce (by default, it treats files as text and sends each line as an independent record), like so:

$ fluvio produce my-topic --raw -f ./file.bin

You can find more info about the Producer API in Rust here and details on the CLI producer here