A CLI for consuming protobuf messages over Kakfa.
Serializing Kafka messages to protobuf has many benefits, but it can make debugging and peeking at messages can be difficult. Milena aims to make it easy to produce and consume protobuf messages over Kafka using JSON.
-T
cargo install --git https://github.com/VanceLongwill/milena
docker pull vancelongwill/milena:latest
Usage: milena [OPTIONS] <COMMAND>
Commands:
consume Consume & decode protobuf messages to JSON
produce Encode JSON messages as protobuf and produce them to a topic
decode Decode an arbitrary protobuf message
encode Encode an arbitrary protobuf message
serve Start a HTTP server that can proxy kafka/protobuf to http/json
help Print this message or the help of the given subcommand(s)
Options:
-F, --config <FILE>
Sets a custom config file
-f, --file-descriptors <FILE_DESCRIPTORS>
The path to the protobuf file descriptors [default: ./descriptors.binpb]
-X, --rdkafka-options <RDKAFKA_OPTIONS>
A catchall for specifying additional librdkafka options
-v, --verbose...
Enable verbose logging, can be repeated for more verbosity up to 5 times
-h, --help
Print help
-V, --version
Print version
Based on the docker example.
milena produce \
-X bootstrap.servers=localhost:9092 \
--topic sometopic \
--file-descriptors descriptors.binpb \
--message-name example.v1.UserUpdated \
--key 123 \
--header 'message-name=example.v1.UserUpdated' \
--data '{"name": "John Smith", "age": 40}'
milena consume \
-X bootstrap.servers=localhost:9092 \
-t sometopic \
-f descriptors.binpb \
-H "message-name" \
-o=beginning
milena
can also proxy kafka/protobuf to HTTP/json using the serve
command. Each request starts a new consumer stream.
This provides a number of advantages:
milena serve -p 29999 \
-X bootstrap.servers=localhost:9092 \
-f descriptors.binpb
The JSON request body can contain any option provided to the
consume
command. Seemilena consume -h
for more details.
curl localhost:29999 -H 'Content-type: application/json' -d '{
"topic": "sometopic",
"message_name_from_header": "message-name",
"exit_on_last": true
}'
(while also simultaneously piping to jq
for pretty printing)
curl localhost:29999 -H 'Content-type: application/json' -d '{
"topic": "sometopic",
"message_name_from_header": "message-name",
"offsets": ["end"]
}' -N -s | jq -rc
-N
prevents curl buffering the output so we can can pipe it to jq as soon as we receive something.-s
hides other output from curl
See milena encode -h
or milena decode -h
for more options.
milena encode -N example.v1.UserUpdated '{"name": "alice", "age": 90}' |\
milena decode -N example.v1.UserUpdated
{"name":"alice","age":90}%
Set the verbose flag up to 5 times -vvvvv
to get increasingly more
detailed logging.
Ensure that openssl@1.1
installed with brew & set the following environment variables.
export OPENSSL_ROOT_DIR=$(brew --prefix openssl@1.1)
export OPENSSL_LIBRARIES="${OPENSSL_ROOT_DIR}/lib"
Protobuf file descriptors can be generated using the buf CLI or with protoc.
buf build ./ -o descriptors.binpb
protoc example.proto -o protocdescriptors.binpb