Usage: zoe topics produce [OPTIONS]
produce messages into topics
Options:
--dry-run Do not actually produce records
-t, --topic VALUE Topic to write to
--subject TEXT Avro subject name to use
-k, --key-path TEXT Jmespath (or jq) expression to extract the key
-v, --value-path TEXT Jmespath (or jq) expression to extract the value
--ts-path TEXT Jmespath (or jq) expression to extract the timestamp
--streaming Read data line by line continuously
--timeout INT Timeout in millis
--from-stdin Consume data from stdin
--from-file PATH Consume data from a json file
-h, --help Show this message and exit
Examples:
Produce a record into the 'input' topic from stdin (requires input to be a json array):
> echo '[{"id": "1", "msg": "hello"}]' | zoe -c local topics produce -t input --from-stdin
Use streaming mode to accept records one by one (does not require input to be a json array):
> echo '{"id": "1", "msg": "hello"}' | zoe -c local topics produce -t input --from-stdin --streaming
Use the id field of the input messages to determine the Kafka record key:
> echo '[{"id": "1", "msg": "hello"}]' | zoe -c local topics produce -t input --from-stdin --key-path 'id'
Write the data from a json file (requires the content in the file to be a json array):
> zoe -c local topics produce -t input --from-file data.json
Pipe data from another topic:
> zoe -c remote topics consume input --continuously | zoe -c local topics produce -t output --from-stdin --streaming
Which suggests that I can use either jmespath or jq for -k and -v options.
However, when I try to use jq syntax, I am getting the following error:
2022-05-18 14:01:46 INFO zoe: producing '1' records to topic 'my-topic'
failure: runner 'local' failed
cause:
error: Invalid jmespath expression '.__metadata__.key'
The option --dialect (by analogy with zoe topics consume) also doesn't work:
Error: no such option: "--dialect".
The reason I am trying to use jq and not jmespath is because I need to pipe data between topics in different clusters. And my topic has a key, which I also need to pipe.
I haven't found another way to do this easily other than to use --expose-metadata option for the consumer, and then remove the __metadata__ attribute on the producer side for the value, which is not possible with jmespath (at least I don't know the way), but can be easily done with jq.
zoe version print output:
{"projectVersion":"0.28.0","buildTimestamp":"2021-11-04T23:39:18.809435","createdBy":"Gradle 7.2","buildJdk":"14.0.2 (Azul Systems, Inc. 14.0.2+12)","buildOS":"Linux amd64 5.11.0-1020-azure"}
When I run such a command:
I am getting this feedback:
Which suggests that I can use either
jmespath
orjq
for-k
and-v
options.However, when I try to use
jq
syntax, I am getting the following error:Error:
The option
--dialect
(by analogy withzoe topics consume
) also doesn't work:The reason I am trying to use
jq
and notjmespath
is because I need to pipe data between topics in different clusters. And my topic has a key, which I also need to pipe.I haven't found another way to do this easily other than to use
--expose-metadata
option for the consumer, and then remove the__metadata__
attribute on the producer side for the value, which is not possible withjmespath
(at least I don't know the way), but can be easily done withjq
.zoe version print
output: