adevinta / zoe

The Kafka CLI for humans
https://adevinta.github.io/zoe
MIT License
287 stars 21 forks source link

[Bug] jq dialect doesn't work for `zoe topics produce` #51

Open whatsupbros opened 2 years ago

whatsupbros commented 2 years ago

When I run such a command:

zoe --cluster local topics produce --help

I am getting this feedback:

Usage: zoe topics produce [OPTIONS]

  produce messages into topics

Options:
  --dry-run              Do not actually produce records
  -t, --topic VALUE      Topic to write to
  --subject TEXT         Avro subject name to use
  -k, --key-path TEXT    Jmespath (or jq) expression to extract the key
  -v, --value-path TEXT  Jmespath (or jq) expression to extract the value
  --ts-path TEXT         Jmespath (or jq) expression to extract the timestamp
  --streaming            Read data line by line continuously
  --timeout INT          Timeout in millis
  --from-stdin           Consume data from stdin
  --from-file PATH       Consume data from a json file
  -h, --help             Show this message and exit

Examples:

  Produce a record into the 'input' topic from stdin (requires input to be a json array):
  > echo '[{"id": "1", "msg": "hello"}]' | zoe -c local topics produce -t input --from-stdin

  Use streaming mode to accept records one by one (does not require input to be a json array):
  > echo '{"id": "1", "msg": "hello"}' | zoe -c local topics produce -t input --from-stdin --streaming

  Use the id field of the input messages to determine the Kafka record key:
  > echo '[{"id": "1", "msg": "hello"}]' | zoe -c local topics produce -t input --from-stdin --key-path 'id'

  Write the data from a json file (requires the content in the file to be a json array):
  > zoe -c local topics produce -t input --from-file data.json

  Pipe data from another topic:
  > zoe -c remote topics consume input --continuously | zoe -c local topics produce -t output --from-stdin --streaming

Which suggests that I can use either jmespath or jq for -k and -v options.

However, when I try to use jq syntax, I am getting the following error:

zoe --cluster remote topics consume my-topic --max-records 1 --expose-metadata \
| zoe --cluster local topics produce --from-stdin --topic my-topic --subject my-topic-value \
--key-path ".__metadata__.key" \
--value-path ". | del(.__metadata__)" \
--streaming

Error:

2022-05-18 14:01:46 INFO zoe: producing '1' records to topic 'my-topic'
failure: runner 'local' failed
cause:
  error: Invalid jmespath expression '.__metadata__.key'

The option --dialect (by analogy with zoe topics consume) also doesn't work:

Error: no such option: "--dialect".

The reason I am trying to use jq and not jmespath is because I need to pipe data between topics in different clusters. And my topic has a key, which I also need to pipe.

I haven't found another way to do this easily other than to use --expose-metadata option for the consumer, and then remove the __metadata__ attribute on the producer side for the value, which is not possible with jmespath (at least I don't know the way), but can be easily done with jq.


zoe version print output:

{"projectVersion":"0.28.0","buildTimestamp":"2021-11-04T23:39:18.809435","createdBy":"Gradle 7.2","buildJdk":"14.0.2 (Azul Systems, Inc. 14.0.2+12)","buildOS":"Linux amd64 5.11.0-1020-azure"}