search

twmb / kcl

Your one stop shop to do anything with Kafka. Producing, consuming, transacting, administrating; 0.8.0 through 3.2+
BSD 3-Clause "New" or "Revised" License
194 stars 19 forks source link

Add a pair of options to kcl consume for unmarshalling kafka messages in protobuf to json #16

Closed moming00 closed 1 year ago

moming00 commented 1 year ago

hello, it will be good if kcl supports unmarshalling kafka messages in protobuf format to Json. So introduced two commandline options: --proto-file to specify the Proto Source File or ProtoSet File which includes the original message defination, --proto-message to specify the message name in "packageName.messageName" format. For example: kcl consume mytopic -n=1 --proto-file=/usr/local/src/proto_Generator/sample.proto --proto-message=sample.FileState kcl consume mytopic -n=1 --proto-file=/usr/local/src/proto_Generator/sample.protoset --proto-message=sample.FileState

this could return message like: { "Message": "Hello World", "Id": "7b1271413a48a80cfa0c38e83788b32a", "Timestamp": "2022-12-13T16:03:30.760+08:00", "HostName": "ck10", "Path": "/data01//flow.log", "RowNumer": "59", "Seq": "59", "TaskId": "111359604125616", "Topic": "fc111359604125096", "IP": "192.168.110.66" }

proto-file with "protoset" format supposed to be more efficient, as which already includs compiled info from source proto file, so requires no parsing in runtime. Aslo protoset file could be self-contained if --include_imports is specified when generating the file, while proto souce file may require additional efforts to locate the imported protos.

moming00 commented 1 year ago

there is voice suggesting to use the compiled proto file, will reopen the PR after everything settled

twmb commented 1 year ago

fwiw compiled proto file might be difficult to use as well -- I usually work with the non-compiled files. Perhaps support both in the same flag?

moming00 commented 1 year ago

fwiw compiled proto file might be difficult to use as well -- I usually work with the non-compiled files. Perhaps support both in the same flag?

this may be the solution =)

twmb commented 1 year ago

Awesome, lgtm, ty. I'll bump the latest franz pin and release.