search

whiver / nifi-protobuf-processor

An Apache NiFi processor to encode and decode data using Google Protocol Buffer schemas.
MIT License
36 stars 20 forks source link

protobuf-decoder #2

Open Htuotuo opened 5 years ago

Htuotuo commented 5 years ago

I am using the .proto mode. During the decoding process, my hard disk will generate a 5M protocjarxxx file. I don't want it to generate this file. What should I do?

Htuotuo commented 5 years ago

image like this!

whiver commented 5 years ago

Hi :) The .proto mode is in fact simply a way to dynamically compile the .proto file at runtime, in case you have a dynamic schema or to ease the dev loop for example. Though, you'll still have to build the schema file, and to do that the we download the protoc compiler and store it in a temp folder. That's why you got this file, and while you'll be using the proto mode, you'll have this file.

Htuotuo commented 5 years ago

Will this temporary file be deleted automatically? I use (protoc --descriptor_set_out=D:\TSBrowserDownloads\embedded-jetty-websocket-examples-master\native-jetty-websocket-example\src\main\java\org\eclipse\jetty\demo\proto\netcarBaseInfoCompanyStat.desc netcarBaseInfoCompanyStat.proto) The desc file generated by this command will report an error in nifi. I use the .proto mode in nifi. image I added the above image property before the protocolDecoder processor. If I don't want to generate the protoc.exe file, should I use the desc mode?

whiver commented 5 years ago

The file will be deleted on reboot depending on the config of your server, since they are stored in the temp folder. Concerning your invalid .desc file, make sure that you embed all the dependencies of your .proto file at compile time (there is an option for that in the protoc compiler, I don't remember which one), as the Protobuf plugin only loads a single .desc file, it has to be standalone.

And if you want to disable the runtime compilation of the proto file, simply set the protobuf.compileSchema property to false in the processor :)

Htuotuo commented 5 years ago

ok,thanks,Let me try

Htuotuo commented 5 years ago

hello,whiver ,I am using Unable to decode data: Protocol message tag had invalid wire type when using .proto compiled desc decoding in the following format. Is the desc file not supporting such multi-level nested parsing?
image

Htuotuo commented 5 years ago

The above picture is the .proto source file, which has been used on other platforms.

Htuotuo commented 5 years ago

image This is my configuration；