search

assimbly / consumewithcamel

Apache NiFi processor that uses Apache Camel to consume messages
Apache License 2.0
0 stars 1 forks source link

Nifi Attributes -> Camel Headers -> Nifi Attributes #1

Closed fliot closed 4 years ago

fliot commented 4 years ago

Hi,

First of all, I really like your approach, it makes really possible to use Camel components from Nifi immediately, and without third party deployment ! Even, if I have some doubt this implementation could really handle heavy loads, since creating routes on execution, instead of maintaining a Spring context, may be a performance bottleneck.

Nevertheless, it's elegant, and working ! Then, a pleasure to use it.

My post is not a bug, but clearly a road-map and cooperation proposal :-)

I believe there is a real interest to deal deeper with camel headers. As you probably know, headers are a big matters on camel components execution, and managing them is a very important functional requirement.

Here is my functional case : (1) Nifi : GetFile (2) Nifi : LogAttribute (3) Nifi : ProduceWithCamel -> log:com.mycompany.order?showAll=true&multiline=true (4) Nifi : LogAttribute (5) Nifi : PutFile nifi-process-group producewithcamel-logs.txt

It does work but :

a) Any of the FlowFile attributes are available within the camel route. I believe it worth to make them available prefixed ("nifi.xxx") as a camel header.

b) The Camel Headers, are not transferred back to nifi FlowFile. I believe it worth to make them available prefixed ("camel.xxx") as a nifi attribute.

c) on ProduceWithCamel nifi processor, it would be fabulous being able to add prefixed "full text" properties, which will be reused as camel header. Typically, supporting such configuration example will be great : camel.CamelHttpMethod = POST camel.Content-Type = application/xml; charset=utf-8 camel.CamelHttpUri = http://some.bind.point/there To_URI = http://there?httpClient.cookiePolicy=ignoreCookies&httpClient.soTimeout=30000 prefixed-freetext-properties

I review quickly your code, and my proposal would impact producewithcamel and connector projects, Let me know if you are interested by some coordinates efforts to reach such functionalities.

assimbly commented 4 years ago

Hi Francois,

Thanks for your positive feedback.

Both processors (ConsumeWithCamel) and (ProduceWithCamel) were created as Proof-Of-Concept. Just to proof it could be done. They are not production-ready. For this I use Assimbly gateway (which creates an independent CamelContext). See also https://github.com/assimbly/gateway.

Mostly the integration pattern I use is:

Assimbly Gateway (Connector + ActiveMQ) --> NiFi

I use this pattern to separate Connector/Transport from processing logic. NiFi could in this respect replaced by another platform. When using Assimbly Gateway headers can be used (https://github.com/assimbly/gateway/wiki/headers) when picked up from an ActiveMQ queue body/headers are automatically translated to a flowfile.

Of course using just one platform has also positive sides :)

To comment some of your points in some details:

1) Load

"I have some doubt this implementation could really handle heavy loads, since creating routes on execution, instead of maintaining a Spring context, may be a performance bottleneck."

This is true. To scale the CamelContext probably should be created/handled in a separate "Controller".

2) Headers

Yes, that would be nice to handle such headers in this way (with prefix and handling of such headers). Feel free to make a pull request. Otherwise when I find some time I can try to build it in.

Raymond

assimbly commented 4 years ago

Ah I see the pull request is on the Producer, I will look into it.