search

akka / alpakka

Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
https://doc.akka.io/libraries/alpakka/current/
Other
1.26k stars 644 forks source link

Generic Camel endpoint connector #74

Closed krasserm closed 7 years ago

krasserm commented 8 years ago

Provide a generic Camel endpoint connector that is compatible with the back-pressure model of Reactive Streams. Camel endpoints that are compatible with back-pressure are, for example, polling consumers, batch consumers and (async) producers. Not compatible are consumers that push messages downstream independent of demand, for example. A generic connector would make a large number of Camel endpoints re-usable for Akka Streams (and will hopefully increase the adoption of Alpakka 😃)

I've done some experimental work into this direction with an FS2 - Camel integration in the Streamz project (although the consumer side doesn't support back-pressure yet). This integration internally uses akka-camel. Using the FS2 - Akka Streams interface one can already use Camel endpoints with Akka Streams in a generic way. Anyway, to have an integration with small runtime overhead, there should be an integration of Akka Streams with the Camel API directly.

Thoughts?

patriknw commented 8 years ago

Sounds great, I have to review the links you provided. Thanks

krasserm commented 8 years ago

Thanks Patrik. The linked FS2 - Camel integration will be obsolete soon. I'm currently about to rewrite it to fully support back-pressure (using the Camel API directly and not akka-camel). This will allow us to have an FS2-based generic connector in the beginning. In a second step, there should be a direct Akka Streams - Camel integration. I'll let you know when I have an update.

patriknw commented 8 years ago

Sounds good, thanks

patriknw commented 7 years ago

@krasserm I took a closer look at https://github.com/krasserm/streamz#dsl-for-as and some of the source code in that project. I have some questions.

krasserm commented 7 years ago

There is no Java API?

It's mentioned in the docs that it is coming soon and there's also a ticket (#27)

How is back-pressure handled? Where are the blocking calls and how are they managed?

Back-pressure is handled by using a polling consumer. Whenever there is downstream demand consumerTemplate.receive is called (with a timeout of 500 ms at the moment). Without demand, the EndpointConsumer switches to waiting state waiting for downstream request, otherwise it is consuming i.e. polling.

It is endpoint-specific how the endpoint internally back-pressures (sometimes the endpoint natively supports polling, sometimes it internally blocks upstream producers when an internal queue reaches maximum capacity but that blocking is Camel internally). There is no Camel API that supports back-pressure in another way.

send and request use producerTemplate.asyncCallback to send messages to an endpoint. Depending on the endpoint, this may be either blocking or non-blocking.

At the moment, the Camel integration doesn't run the endpoint interactions on a dedicated thread. That's a to-do for the next release.

There are still many things I want to experiment with to have a more "reactive" integration (if possible at all with the Camel API). Although the current release is stable (I'm also using it in projects) it is still a POC (or proposal) how a Camel integration could look like.

It would be better to implement with GraphStage instead of ActorPublisher. Do you see any specific reason why that would not be possible?

I found implementing the state machine with actors easier than with getAsyncCallback, no other reason. Why is GraphStage preferred over ActorPublisher?

It looks like there is not much code that is doing the actual camel integration, which is great.

Yeah, I was surprised either 😃 Hope this doesn't have to change ...

What do you think about moving/copying that into Alpakka instead and make it a more pure Camel integration (including javadsl) instead of mixing it with other things in streamz and fs2?

Splitting the streamz-camel module into one for Akka Streams and another one for FS2 is a good idea. What speaks against keeping the Akka Stream - Camel integration in the streamz project?

Thanks for your feedback on the current implementation!

patriknw commented 7 years ago

Thanks for clarifying. I was mostly curious about the back-pressure. That sounds good, but dedicated dispatcher is probably needed. We default to akka.stream.default-blocking-io-dispatcher for such things.

Why is GraphStage preferred over ActorPublisher

ActorPublisher is a low level api for implementing Reactive Streams Publisher with higher risk of getting it wrong. It's not at all as natively integrated with Akka Streams as GraphStage. E.g. it can't be fused with downstream stages, i.e. there will always be an async boundary and those actor messages can be a performance bottleneck for high throughput streams.

I think the API is a matter of how familiar you are with it. GraphStage is pretty nice when you get used to it.

We will deprecate ActorPublisher and ActorSubscriber in Akka 2.5.

streamz

I'm looking for Alpakka Camel integration that is easy to learn, especially for Java developers that already know Camel. Introducing as few new concepts as possible. FS2 is probably the scary part. Do I have to learn that also? What dependencies does that include? This can perhaps also be solved with more structured/targeted documentation. We could have a separate page in the Alpakka documentation for the Camel integration, even though the actual source code is in streamz.

krasserm commented 7 years ago

The Akka Streams - Camel integration of Streamz is completely independent of FS2. After splitting the current streamz-camel jar into one for Akka Streams (e.g. streamz-camel-akka) and another one for FS2, the only dependencies of streamz-camel-akka will be Akka Streams and Camel (no FS2).

I will also move the Akka Streams - Camel documentation into a separate self-contained markdown file that users can read without seeing anything of FS2 and that can be linked to or inlined from Alpakka (not sure if markdown or paradox support inlining).

From our discussion I see the following next steps ordered by descending priority:

Step1:

Step 2:

Step 3:

If I find enough time I'll also include step 2 into the 0.7 release but I cannot promise. Is there a planned date for the next Alpakka release? I'll try to have Streamz 0.7 ready before. WDYT?

patriknw commented 7 years ago

Sounds excellent, @krasserm We release Alpakka frequently whenever something interesting enough has been merged.

krasserm commented 7 years ago

Alright, I'm quite busy at the moment but I hope to have step 1 ready in 2 weeks or so.

patriknw commented 7 years ago

thanks

krasserm commented 7 years ago

Step 1 is completed:

krasserm commented 7 years ago

I'm now resuming work on this issue with a slight modification of the plan:

Step 2:

Step 3:

After step 2, applications will then be able to use all Camel endpoints, including those consumer endpoints that require an in-out message exchange. I'll cut a new Streamz release then.

johanandren commented 7 years ago

Sounds good, One problem with upgrading to 2.5 here is that it may make the connector impossible to use with 2.4 (we're binary backwards compatible but not forward binary compatible), best would be if you could use 2.4 but test it also with 2.5 as that has stricter guard rails for custom graph stages.

krasserm commented 7 years ago

Ok, will revert to 2.4 then. There are no features from Akka 2.5 I need at the moment. What are the plans for migrating Alpakka to Akka 2.5?

johanandren commented 7 years ago

We haven't said anything explicitly, but at the latest when we EOL 2.4, I think that will be the end of the year. We want it to work with 2.5 though, so we should probably make the connectors run their tests on both 2.4 and 2.5 even if we list 2.4 as the default target Akka version.

2m commented 7 years ago

Thanks for reminding. I had this from earlier, but not pushed: https://github.com/akka/alpakka/pull/305

krasserm commented 7 years ago

Streamz 0.8 is now released https://github.com/krasserm/streamz/releases/tag/v-0.8. Any objections to close this issue with that release? It covers all items mentioned in step 2. Step 3 are only internal refactorings and not user-facing. They are tracked at https://github.com/krasserm/streamz/issues.

patriknw commented 7 years ago

Sounds good, thanks @krasserm