search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
277 stars 3.15k forks source link

Unsafe to create producer within consumer callback when using SSL. #1332

Closed alanbutt closed 7 years ago

alanbutt commented 7 years ago

Description

It is unsafe to create a new producer within a consumer callback when using SSL protocols.

In our normal operations a message is received analyzed and easily dealt with. However, rarely a message requires deeper analysis. When this happens, we process the message as normal and then send the message onto a different topic where the deeper analysis will happen. This has been working fine when using plaintext protocols.

We recently changed to SASL_SSL (plain) so we could add some ACLs to the topics. When the producers were created used we would crash usually in OpenSSL crypto error handling and see heap corruptions.

Our service was is written c#. To analyse this I created a test application in C/C++ that mimics the operations. From this perspective it quickly became obvious that creation of producers that use SSL within the callback is not safe (use of sasl doesn't appear to be a factor). The simple work around is to perform the "produce" operations on another thread.

However, it was not obvious in the C# event handlers that this is unsafe (and I assume when using other "higher" level language bindings).

I would suggest that the restrictions are documented and that the non-thread safe aspects are addressed as possible.

How to reproduce

1) Put several thousand messages into the topic spread across several partitions (I used 8 partitions). 2) Create a multi-threaded consumer--one thread per partition on the topic use the SSL protocol (always start beginning offset so we can repeat the failure easily). 3) In consumer callback, create a producer to a different topic (also using SSL protocol), produce the message, flush and discard the producer (do this for every message as want the crash to happen rapidly).

Note: crash doesn't happen if plaintext is used.

Checklist

Please provide the following information:

edenhill commented 7 years ago

interesting, can you share your C/C++ reproduction code?

alanbutt commented 7 years ago

Here you go. Readme in the .zip

rdtest.zip

alanbutt commented 7 years ago

Did source code reproduce issue?

From: Magnus Edenhill [mailto:notifications@github.com] Sent: Tuesday, July 25, 2017 1:44 AM To: edenhill/librdkafka librdkafka@noreply.github.com Cc: Alan Butt alan.butt@fidelissecurity.com; Author author@noreply.github.com Subject: Re: [edenhill/librdkafka] Unsafe to create producer within consumer callback when using SSL. (#1332)

interesting, can you share your C/C++ reproduction code?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/edenhill/librdkafka/issues/1332#issuecomment-317657203, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AblrngvdfyQ0PNV7hx6_Wys52tmIe2K0ks5sRZzLgaJpZM4Obw8N.

edenhill commented 7 years ago

Believe this is fixed in fee1e42131b13b6cb37beb59e3e0b0fd9dc21416

Can you try to reproduce on latest master?

alanbutt commented 7 years ago

Yes! this did fix the problem. Sorry it to so long to test.