Producer Security with Kerberos

[d1820]

In the current version of Kafka 9.0.x. The producer/consumer API can support kerberos authentication to an AD server. In review your code I dont clearly see a way to do this. Is this not currently possible? From the java perspective possible by running these steps:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_secure-kafka-ambari/content/ch_secure-kafka-config-options.html#d6e422

Any insight to this? or how i would be able to accomplish this?

Update - looks as if it could possibly be done by editing the KafkaTcpSocket:GetStreamAsync() but this is a private method. Would need to be able to get access to the networkStream so could wrap that in a NegotiateStream.

Any way we can add a optional Func to the KafkaTcpSocket class to allow for a way to get the stream and alter it before its returned and allow for a task System.IO.Stream return type or maybe use the visitor/decorator pattern on the stream? Otherwise I think the only other option would be to re-create the KafkaTcpSocket class and make the changes..

Thanks

[natami]

Did you guys find a solution to do kerberos authentication for both consumer/producer?

[d1820]

I created a branch out there that implements decorators into the producer at least which will allow you to implement your own security on the stream. This will let you do Kerberos. You will need to ensure all the correct service principal names are set up on auth server as well..

Ultimately I ended up just writing my stuff in Java which natively supported Kerberos in the Kafka api.. Was super simple to do using all the examples online from Kafka sight.. We didn't like the fact that the. Net lib had not been very active in putting in fixes and updates and we didn't want to support parallel branches. On Apr 27, 2016 6:21 AM, "natami" notifications@github.com wrote:

Did you guys find a solution to do kerberos authentication for both consumer/producer?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/Jroland/kafka-net/issues/82#issuecomment-215080736

[natami]

Thanks for your answer ;)

Did you end up with any code I can reuse?

[d1820]

For Kafka.net there is a branch out there.. Added support for stream decorator pattern in tcpsocket #84

That is how can implement Kerberos stuff. From there can follow example from Microsoft on using negotiated streams On Apr 27, 2016 12:35 PM, "natami" notifications@github.com wrote:

Thanks for your answer ;)

Did you end up with any code I can reuse?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/Jroland/kafka-net/issues/82#issuecomment-215203784

[bcwtang]

Thanks for sharing the branch. I'm new to .net so I'm not sure how to set the principal names in the decorator you created. Can you give me some hints? THanks.

[bcwtang]

Thanks very much. Really appreciate it. I'm curious how I can get the stream to convert to NetworkStream to wrap for NegotiateStream and encrypt the producer to send the msssage. I know I have to do something with the decorator class but I have no idea how to use it. Thanks again and look forward to your example.

[d1820]

Here is a decorator and how you can set it up. there is also links available for help getting s SPN setup (which is required for Kerberos to work).

You can also pull the branch down and set kafka.net references to that source branch to debug though it if needed as well.

I found most of the issues with it all was getting the correct SPN and AD stuff all talking and working correctly

public class KerberosStreamDecorator : IStreamDecorator
    {
        private readonly string _userName;
        private readonly string _password;
        private readonly string _domain;
        private readonly string _servicePrincipalName;

        //servicePrincipalName is created on the server hosting kafka
        //see https://technet.microsoft.com/en-us/library/dd632778.aspx
        //see http://blogs.iis.net/brian-murphy-booth/the-biggest-mistake-serviceprincipalname-s
        //see http://serverfault.com/questions/350782/can-someone-please-explain-windows-service-principle-names-spns-without-oversi
        //see https://blog.rhysgoodwin.com/windows-admin/active-directory-and-kerberos-spns-made-easy/
        public KerberosStreamDecorator(string userName, string password, string domain, string servicePrincipalName)
        {
            _servicePrincipalName = servicePrincipalName;
            _domain = domain;
            _password = password;
            _userName = userName;
        }

        public Type WrapStreamType => typeof (NegotiateStream);

        public Stream WrapStream(Stream stream)
        {
            if (stream == null) return null;
            NegotiateStream neg = new NegotiateStream(stream, false);
            NetworkCredential credentials = new NetworkCredential(_userName, _password, _domain);
            //credentials = CredentialCache.DefaultNetworkCredentials;
            try
            {        
                //there arew different overloads here, you may need to expirement with what works os is needed for your env.
                neg.AuthenticateAsClient(credentials, _servicePrincipalName,  ProtectionLevel.None, TokenImpersonationLevel.Identification);
            }
            catch (Exception ex)
            {
                var error = ex.Message;               
            }
            if (!neg.IsAuthenticated)
            {
                throw new UnauthorizedAccessException($"Access to kafka denied. User: {_userName}");
            }

            return neg;
        }
    }
public class RunIt{
     private readonly Producer _producer;

    //can inject the params and create an instance manually  or susing some type of DI to inject KerberosStreamDecorator automatically in class
    public void StartProducer(string userName, string password, string domain, string servicePrincipalName){
        var router = new BrokerRouter(new KafkaOptions{

            StreamDecorator = new KerberosStreamDecorator(username, password, domain, servicePrincipalName);
        });
        _producer = new Producer(router, 100, 1000);

    }

     public async Task<List<ProduceResponse>> SendMessageAsync(string topic, IEnumerable<Message> messages, Int16 acks = 1,
                                                            TimeSpan? timeout = null, MessageCodec codec = MessageCodec.CodecNone)
    {
        return await _producer.SendMessageAsync(topic, messages, acks, timeout, codec);
    }
}
[Example.cs.txt](https://github.com/Jroland/kafka-net/files/324812/Example.cs.txt)

[bcwtang]

Thanks very much! You are a life saver! Really appreciate it! I'll try it out and let you know if I have any questions. Thanks again!!

[bcwtang]

Quick question on SPN. I was provided with a principal name in this format username@domain, i believe this is what they created from the kafka server (hosted on linux). Can I use that as SPN because I read the articles you provided, it appears I need an SPN not an UPN, is that true?

[d1820]

You will need a spn which is different.. the principal will also be used as the credentials for stream On Jun 22, 2016 12:37 PM, "bcwtang" notifications@github.com wrote:

Quick question on SPN. I was provided with a principal name in this format @, i believe this is what they created from the kafka server. Can I use that as SPN because I read the articles you provided, it appears I need an SPN not an UPN, is that true?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Jroland/kafka-net/issues/82#issuecomment-227853460, or mute the thread https://github.com/notifications/unsubscribe/AAXDH3yTM5J-0d4hRfb7nwDUcPtokLuKks5qOY79gaJpZM4HJ7iu .

[sskapci]

Hello, I am trying to implement your changes to my code. But i am getting Invalid receive size error. Did you get this error?

[veereev]

I tried to use this kerberos stream decorator. I am getting unauthorized exception. "access denied to kafka". Anyone able to run this successfully? I feel that is an issue with the service principal name. I have tried "serviceid@domain.com", "kafka/domain.com@domain.com", "kafka/_HOST@domain.com". But no luck.

[d1820]

When i originally wrote this and used it I was able to get it to work, but i did have to work with the admin team to ensure the SPN was created correctly. It took a few times getting it. To be honest I dont remember what the finally combination was. Me and the Admin team ran against the scenarios outlined in the above links.

[veereev]

Thanks for the prompt response. Actually, in my scenario we have the linux brokers secured with kerberos keytabs. So in order for me to use the .net producer with kerberosstreamdecorator, i need to configure the kerberos over AD rather than the keytab system. Am I getting it correct ? I dont see any .net library that supports the keytabs like Jaava jaas does. Unfortunately, I have to implement only in .net as my current data producing application is in .net and we need to integrate with kafka by writing data to kafka.