Basic Authentication not working

[vthorwat]

Hello ,

As per the documentation & application.example.yaml from this page we are trying to implement basic authentication with single user as of now.

But it seems we may be missing something.

Can anyone help us to validate the configurations for KafkaHQ.

micronaut:
security:
enabled: true
server:
port: 7070
kafkahq:
connections:
bdl-dev:
properties:
bootstrap.servers: "BROKER:6667,BROKER:6667,BROKER:6667"
security.protocol: SASL_PLAINTEXT
authorizer.class.name: kafka.security.auth.SimpleAclAuthorizer
sasl.kerberos.service.name: kafka
sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" principal="kafka/HOSTNAME@REALM.COM";

security:
default-groups:
-
groups:
admin:
roles:
- topic/read
- topic/insert
- topic/delete
- topic/config/update
- node/read
- node/config/update
- topic/data/read
- topic/data/insert
- topic/data/delete
- group/read
- group/delete
- group/offsets/update
- registry/read
- registry/insert
- registry/update
- registry/delete
- registry/version/delete
- acls/read
- connect/read
- connect/insert
- connect/update
- connect/delete
- connect/state/update
attributes:
topics-filter-regexp: "test.*"

topic-reader:
roles:
- topic/read
attributes:
topics-filter-regexp: "test\\.reader.*"
basic-auth:
user: admin
password: 6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e -
groups:
- admin
endpoints:
env:
enabled: true
sensitive: false

The password in sha256sum we are generating as below :

[kafka@hostkafkahq ~]$ echo "password" | sha256sum
6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e -

But every time we are trying to login we get below error in CLI :

[kafka@hostkafkahq ~]$ jdk-13.0.2/bin/java -Djava.security.krb5.conf=/etc/krb5.conf -Dmicronaut.config.files=/home/kafka/app.yml -jar /home/kafka/kafkahq.jar
2020-02-03 11:24:34,093 INFO main i.m.runtime.Micronaut Startup completed in 1868ms. Server Running: http://localhost:7070
2020-02-03 11:24:39,948 INFO pGroup-1-2 org.kafkahq.log.access [Date: 2020-02-03T11:24:39.561898+05:30] [Duration: 384 ms] [Url: GET /login/failed HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 679] [Port: 7070]
2020-02-03 11:24:40,088 INFO pGroup-1-6 org.kafkahq.log.access [Date: 2020-02-03T11:24:40.027575+05:30] [Duration: 60 ms] [Url: GET /static/img/logo.svg HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 3921] [Port: 7070]
2020-02-03 11:24:40,118 INFO pGroup-1-3 org.kafkahq.log.access [Date: 2020-02-03T11:24:40.020809+05:30] [Duration: 97 ms] [Url: GET /static/css/main.css HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 3188] [Port: 7070]
2020-02-03 11:24:40,134 INFO pGroup-1-2 org.kafkahq.log.access [Date: 2020-02-03T11:24:39.996698+05:30] [Duration: 137 ms] [Url: GET /static/css/vendor.css HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 40183] [Port: 7070]
2020-02-03 11:24:40,150 INFO pGroup-1-5 org.kafkahq.log.access [Date: 2020-02-03T11:24:40.020928+05:30] [Duration: 129 ms] [Url: GET /static/js/main.js HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 5688] [Port: 7070]
2020-02-03 11:24:40,308 INFO pGroup-1-4 org.kafkahq.log.access [Date: 2020-02-03T11:24:40.015137+05:30] [Duration: 293 ms] [Url: GET /static/js/vendor.js HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 560062] [Port: 7070]
2020-02-03 11:24:40,642 INFO pGroup-1-4 org.kafkahq.log.access [Date: 2020-02-03T11:24:40.621822+05:30] [Duration: 20 ms] [Url: GET /static/font/fontawesome-webfont.woff2 HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 77160] [Port: 7070]
2020-02-03 11:24:40,860 INFO pGroup-1-4 org.kafkahq.log.access [Date: 2020-02-03T11:24:40.843265+05:30] [Duration: 17 ms] [Url: GET /static/img/icon.png HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 30966] [Port: 7070]
2020-02-03 11:24:46,210 INFO pGroup-1-4 org.kafkahq.log.access [Date: 2020-02-03T11:24:46.058295+05:30] [Duration: 152 ms] [Url: POST /login HTTP/1.1] [Status: 303] [Ip: IP ADDRESS] [Length: 0] [Port: 7070]
2020-02-03 11:24:46,222 INFO pGroup-1-5 org.kafkahq.log.access [Date: 2020-02-03T11:24:46.217264+05:30] [Duration: 5 ms] [Url: GET /login/failed HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 679] [Port: 7070]

[tchiotludo]

Very hard to validate the configuration files with indentation 🤯

One thing I see is the password generation that is wrong : echo -n "password" | sha256sum. You forgot the -n, without this, the sha will contain a \n making impossible to fill on the web.

[vthorwat]

We have generated below password in hash value :

[kafka@hostkafkahq ~]$ echo -n "password" | sha256sum
5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8 -

In above generated has value I would like to ask regrading the "-" which is at the end of the generated hash value.

We have to mention like this right?

basic-auth:
  user: admin
  password: 6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e -
  groups:
    - admin

[tchiotludo]

No you need to remove it 👍

[vthorwat]

Hi @tchiotludo ,

As per the last update we have changed our configurations as below :

basic-auth:
  user: admin
  password: 6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e
  groups:
    - admin

But we are facing same issue.

Can you validate the app.yml content which we are using is correct or not.?

[tchiotludo]

Must be :

basic-auth:
  admin: # the username is here, and the password is indent 
    password: 6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e
    groups:
      - admin

[vthorwat]

Hello @tchiotludo ,

We were able to authenticate with above configuration changes.

We are able to see all the topics through admin user. We also configured the "readonly" user for only viewing the topics & their consumer groups, offset & lag.

So right now we are able to see all consumer groups, offsets & lag but not able to see the TOPICS with readonly user.

Please let us know if there any roles which we are missing for readonly user to view the topic list

Here is the current updated app.yml which we are using :

micronaut:
  security:
      enabled: true
  server:
    port: 7070
kafkahq:
 connections:
    jbdl-dev:
      properties:
        bootstrap.servers: "BROKER:6667,BROKER:6667,BROKER:6667"
        security.protocol: SASL_PLAINTEXT
        authorizer.class.name: kafka.security.auth.SimpleAclAuthorizer
        sasl.kerberos.service.name: kafka
        sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytabrincipal="kafka/HOSTNAME@REALM.COM";

 security:
   basic-auth:
     admin:
       password: 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8
       groups:
         - admin

     readonly:
       password: 8171bacf32668a8f44b90087ad107ed63170f57154763ba7e44047bf9e5a7be3
       groups:
         - topic-reader
   default-groups:
        -
   groups:
     admin:
      roles:
        - topic/read
        - topic/insert
        - topic/delete
        - topic/config/update
        - node/read
        - node/config/update
        - topic/data/read
        - topic/data/insert
        - topic/data/delete
        - group/read
        - group/delete
        - group/offsets/update
        - registry/read
        - registry/insert
        - registry/update
        - registry/delete
        - registry/version/delete
        - acls/read
        - connect/read
        - connect/insert
        - connect/update
        - connect/delete
        - connect/state/update
      attributes:
        topics-filter-regexp: "test.*"

     topic-reader:
       roles:
        - topic/read
        - topic/config/update
        - node/config/update
        - node/read
        - topic/data/read
        - group/read
        - group/offsets/update
        - registry/read
        - acls/read
        - connect/read
        - connect/state/update
        - registry/update
       attributes:
         topics-filter-regexp: "test\\.reader.*"
endpoints:
    env:
    enabled: true
    sensitive: false

[vthorwat]

We were able to figure out the TOPIC listing after removing the attributes property in app.yml.

But while accessing the topic from admin user we getting below error :

org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [Topic authorization failed.]

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [Topic authorization failed.]
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)

In our environment we are using Apache Ranger for allowing access to user's for Kafka access.

In app.yml we are using super user (kafka) keytab for accessing the Kafka service which means it has to get access to view the contents of the topic.

Please correct me if I am wrong.

But in this case we are getting above error while listing the topics.

[tchiotludo]

Seems not to be the case, your KafkaHQ user seems don't have read acl on these topic.

The better will be to try with Kafka commands like this kafka-topics.sh or kafkacat to see if your users is really able to fetch the topics with the same user.

But I don't think since the Kafka Client from KafkaHQ is returning that no right is provide.

To be honest, don't know anything about Apache Ranger, so couldn't help really sorry !

[vthorwat]

@tchiotludo ==>" your KafkaHQ user seems don't have read acl on these topic "

We are using Kafka user which is superuser in our environment which means it has to access all the topics as of now it is able to list topic, consumer groups ,lags & live tail.

The only issue is it is not able to read the data which is stored in the topic.

While doing that it is giving issue like below :

Seems this below application.yml is currently working with Basic Auth . If anyone want to refer for their environment to work with SASL_PLAINTEXT & Kerberos enabled.

Appplication.yml

micronaut:
  security:
      enabled: true
  server:
    port: 7070
kafkahq:
 connections:
    dev:
      properties:
        bootstrap.servers: "BROKER:6667,BROKER:6667,BROKER:6667"
        security.protocol: SASL_PLAINTEXT
        authorizer.class.name: kafka.security.auth.SimpleAclAuthorizer
        sasl.kerberos.service.name: kafka
        sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true useKeyTab=true keyTab="/etc/keytabs/kafka.service.keytab principal="kafka/HOSTNAME@REALM.COM";

 security:
   basic-auth:
     admin:
       password: 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8
       groups:
         - admin

     readonly:
       password: 8171bacf32668a8f44b90087ad107ed63170f57154763ba7e44047bf9e5a7be3
       groups:
         - topic-reader
   default-groups:
        -
   groups:
     admin:
      roles:
        - topic/read
        - topic/insert
        - topic/delete
        - topic/config/update
        - node/read
        - node/config/update
        - topic/data/read
        - topic/data/insert
        - topic/data/delete
        - group/read
        - group/delete
        - group/offsets/update
        - registry/read
        - registry/insert
        - registry/update
        - registry/delete
        - registry/version/delete
        - acls/read
        - connect/read
        - connect/insert
        - connect/update
        - connect/delete
        - connect/state/update
     topic-reader:
       roles:
        - topic/read
        - node/read
        - topic/data/read
        - group/read
        - registry/read
        - acls/read
        - connect/read
endpoints:
    env:
    enabled: true
    sensitive: false

[tchiotludo]

Glad to know that mostly all is working. For the last bug, I look at this issue but as I remember, KafkaHQ must capture silently the missing acl on Configuration. Stay tune

[tchiotludo]

I just try to capture this exception.

You can try in 5 minutes on the dev docker image.

[vthorwat]

Actually we have configured KafkaHQ as Standalone not on docker image so let me know is there kafkahq.jar we need to use or latest one.

As of now we are using latest jar which you release yesterday.

I think the ACL's are not working due to below warning which we are getting in logs:

2020-02-04 12:36:33,926 WARN 1-thread-4 .a.k.c.a.AdminClientConfig The configuration 'authorizer.class.name' was supplied but isn't a known config.
2020-02-04 12:36:34,328 WARN 1-thread-4 o.a.k.c.c.ConsumerConfig The configuration 'authorizer.class.name' was supplied but isn't a known config.
2020-02-04 12:36:35,432 WARN 1-thread-4 o.a.k.c.c.ConsumerConfig The configuration 'authorizer.class.name' was supplied but isn't a known config.
2020-02-04 12:36:36,032 INFO Group-1-11 org.kafkahq.log.access [Date: 2020-02-04T12:36:33.790856+05:30] [Duration: 2241 ms] [Url: GET /bdl-dev/topic HTTP/1.1] [Status: 200] [Ip: IP ADDRESS] [Length: 4190] [Port: 7070]

[tchiotludo]

The instructions to have a dev jar are available at the bottom of the Readme.

For the warning, it's a known bug on kafka. I can't do anything with it have no impact

[vthorwat]

Hi @tchiotludo , I just looked into our Kafka Cluster configurations for authorizer.class.name property

This above property we are using for Kafka authorizer.

Is this might be affecting for reading data from topic?

[tchiotludo]

I really don't know anything about this, really sorry. As I understand, the authorizer.class.name don't have any impact on acls. I just think that your kafka cluster don't allow to read / write configuration.

The topic I've done must keep KafkaHQ working even if there is no authorization. Please try and tell me if it's working

[tchiotludo]

Closing due to no activity, feel free to reopen with more details