401 Unauthorized error while accessing Schema Registry secured with basic auth

[Vathsan]

I referred to the following doc https://docs.confluent.io/platform/current/schema-registry/security/index.html#configuring-the-rest-api-for-basic-http-authentication and set up basic auth for schema registry (confluent-5.4.0).

The configurations are as below,

1. Added the following in schema-registry.properties

   authentication.method=BASIC
   authentication.roles=admin,developer,user,sr-user
   authentication.realm=SchemaRegistry-Props

2. jaas_config.file

   SchemaRegistry-Props {
   org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
   file="/Users/srivathsan/Downloads/confluent-5.4.0/etc/schema-registry/password.properties"
   debug="false";
   };

3. password.properties file fret: letmein,developer,admin

But when I try to access the schema registry I get 401 unauthorized. I have passed the following properties from Kafka producer,

producerConfigProperties.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
producerConfigProperties.put(KafkaAvroSerializerConfig.USER_INFO_CONFIG, "fret:lermein");

io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unauthorized; error code: 401
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:266)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:322)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:422)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:414)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:400)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:140)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:196)
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:172)
    at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:70)
    at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53)

Curl commands tried:

curl -X GET -i --user "fret:letmein" -H "Content-Type: application/vnd.schemaregistry.v1+json"  http://localhost:8081/subjects
curl -X GET -i -H "Authorization: Basic ZnJldDpsZXRtZWlu" -H "Content-Type: application/vnd.schemaregistry.v1+json"  http://localhost:8081/subjects

Without security my Kafka producer and curl command works fine. I get the authorization error upon enabling basic auth for schema registry. Any help on this would be highly appreciated.

[ziggythehamster]

The properties aren't being forwarded to the schema registry client, so it assumes it's unauthenticated and does not try to pass in credentials. Even if you were to embed the credentials in the URL (which would make the most sense), that doesn't work. You'd have to look for a constructor that took a preconfigured client and then supply it with a client that is configured with the right properties.

See: https://github.com/confluentinc/schema-registry/blob/bca2842e332c5ec0ed388b1526fa3d028b369ae9/schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDe.java#L65

This is basically the same problem that led me to file #1740, just a different piece of software doesn't forward the properties. They really should fix #1740 by making Basic auth in URLs get parsed properly (which it would, if only you could set KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE to "URL", but you can't in this case without a whole bunch of extra steps)

[ungaralex]

Is there any update on this? We have the same problem with the registry replying 401 for basic authenticated requests (e.g. with curl or Postman). Setup is dockerized and we followed the official documentation.

The only difference is that due to compatibility reasons we have to use an older version of the registry (5.5.5). But this shouldn't matter in this case, should it?

[felix786]

Hi

I am also facing the same issue 401 unauthorized when accessing schema registry with basic auth.

Any solution or possible alternatives?

[ziggythehamster]

There is no version of Schema Registry which will do anything with auth information passed with https://foo:bar@baz:1234 by default, but if you have access to the schema registry client, there are several ways you can either have it accept it passed in the URL (the normal way you do this in other systems, which should be the default, see #1740) or set it in another property.

If you don't have access to the schema registry client, you need to implement whatever you need to implement to give yourself access to the schema registry client. This results in you creating your own class hierarchy that more or less exactly matches the hierarchy that already exists, but with the capability of passing the options passed into the schema registry client. Consider making a PR to the project using the schema registry client to do this.

Another way would be to fork the Schema Registry client. This is what I did for 4.1.4, but the same idea works in newer versions because it works the same. Older versions don't work the same way and need a different fix.

[felix786]

@ziggythehamster Thanks for the information. It is useful..

[DevarajKR]

I am using below jars, still getting the auth error, providing everything kafka-schema-registry-7.1.1.jar kafka-schema-registry-client-7.1.1.jar

basic.auth.credentials.source=USER_INFO basic.auth.user.info=abc:xyz

errors.SerializationException: Error retrieving Avro schema for id 100005\nCaused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unauthorized; error code: 401

Please someone help me with solution

[gassayy]

@DevarajKR and @Vathsan or anyone is relevant,

Not sure if you guys are still facing this problem, I have a project which is a spark application build upon Databricks Spark and Confluent Kafka with schema registry. The schema registry is using basic authentication.

We packaged the application as an uber jar via sbt-assembly.

Within the application, we have some logic that requires to manually retrieve AVRO schema from schema registry.

The problem we are facing is that every request sent to the schema registry is return with status code: 401 Unauthorized. However, if we upload an extra kafak-schema-registry-client.jar to the cluster that running the application, then everything passed.

After dissection of our uber.jar as well as the source code of schema registry, it seems the problem is classloading, I can't figure out an optimal solution at this moment. Instead, I created an instance of RestService with UserInfoCredentialProvider, then pass them to CachedSchemaRegistry.

Any suggestion or idea are welcome.

Thanks.

The assembly strategy and temp fix as following (any sensitive info have been replaced or generalized)

// assembly code in build.sbt
assemblyMergeStrategy := {
  case PathList(ps @ _*) if isReadme(ps.last) || isLicenseFile(ps.last) =>
    MergeStrategy.rename
  case PathList("META-INF", xs @ _*)         => MergeStrategy.rename
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*)    => MergeStrategy.last
  case x                                     => MergeStrategy.first
},

// tmp fix
val props = Map(
  SchemaRegistryClientConfig.BASIC_AUTH_CREDENTIALS_SOURCE -> "USER_INFO",
  SchemaRegistryClientConfig.USER_INFO_CONFIG -> s"${config.username}:${config.password}").asJava
val restService = new RestService(config.address)
val provider = new UserInfoCredentialProvider()
provider.configure(props)
restService.setBasicAuthCredentialProvider(provider)

val cachedClient = new CachedSchemaRegistryClient(restService, 10)
val schema = cachedClient.getLatestSchemaMetadata("xxxx-value")