AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
231 stars 76 forks source link

Abris not working when used with confluent schema registry over basic auth (found on dataproc) #364

Open DwivediPrasoon opened 2 months ago

DwivediPrasoon commented 2 months ago

This is an issue with confluent-schema-registry-client though but getting extended to Abris. The issue is, when schema registry is being used with basic auth over a confluent instance, it is showing 401 while fetching schema from confluent (incase of dataproc cluster over a private VPN). One can not just pass credentials like following:

AbrisConfig.fromConfluentAvro.downloadReaderSchemaByLatestVersion
          .andTopicNameStrategy(getTopic)
          .usingSchemaRegistry(
            Map(
              AbrisConfig.SCHEMA_REGISTRY_URL -> Configuration.schemaRegistryUrl,
              SchemaRegistryClientConfig.BASIC_AUTH_CREDENTIALS_SOURCE -> "USER_INFO",
              SchemaRegistryClientConfig.USER_INFO_CONFIG -> s"${Configuration.schemaRegistryUsername}:${Configuration.schemaRegistryPassword}"
            )
          )

instead confluent can give you schema if used with a rest service created and passed through. e.g.

val restService = new RestService(urls)
    val provider = new UserInfoCredentialProvider()
    provider.configure(configs.asJava)
    restService.setBasicAuthCredentialProvider(provider)
    new CachedSchemaRegistryClient(restService, maxSchemaObject)

The current implementation of this is not allowing to use restService based constructor with CachedSchemaRegistryClient, so I have tried changing Abris code and made a separate jar with above change. Raising a PR to add this feature.

DwivediPrasoon commented 2 months ago

https://github.com/AbsaOSS/ABRiS/pull/365 for above issue