AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
230 stars 75 forks source link

Proposal to support a custom registry client #210

Closed froesef closed 3 years ago

froesef commented 3 years ago

This PR proposes to support a custom registry client that can be specified in the config map that is configured with usingSchemaRegistry(config: Map[String, String]) .

This enables custom registry clients that implement different protocols or authentications. Configuration of these clients can be done by the same Map[String, String].

An example custom registry (with no actual implementations of the functions) can be found in MyRegistry.

cerveada commented 3 years ago

Thanks for the PR, could you explain a bit why is this needed? For confluent Registry their client should be enough, or not? Or is there some other registry that you need to access?

froesef commented 3 years ago

We have an internal registry that is not fully confluent compatible. Also we're using different protocols than the standard CachedSchemaRegistryClient.

froesef commented 3 years ago

Thanks for considering this PR! I did some general changes as well in this PR:

Is it fine if I leave these changes also in this PR or do you wish a separate one for these?

cerveada commented 3 years ago

Is it fine if I leave these changes also in this PR or do you wish a separate one for these?

Thanks, these are small changes let's keep them here.

What is reason for settingspark.driver.bindAddress? Does it improve performance?

froesef commented 3 years ago

If you look at the earlier commits it failed to bind the spark driver to a port when starting the spark context. I'm not exactly sure why this happened now as nothing in my PR remotely touches areas like this so it might be a change in GH actions. Anyways the error message said to set a binding address so I did. This is not related to performance. (the UI part is saving like 10 seconds in the Spark context startup time)

kevinwallimann commented 3 years ago

Thanks, @froesef for your contribution!