Proposal to support a custom registry client

[froesef]

This PR proposes to support a custom registry client that can be specified in the config map that is configured with usingSchemaRegistry(config: Map[String, String]) .

This enables custom registry clients that implement different protocols or authentications. Configuration of these clients can be done by the same Map[String, String].

An example custom registry (with no actual implementations of the functions) can be found in MyRegistry.

[cerveada]

Thanks for the PR, could you explain a bit why is this needed? For confluent Registry their client should be enough, or not? Or is there some other registry that you need to access?

[froesef]

We have an internal registry that is not fully confluent compatible. Also we're using different protocols than the standard CachedSchemaRegistryClient.

[froesef]

Thanks for considering this PR! I did some general changes as well in this PR:

• Changing the Confluent repository to https
• Adding some Spark configs to fix tests in the CI and speed them up (disable the UI)

Is it fine if I leave these changes also in this PR or do you wish a separate one for these?

[cerveada]

Is it fine if I leave these changes also in this PR or do you wish a separate one for these?

Thanks, these are small changes let's keep them here.

What is reason for settingspark.driver.bindAddress? Does it improve performance?

[froesef]

If you look at the earlier commits it failed to bind the spark driver to a port when starting the spark context. I'm not exactly sure why this happened now as nothing in my PR remotely touches areas like this so it might be a change in GH actions. Anyways the error message said to set a binding address so I did. This is not related to performance. (the UI part is saving like 10 seconds in the Spark context startup time)

[kevinwallimann]

Thanks, @froesef for your contribution!