search

deephaven / deephaven-core

Deephaven Community Core
Other
257 stars 80 forks source link

Extensible Kafka KeyOrValueSpecs #4346

Closed cpwright closed 1 year ago

cpwright commented 1 year ago

As a data administrator/developer, I want to be able to substitute my own key or value specifications instead of ones that Deephaven has built so that I can customize my ingestion for new formats or as a developer experiment with alternative processors for existing formats.

The current architecture has switch statements that are not conducive to extending the ingester framework at runtime.

devinrsmith commented 1 year ago

Agreed. It's rather cumbersome the way it is now. During recent refactorings, the switch statements were removed, and the implementations were made cleaner. That said, it's really an internal interface still, while it doesn't need to be.

As part of the work in https://github.com/deephaven/deephaven-core/pull/4375, we'll be a lot closer to achieving this reality.

cpwright commented 1 year ago

I think the change in #4248 might be close enough in terms of code. The protobuf spec was added without changing post-refactor. However, the io.deephaven.kafka.KafkaTools.Consume.KeyOrValueSpec is undocumented and the SchemaProviderProvider interface is private (and undocumented) which likely makes it hard to understand how one is meant to implement a KeyOrValueSpec that extends the SchemaProviderProvider.