Two customers have requested that we add encryption support into the Kafka connector. They need encryption to ensure data cannot be intercepted in transit (even within the Ably network), and only intended recipients can decrypt the data.
Whilst adding encryption support is trivial in the Kafka Connector, when considering the use case and expected implementation, it very quickly becomes non-trivial. Both customers have requested that encryption keys are unique based on the target, in one instance this is by user, and in the other it's by channel. This immediately presents some significant challenges for the Kafka Connector because it has a static config, and no means to obtain encryption keys dynamically based on the target. In addition, it's likely there will be other requirements like recycling of keys, revocation of keys, etc. which too adds significant complexity.
At present, we do not think the Ably Kafka connector should support encryption because of these challenges. Customers that want to use our encryption features will likely need dynamic configuration of keys based on the target, key revocation, key recycling, and more, and this complexity should not sit within the Connector, but instead with the customer's code.
Our recommendation at present is that for customers who have encryption needs, they should either build their own Kafka consumer, implement their own custom encryption logic, and publish to Ably, or they should fork this connector and add their own custom logic. We'd of course be very open to any pull requests from customers who feel there is a way to offer a generic encryption capability in the connector that would be useful to others.
Two customers have requested that we add encryption support into the Kafka connector. They need encryption to ensure data cannot be intercepted in transit (even within the Ably network), and only intended recipients can decrypt the data.
The requests are CSP-99 and CSP-100.
Whilst adding encryption support is trivial in the Kafka Connector, when considering the use case and expected implementation, it very quickly becomes non-trivial. Both customers have requested that encryption keys are unique based on the target, in one instance this is by user, and in the other it's by channel. This immediately presents some significant challenges for the Kafka Connector because it has a static config, and no means to obtain encryption keys dynamically based on the target. In addition, it's likely there will be other requirements like recycling of keys, revocation of keys, etc. which too adds significant complexity.
At present, we do not think the Ably Kafka connector should support encryption because of these challenges. Customers that want to use our encryption features will likely need dynamic configuration of keys based on the target, key revocation, key recycling, and more, and this complexity should not sit within the Connector, but instead with the customer's code.
Our recommendation at present is that for customers who have encryption needs, they should either build their own Kafka consumer, implement their own custom encryption logic, and publish to Ably, or they should fork this connector and add their own custom logic. We'd of course be very open to any pull requests from customers who feel there is a way to offer a generic encryption capability in the connector that would be useful to others.