search

qubole / kinesis-sql

Kinesis Connector for Structured Streaming
http://www.qubole.com
Apache License 2.0
137 stars 80 forks source link

Kinesis encrypted stream #109

Closed BreakpointsCA closed 2 years ago

BreakpointsCA commented 2 years ago

I have issues reading an encrypted data stream in AWS. I have no errors in my logs but my spark job is not fetching any records from the stream. I am using this version of the library: spark-sql-kinesis_2.12-1.2.0_spark-3.0.jar

My stream is configured to use Server-side encryption. Based on the documenation: "Kinesis Data Streams uses AWS Key Management Service (KMS) to encrypt your data."

When I deactivate this feature, my spark job is able to pull records from the stream without any problems.

Do I need to configure something or it should be transparent?

We have Lambda that reads from that same stream without problems.

BreakpointsCA commented 2 years ago

The issue is related to #107