search

delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
343 stars 73 forks source link

Remove direct dynamodb lock interactions #182

Open rtyler opened 2 weeks ago

rtyler commented 2 weeks ago

With the adoption of deltalake >= 0.17.0 the deltalake-aws library uses S3DynamoDbLogStore. We still carry a likely unnecessary dynamodb_lock dependency which is potentially problematic or inconsistent when used in conjunction with the S3DynamoDbLogStore