search

delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
337 stars 71 forks source link

Enable Azure writes #136

Closed rtyler closed 1 year ago

rtyler commented 1 year ago

This branch is a work in progress with @mightyshazam and is not yet ready for merging into main

rtyler commented 1 year ago

Frustratingly I cannot reproduce these test failures locally on Ubuntu :angry:

mightyshazam commented 1 year ago

Frustratingly I cannot reproduce these test failures locally on Ubuntu 😠

I had these tests working before I created the new branch. I must have accidentally added the ignore attributes back. I believe the ultimate fix came down to increasing the duration of the calls to std::thread::sleep. It's harder to reproduce on Ubuntu because docker isn't running in a VM.

rtyler commented 1 year ago

@mightyshazam thanks for the suggestions, I've gone ahead and re-ignored the tests for now. I have also force pushed the branch after rebasing some commits together. If this goes :heavy_check_mark: then I will merge!