feat: add KafkaBatchPublisher - Githubissues

DataChefHQ / sparkle

✨ A meta framework for Apache Spark, helping data engineers to focus on solving business problems with highest quality!

Apache License 2.0

3 stars 0 forks source link

feat: add KafkaBatchPublisher #19

Closed wtfzambo closed 1 month ago

wtfzambo commented 1 month ago

In this PR

KafkaBatchPublisher class as requested
Related unit tests
Some refactoring of adjacent unit tests
Dev package chispa for easy handling of pyspark DataFrames equality assertions (see screenshot for example)
Clean up of Pytest logs so now they're readable

chispa output example