search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
230 stars 75 forks source link

Add full end to end #224

Open edwardcapriolo opened 3 years ago

edwardcapriolo commented 3 years ago

So this is a way to move the "examples" to actually working end to end. It is a real PITA to try to get schema registry classpath in this classpath and this embedded tool does it nice. But maybe testcontainers would be easier.

cerveada commented 3 years ago

The tests are failing because the license is missing in FullSpec.scala

gintautassulskus-elsevier commented 2 years ago

There is a library for embedded Kafka-SchemaRegistry setup that does not require you to fiddle with installation, Java params and jars directly - https://github.com/embeddedkafka/embedded-kafka-schema-registry. Not sure though if you can use it to deploy Schema Registry without Kafka.

My advice would be to use testcontainers. Unlike an embedded service, the containerised counterpart is OS-agnostic. In the vast majority of cases such as proposed in this PR, an embedded service will not cause an issue, but it's that 1% that gives the most headaches.

For example, (embedded) Kafka is not supported on Windows leading to issues under certain circumstances (https://github.com/apache/kafka/pull/6329 and KAFKA-1194).

There is a caveat if your CI runs on Kubernetes. Running testcontainers in a container (pod) necessitates a Docker-in-Docker setup that introduces a significant security risk and is a no go.