Support for Scala 11

[asicoe]

Hi,

We are looking at using maze to implement integration testing in our Scala projects using different open source distributed engines/storage systems/messaging systems like Flink, Spark, Kafka, Redis, MQTT, Crate, Cassandra etc.

At a first glance the library Api looks promising, especially for the advance abnormal network conditions support.

One major blocker for us starting experimenting with it at the moment is the fact that the prebuilt packages are only compiled against Scala 2.12. Most Engines we use like (Flink, Spark) have libraries up to Scala 11.

Is there a reason why only Scala 12 is supported? Are there plans to offer support for Scala 11? In our opinion doing this would opening the use of this library to many more teams.

Please let us know, Alex

[larochef]

Hi Alex,

The aim of the library is to orchestrate docker containers, meaning that you can have an application in a container with scala 2.11, and have the tests in another module, running scala 2.12.

For instance, when I did some tests with kafka, I had:

• Containers for zookeeper
• Containers for Kafka
• Containers for consumers
• Containers for producers
• A module with the tests, communicating with the rest

The module with the tests only had unit tests inside and some utility methods to communicate with kafka/ zookeepers, consumers and producers.

Since maze doesn't know how to build the containers, you can use any technology inside.

One of the reason of 2.12 is that we integrate with some java library (docker-java) and 2.12 makes it easier to use. Since I didn't really see blocking points, I didn't investigate much on cross compiling with 2.11.

If scala 2.11 is still a blocking issue, I can have a look at how to cross build it.

I'll be glad to help you if you need help to set it up.

François

[asicoe]

Hi François,

Thanks for the detailed reply.

The usage pattern I had in mind is this: we have one scala 2.11 sbt project for our Flink streaming app that needs to talk to a bunch of services Redis, CrateDB, MQTT etc. I wanted to have a integration tests within this project that spin up all the containers it needs for Redis, CrateDB, MQTT (using maze) and then runs the actual Flink app from within the test talking to all of those. The idea was that this would make writing the tests more easy and user friendly for the developer. Also, all tests are within the same repo and are executed within the build lifecycle which makes it easy to reason about, change, deploy. Self contained repo.

What you are suggesting if I understand correctly is have a separate repo just for the integration tests which has to spin up all the containers for services as before as well as an extra one for running a Flink cluster. Then the tests themselves would have to clone the main project repo, build it to generate the uber jar and then deploy it to Flink cluster. Is that correct?

Does testing with our approach make sense to you?

After writing all this I still see the value in having the flexibility to run Maze from within our Scala 11 project which is still a blocker at the moment. It would be great if you would consider that.

Mercy, Alex

[larochef]

Hi Alex,

Instead of recloning everything, I was rather thinking (and it's what we did in some other projects) that the main application module produces the docker image, and the maze tests are included in the same project as another module, and use the image built by the previous module.

We haven't really tried it the way you plan to do it, but I don't see anything wrong with it.

I'll work on it and will keep in touch in this thread once it is done.

[larochef]

I just had a look at it, and there weren't many places impacted. #25 should be integrated soon.

[asicoe]

Awesome! Thanks for the explanations, for cross compiling and very fast replies François!

[larochef]

I just released on central version 1.0.16, cross compiled with scala 2.11 and 2.12.

I'm looking forward to reading how you're going to use it!

[asicoe]

Awesome. Thanks for the effort and definitely I'll keep you posted ;)