ing-bank / scruid

Scala + Druid: Scruid. A library that allows you to compose queries in Scala, and parse the result back into typesafe classes.
Apache License 2.0
115 stars 29 forks source link

Move to Apache Druid 0.15.1 Docker image for CI #69

Closed Fokko closed 4 years ago

Fokko commented 4 years ago

I'd like to move to another CI image for Scruid. My major objection is that the image is pushed by hand. I'd prefer it being built by a Dockerhub autobuild, so we actually know from what source the image is built.

Also took the moment to update to Druid 0.15.0. This includes building the Docker image from the Druid binaries and greatly simplifying the Dockerfile: https://github.com/Fokko/docker-druid/commit/45165822c3c353abebed441b9eb3c8c338305ca3 I've opened a PR to the Druid docker image as well: https://github.com/druid-io/docker-druid/pull/72

One thing changed with the Druid update. The predefined count metric has been removed, and therefore I had to updates some tests. Instead of posting the JSON ingestion task, I now use the CLI that ships with Druid: https://github.com/Fokko/docker-druid/blob/master/Dockerfile#L17-L18

Going to Druid 0.15.0 also gives us a nice UI to test queries:

image

image

anskarl commented 4 years ago

@Fokko you may also change the base image of Druid in order to use a smaller distribution. For example Alpine Linux with JRE like https://github.com/anskarl/druid-docker-cluster/blob/master/src/Dockerfile.druid

Fokko commented 4 years ago

Thanks, @anskarl for the pointer. I've had some issues with the alpine one since it is really bare. For example, not even bash. I missed a lot of tools to run the import script. Moving to the jre already cuts the size in half. Going to the alpine would even make the images three times as small:

MacBook-Pro-van-Fokko:docker-druid fokkodriesprong$ docker image inspect openjdk:8 --format='{{.Size}}'
488072696
MacBook-Pro-van-Fokko:docker-druid fokkodriesprong$ docker image inspect openjdk:8-jre --format='{{.Size}}'
246324763
MacBook-Pro-van-Fokko:docker-druid fokkodriesprong$ docker image inspect openjdk:8-jre-alpine --format='{{.Size}}'
84920975

Moving to openjdk:8-jre for now. Feel free to open a PR.

Fokko commented 4 years ago

Thanks @bjgbeelen, and sorry for the late response. However, I think you have more important stuff on your mind right now :-)

The README is actually correct. This one uses CountAggregation(name = "count") which is good. If it would be like LongSumAggregation(name = "count", fieldName = "count") then it will not work anymore.