search

ing-bank / scruid

Scala + Druid: Scruid. A library that allows you to compose queries in Scala, and parse the result back into typesafe classes.
Apache License 2.0
115 stars 29 forks source link

Adds Javascript aggregator and post-aggregator #77

Closed anskarl closed 4 years ago

anskarl commented 4 years ago

GH-72

anskarl commented 4 years ago

To pass the new test for Javascript, the latest docker image for Druid server from https://github.com/Fokko/docker-druid should be pushed to Docker hub (that is the one with Javascript enabled).

[info] DQL also work with 'javascript' aggregations
[info] - should successfully be interpreted by Druid *** FAILED ***
[info]   The future returned an exception of type: ing.wbaa.druid.client.HttpStatusException, with message: Received response with HTTP status code 500 Internal Server Error. (DQLSpec.scala:383)

The test passes locally, since I have manually build the latest version of https://github.com/Fokko/docker-druid

anskarl commented 4 years ago

I've also added Javascript post-aggregator in DQL and updated the corresponding documentation and unit tests. Similar to Javascript aggregator, the addition of Javascript post-aggregator also requires a Druid docker image with javascript enabled.

Fokko commented 4 years ago

It looks like the build is failing:

Building in Docker Cloud's infrastructure...
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
Reset branch 'master'
Your branch is up-to-date with 'origin/master'.
KernelVersion: 4.4.0-1060-aws
Components: [{u'Version': u'18.03.1-ee-3', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.4.0-1060-aws', u'Os': u'linux', u'BuildTime': u'2018-08-30T18:42:30.000000000+00:00', u'ApiVersion': u'1.37', u'MinAPIVersion': u'1.12', u'GitCommit': u'b9a5c95', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.10.2'}}]
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2
Starting build of index.docker.io/fokkodriesprong/docker-druid:latest...
Step 1/15 : FROM openjdk:8
---> 08ded5f856cc
Step 2/15 : ENV DRUID_VERSION 0.15.1-incubating
---> Running in 03afbd730885
Removing intermediate container 03afbd730885
---> 56aedfba0301
Step 3/15 : ENV ZOOKEEPER_VERSION 3.4.14
---> Running in e5eac8b6c7bd
Removing intermediate container e5eac8b6c7bd
---> 023cf500177e
Step 4/15 : RUN mkdir -p /tmp && cd /tmp/ && curl -fsLS "https://www.apache.org/dyn/closer.cgi?filename=/incubator/druid/$DRUID_VERSION/apache-druid-$DRUID_VERSION-bin.tar.gz&action=download" | tar xvz && mv apache-druid-$DRUID_VERSION /opt/druid
---> Running in 35d002051666
curl: (22) The requested URL returned error: 403 Forbidden
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Removing intermediate container 35d002051666
The command '/bin/sh -c mkdir -p /tmp && cd /tmp/ && curl -fsLS "https://www.apache.org/dyn/closer.cgi?filename=/incubator/druid/$DRUID_VERSION/apache-druid-$DRUID_VERSION-bin.tar.gz&action=download" | tar xvz && mv apache-druid-$DRUID_VERSION /opt/druid' returned a non-zero code: 2

Locally everything build well, so I've restarted the build.

anskarl commented 4 years ago

It looks like the build is failing:

Building in Docker Cloud's infrastructure...
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
Reset branch 'master'
Your branch is up-to-date with 'origin/master'.
KernelVersion: 4.4.0-1060-aws
Components: [{u'Version': u'18.03.1-ee-3', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.4.0-1060-aws', u'Os': u'linux', u'BuildTime': u'2018-08-30T18:42:30.000000000+00:00', u'ApiVersion': u'1.37', u'MinAPIVersion': u'1.12', u'GitCommit': u'b9a5c95', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.10.2'}}]
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2
Starting build of index.docker.io/fokkodriesprong/docker-druid:latest...
Step 1/15 : FROM openjdk:8
---> 08ded5f856cc
Step 2/15 : ENV DRUID_VERSION 0.15.1-incubating
---> Running in 03afbd730885
Removing intermediate container 03afbd730885
---> 56aedfba0301
Step 3/15 : ENV ZOOKEEPER_VERSION 3.4.14
---> Running in e5eac8b6c7bd
Removing intermediate container e5eac8b6c7bd
---> 023cf500177e
Step 4/15 : RUN mkdir -p /tmp && cd /tmp/ && curl -fsLS "https://www.apache.org/dyn/closer.cgi?filename=/incubator/druid/$DRUID_VERSION/apache-druid-$DRUID_VERSION-bin.tar.gz&action=download" | tar xvz && mv apache-druid-$DRUID_VERSION /opt/druid
---> Running in 35d002051666
curl: (22) The requested URL returned error: 403 Forbidden
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Removing intermediate container 35d002051666
The command '/bin/sh -c mkdir -p /tmp && cd /tmp/ && curl -fsLS "https://www.apache.org/dyn/closer.cgi?filename=/incubator/druid/$DRUID_VERSION/apache-druid-$DRUID_VERSION-bin.tar.gz&action=download" | tar xvz && mv apache-druid-$DRUID_VERSION /opt/druid' returned a non-zero code: 2

Locally everything build well, so I've restarted the build.

@Fokko the download URLs are not valid anymore. You may better use the archive.apache.org repos.

anskarl commented 4 years ago

@Fokko I've prepared a PR for docker-druid image with archive.apache.org repos

Fokko commented 4 years ago

@anskarl the Druid build passed, can you rebase?

anskarl commented 4 years ago

@Fokko I've rebased with the latest master branch and the tests are passing

Fokko commented 4 years ago

Love the implementation, LGTM