search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.5k stars 3.7k forks source link

Create a travis build that runs Druid on a K8S Cluster and runs the integration tests #10542

Open himanshug opened 4 years ago

himanshug commented 4 years ago

Currently we have many different travis builds that setup a single node druid cluster and run integration tests.

This issue is to have one more that sets up the Druid cluster inside a K8S cluster (maybe using https://github.com/druid-io/druid-operator ) and runs some of the integration tests.

Once above setup exists, as a follow up, it would be modified to run Druid without Zookeeper for continuous integration testing of ability to run Druid without Zookeeper(see #9053 ) inside K8S Cluster.

AdheipSingh commented 3 years ago

@himanshug can help here, i have been testing on kind on travis, plus we can use the same way https://github.com/druid-io/druid-operator/pull/57 , i guess once we get this finalized. let me know how are you planning to run this, can you describe the flow.

himanshug commented 3 years ago

how are you planning to run this

it should be added as a new job in https://github.com/apache/druid/blob/master/.travis.yml , so travis would run it automatically on each PR

can you describe the flow

something along the lines of...

AFTER above is done, as a follow up we will change above Druid cluster configuration to not use Zookeeper and remove the Zookeeper StatefulSet part.

zhangyue19921010 commented 3 years ago

something along the lines of...

  • create a k8s cluster using minikube/kind/whatever-can-work-inside-travis
  • deploy druid-operator in above k8s cluster
  • deploy zookeeper using StatefulSet resource (replica = 1) so as to have predictable name for the pod (we wouldn't need postgres, since we can run overload at coordinator itself and have just one replica, so it can use derby)
  • deploy a druid cluster via druid-operator , druid cluster configuration should be same as that used by other existing builds running integration tests
  • run the integration tests
  • destroy everything

Hi @himanshug Our Dev, Stg and Prd Druid clusters are all deployed on K8s using Helm Chart. Maybe I can do some help for this great work. What we have done is that K8s + tiller(optional) + helmChart(including ZK statefulset) but do not use Druid Operator. If it is convenient, can you roughly say what you have completed and what needs to be done?

himanshug commented 3 years ago

I wouldn't say I have something ready :) , but I have done following many many times in remote k8s clusters...

so, I could get those done somewhat easily if needed.

I am guessing, run the integration tests step would be trivial considering we are doing that in many other builds.

So, most important step missing is setting up a k8s cluster in travis build env that has enough resources to run a Druid cluster which can handle the things we do in integration tests.

zhangyue19921010 commented 3 years ago

Understood! Although I am not very familiar with integration tests but maybe I can try to add a new job in travis in the next few days, deployed a druid cluster on minikube K8s and run integration tests. Maybe use Helm Chart? What do you think?

himanshug commented 3 years ago

Maybe use Helm Chart? What do you think?

In the end state, I would prefer the operator.

zhangyue19921010 commented 3 years ago

Hi @himanshug , I just make a PR https://github.com/apache/druid/pull/10669. This PR add a new IT Job 71:

  1. Deploy a K8s cluster based on minikube.
  2. Deploy Druid cluster on K8s: single Druid, use local disk as Deep Storage and use derby as MetaStorage
  3. Run single IT test ITNestedQueryPushDownTest which includes data ingestion, Historical loading and query action.

Now job 71 is successful which means this PR is not WIP anymore!

PTAL :)

himanshug commented 3 years ago

thanks, yeah I will review that.