search

chrislusf / glow

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
3.2k stars 246 forks source link

Example to run on Google compute engine #26

Open chrislusf opened 8 years ago

chrislusf commented 8 years ago

Glow should be very suitable to run on the cloud. It'll be nice to see that we can run a simple driver program and invoke the google compute engine.

Some benefits I can see:

  1. the ingress data transportation is free.
  2. GCE's cost is per second. Glow is very fast to start and stop.

Suggestions:

  1. Starting a globally accessible glow master may make it easier.
  2. Glow just need local SSD.
justicezyx commented 8 years ago

Found this blog post about running glow on Kubernetes (not integrating): https://medium.com/@cedriclam/how-to-use-glow-with-kubernetes-22820d8f6270#.gstrirbcf

It might be straightforward to replicate the steps on GKE, which is a managed Kubernetes service.

chrislusf commented 8 years ago

I am not really familiar with Kurbernetes. But I want to have some script or even Go code to do this:

  1. start a glow master and multiple glow agents on kurbernetes. (May need to provide some Kurbernetes credentials here. May need to return a cluster id.)
  2. start a glow flow on the master and agents. (Maybe this would also need to credential file and the cluster id)
  3. wait for the glow flow to finish
  4. shutdown the master and agents.

The details should be encapsulated inside the script or go code. I prefer that a common user do not even need to know the details.