create Mesos/Marathon specific installation instructions

[olafmol]

currently the mesos/marathon instructions refer to the DCOS instructions, with DCOS specific settings. We need to create mesos/marathon specific installation instructions.

[benniekrijger]

We currently have a Mesos cluster with Marathon running on Ubuntu 16.04 and were planning to use Vamp. Could you already provide us with some quick guidelines?

Some questions we currently have are:

• Should we use the Debian installers (listed here or the docker images?
• On which node should we install which Vamp component?
• What are the required installed components? Should Mesos-DNS be installed? HAproxy?

[olafmol]

Hi @benniekrijger tnx for your interest and questions. We will shortly provide some updated instructions on our website. For now:

• you can use the DC/OS docker image and deploy this on Marathon (http://vamp.io/documentation/installation/dcos/

• it doesn't matter on which node Vamp is being deployed, as long as it can access the Zookeeper and ElasticSearch services

• Vamp installs HAproxy through it's "VGA" workflow, which is a container with a Go application (VGA) and HAproxy. By default it will run on every node (constraint unique) to create a full cluster-mesh. Vamp needs a key-value store, if you run mesos/marathon you can connect to Zookeeper. Vamp also needs a metric store, you can use an ElasticSearch cluster or use our custom ES container which also includes logstash and is configured already for Vamp metric indexing: (project:https://github.com/magneticio/elastic).

Installing MesosDNS makes a lot of sense, as you can use it's service-discovery mechanism to set the Vamp API address in your config, ie vamp.marathon.mesos:

Hope this helps! Otherwise drop in our Gitter channel and we can have a realtime chat.

[benniekrijger]

Hi @olafmol, when i tried your proposed solution (using the DC/OS installer), only the Vamp package was installed and it did not continue to install any other components, like gateways etc.

The current Mesos/Marathon installation involves: Marathon/Mesos/MesosDNS on a cluster with 3 masters, 2 slaves and 1 slave with role: "slave_public". I'am i missing some required components?

[olafmol]

Can you share the versions you're using, and the configuration you have? (/api/v1/info and /api/v1/config) and also check in the Vamp logs (f.e. using mesos) what messages pop up? Make sure Vamp can connect to marathon, zookeeper and Elasticsearch. Some preliminary/draft debugging info: https://github.com/magneticio/vamp.io/blob/master/content/documentation/troubleshoot.md

[olafmol]

Hi @benniekrijger the troubleshooting instructions have been cleaned and published now: http://vamp.io/documentation/troubleshoot/ Could you share the info for us to help reproducing your issues? tnx!

[benniekrijger]

@olafmol i'am using Mesos 1.1.0 with Marathon 1.3.8. The Vamp docker image i'am trying to install is: magneticio/vamp:0.9.2-dcos from the instructions: http://vamp.io/documentation/installation/dcos/#step-2-deploy-vamp

The docker log states: "waiting for http://elasticsearch.marathon.mesos:9200/.kibana" However the ES .kibana index is already created and reachable

[iamtew]

@benniekrijger it's waiting for a HTTP 200 response, could you try hitting it from inside the container? For example:

docker exec <Vamp container ID> wget -q -O - http://elasticsearch.marathon.mesos:9200/.kibana

This is the current check: https://github.com/magneticio/vamp-docker-images/blob/master/vamp-custom/vamp.sh#L5-L14

# Wait for dependency before starting Vamp.
while true; do
  sleep 5
  status=$(curl -s -w %{http_code} ${VAMP_WAIT_FOR} -o /dev/null)
  if [ ${status} -eq 200 ]; then
    break
  else
    echo "waiting for ${VAMP_WAIT_FOR}"
  fi
done

[benniekrijger]

@iamtew thanks, my bad! Turned out the connection was blocked by a firewall rule.

The other problem was that Vamp tries to connected to Zookeeper on the hostname: zk-1.zk, which is probably configured by DCOS by default, but not when using a Mesos/Marathon setup.

Thanks for the quick support!

[iamtew]

@benniekrijger yes that setting is default in 0.9.2, you can overwrite it by setting the environment variable VAMP_PERSISTANCE_KEY_VALUE_STORE_ZOOKEEPER_SERVERS to your ZooKeeper servers in the definition for your Vamp container.

For reference this is the default configuration for Vamp 0.9.2 in DC/OS: https://github.com/magneticio/vamp-docker-images/blob/0.9.2/vamp-dcos/application.conf

We're working improving this for future release so you can more easily modify these settings, but it's not quite there yet.

Let me know if you need more assistance with setting up and/or configuring Vamp!

[benniekrijger]

@iamtew thanks, solved it. However when deploying, the gateway component doesn't get started and does not become visible in Marathon. All other components start just fine. I cannot see any logging related to the gateway agent start problem. Do the slave nodes require special Mesos resource roles or what else could i be missing?

[iamtew]

@benniekrijger there's two ways of having the vamp gateway agent (VGA) deployed, either by workflow, or manually.

Do you have the VGA workflows in Vamp?

[benniekrijger]

@iamtew Yep, 1 instance of the VGA workflow got deployed. Deploying the gateway agent manually according the public and private node install docs works fine.

[iamtew]

@benniekrijger this is the 0.9.2 VGA workflow: https://github.com/magneticio/vamp-artifacts/blob/0.9.2/breeds/vga.js#L21-L51

It looks pretty much the same as the manual installation example, except some config is in parameters.

Could you check the logs for the VGA workflow? You can see these in either Mesos web interface, or by running a docker logs <container ID> on the appropriate container on your host.

[benniekrijger]

@iamtew sure, the log looks fine to me, see:

Starting task vamp_workflow-vga.3d80df9f-e7bd-11e6-8cd1-0cc47a205a64
13:57:45.472 main NOTI ==> 
██╗   ██╗ █████╗ ███╗   ███╗██████╗
██║   ██║██╔══██╗████╗ ████║██╔══██╗
██║   ██║███████║██╔████╔██║██████╔╝
╚██╗ ██╔╝██╔══██║██║╚██╔╝██║██╔═══╝
 ╚████╔╝ ██║  ██║██║ ╚═╝ ██║██║
  ╚═══╝  ╚═╝  ╚═╝╚═╝     ╚═╝╚═╝
                       workflow agent
                       version 0.9.2
                       by magnetic.io
                                       
13:57:45.472 main NOTI ==> Starting Vamp Workflow Agent 
13:57:45.472 main INFO ==> Key-value store type          : zookeeper 
13:57:45.472 main INFO ==> Key-value store connection    : zk-1.zk:2181 
13:57:45.472 main INFO ==> Key-value store root key path : /vamp/workflows/vga 
13:57:45.472 main INFO ==> Workflow file path            : /usr/local/vamp 
13:57:45.472 main INFO ==> Workflow execution period     : 60 
13:57:45.472 main INFO ==> Workflow execution timeout    : 10 
13:57:45.472 main INFO ==> Reading workflow from         : /vamp/workflows/vga 
13:57:45.472 main INFO ==> Initializing ZooKeeper connection: zk-1.zk:2181 
13:57:45.473 main INFO ==> ZooKeeper getting '/vamp/workflows/vga' key value. 
13:57:45.481 main INFO ==> Writing workflow script       : /usr/local/vamp/workflow.js 
13:57:45.482 main NOTI ==> Executing workflow by Node.js. 
13:57:45.698 main INFO ==> WORKFLOW API options: {"host":"http://10.20.0.100:8080","path":"/api/v1","headers":{"Accept":"application/json","Content-Type":"application/json"},"cache":true} 
13:57:45.702 main INFO ==> WORKFLOW API GET /config 
13:57:45.731 main INFO ==> WORKFLOW HTTP REQUEST [0] {"protocol":"http:","port":"8080","hostname":"10.20.0.100","method":"GET","headers":{"Accept":"application/json","Content-Type":"application/json"},"path":"/api/v1/config?page=1"} 
13:57:55.486 main NOTI ==> Workflow process is killed as timeout reached. 
PI GET /config 

[benniekrijger]

@iamtew when deploying the Vamp Gateway Agents manual, the agents do not appear in the Vamp UI.

Here the log of an agent:

Starting task vamp_vamp-gateway-agent.f20ca522-e7c4-11e6-8cd1-0cc47a205a64
14:52:55.993 main NOTI ==> 
██╗   ██╗ █████╗ ███╗   ███╗██████╗
██║   ██║██╔══██╗████╗ ████║██╔══██╗
██║   ██║███████║██╔████╔██║██████╔╝
╚██╗ ██╔╝██╔══██║██║╚██╔╝██║██╔═══╝
 ╚████╔╝ ██║  ██║██║ ╚═╝ ██║██║
  ╚═══╝  ╚═╝  ╚═╝╚═╝     ╚═╝╚═╝
                       gateway agent
                       version 0.9.2
                       by magnetic.io
                                       
14:52:55.993 main NOTI ==> Starting Vamp Gateway Agent 
14:52:55.994 main INFO ==> Connecting to haproxy log socket: /usr/local/vamp/haproxy.log.sock 
14:52:55.994 main INFO ==> Opened Unix socket at: /usr/local/vamp/haproxy.log.sock. Creating Logstash sender. 
14:52:55.994 main NOTI ==> Reloading HAProxy - configuration file: /usr/local/vamp/haproxy.cfg, reload script: /usr/local/vamp/reload.sh 
14:52:55.998 main NOTI ==> Connected to remote UDP address: x.x.x.x:10001 
14:52:57.115 main NOTI ==> Initializing ZooKeeper connection: zk-1.zk:2181 
14:52:57.116 main INFO ==> ZooKeeper connection state: StateDisconnected 
14:52:58.117 main NOTI ==> HAProxy configuration has been changed. 

And my Marathon overview:

Vamp log:

14:49:42.644 | INFO  | io.vamp.http_api.HttpApiBootstrap$       |                                          | Binding: 0.0.0.0:8080
[ERROR] [01/31/2017 14:49:48.886] [vamp-akka.actor.default-dispatcher-5] [akka://vamp/user/artifact-initialization-actor] breeds/health.js (No such file or directory)
java.io.FileNotFoundException: breeds/health.js (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at scala.io.Source$.fromFile(Source.scala:91)
    at scala.io.Source$.fromFile(Source.scala:76)
    at scala.io.Source$.fromFile(Source.scala:54)
    at io.vamp.lifter.artifact.ArtifactLoader$$anonfun$loadFiles$1$$anonfun$apply$1.apply(ArtifactInitializationActor.scala:72)
    at io.vamp.lifter.artifact.ArtifactLoader$$anonfun$loadFiles$1$$anonfun$apply$1.apply(ArtifactInitializationActor.scala:72)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at io.vamp.lifter.artifact.ArtifactLoader$$anonfun$loadFiles$1.apply(ArtifactInitializationActor.scala:72)
    at io.vamp.lifter.artifact.ArtifactLoader$$anonfun$loadFiles$1.apply(ArtifactInitializationActor.scala:72)
    at io.vamp.lifter.artifact.ArtifactInitializationActor$$anonfun$io$vamp$lifter$artifact$ArtifactInitializationActor$$fileLoad$1.apply(ArtifactInitializationActor.scala:59)
    at io.vamp.lifter.artifact.ArtifactInitializationActor$$anonfun$io$vamp$lifter$artifact$ArtifactInitializationActor$$fileLoad$1.apply(ArtifactInitializationActor.scala:59)
    at scala.Function1$$anonfun$andThen$1.apply(Function1.scala:52)
    at io.vamp.lifter.artifact.ArtifactInitializationActor$$anonfun$receive$1.applyOrElse(ArtifactInitializationActor.scala:46)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at io.vamp.lifter.artifact.ArtifactInitializationActor.aroundReceive(ArtifactInitializationActor.scala:29)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)