Getting "Deploy failed" when creating any service...

[roynasser]

Hi,

Trying to test out paz, but when I try adding any service, prior to adding an app, I just get "Deploy Failed".

I am trying the vagrant cluster. Looks like things installed well, added entries to hosts, and accessed the web panel without issue, but when I try to add the demo-api service from the docs, or the registry container (to make a private registry in the cluter), I get a "deploy failed" error immediately as I click the deploy button...

fleetctl list units looks like everything is up everywhere?

UNIT                    MACHINE             ACTIVE  SUB
$ fleetctl -strict-host-key-checking=false -endpoint=http://172.17.8.101:4001 list-units
paz-orchestrator-announce.service   6bf8fd0d.../172.17.8.101    active  running
paz-orchestrator.service        6bf8fd0d.../172.17.8.101    active  running
paz-scheduler-announce.service      f069df3f.../172.17.8.102    active  running
paz-scheduler.service           f069df3f.../172.17.8.102    active  running
paz-service-directory-announce.service  6bf8fd0d.../172.17.8.101    active  running
paz-service-directory.service       6bf8fd0d.../172.17.8.101    active  running
paz-web-announce.service        51e62345.../172.17.8.103    active  running
paz-web.service             51e62345.../172.17.8.103    active  running

PS: what is the "public facing" setting that is when we choose a service? (I tried both true and false with same results) It would probably make sense to add a line to http://paz.readme.io/v1.0/docs/deploying-your-first-application-using-paz to explain what "public facing" does.

[lukebond]

Hello @RVN-BR thanks for your interest in Paz! I'm keen to have more and more people trying it out :)

I'll answer your second question first: the "public facing" checkbox determines whether or not Paz will try and configure DNS for you. Since you're running locally it will do nothing one way or another but read on if you're interested... If you look at the code of the Orchestrator (https://github.com/paz-sh/paz-orchestrator/blob/master/lib/dns-handler.js#L86) you'll find the only place that variable is referenced. If you're running "on the internet" rather than locally on Vagrant, and are using DNSimple as a DNS provider to administer the domain you're using for your Paz cluster, then it will set up a subdomain for that service for you.

I've updated the documentation as per your suggestion. I think I forgot it because it was cut off the bottom of the screenshot :)

As for why your deployment failed, I'll need more information. Can you check the following logs and send me anything useful you find?

• Browser Javascript console output
• $ fleetctl journal paz-orchestrator
• $ fleetctl journal paz-scheduler

Thanks again!

[roynasser]

Here goes the console output (which I think may be a big hint looking at it...)

Loading service page: GET http://paz-orchestrator.paz/services/demo-api/config/last 404 (Not Found)send @ vendor-276a96b3d47554b726b9b9b4f2a90b11.js:10 Error: Not Found at fe.extend.ajaxError (vendor-276a96b3d47554b726b9b9b4f2a90b11.js:16) at a.error (vendor-276a96b3d47554b726b9b9b4f2a90b11.js:16) at c (vendor-276a96b3d47554b726b9b9b4f2a90b11.js:2) at Object.h.fireWith (vendor-276a96b3d47554b726b9b9b4f2a90b11.js:2) at r (vendor-276a96b3d47554b726b9b9b4f2a90b11.js:3) at XMLHttpRequest.t (vendor-276a96b3d47554b726b9b9b4f2a90b11.js:3)c.onerrorDefault @

(after click on deploy) vendor-276a96b3d47554b726b9b9b4f2a90b11.js:3 OPTIONS http://localhost:9002/hooks/deploy net::ERR_CONNECTION_REFUSED

$ fleetctl -strict-host-key-checking=false -endpoint=http://172.17.8.101:4001 journal paz-orchestrator
-- Logs begin at Wed 2015-05-13 20:56:02 UTC, end at Thu 2015-05-14 19:21:21 UTC. --
May 14 15:20:23 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":50,"err":{"message":"\"Key not found in database [latest!my-registry]\"","name":"NotFoundError","stack":"NotFoundError: \"Key not found in database [latest!my-registry]\"\n    at ClientRequest.onResponse (/usr/src/app/node_modules/restify/lib/clients/http_client.js:161:26)\n    at ClientRequest.g (events.js:180:16)\n    at ClientRequest.emit (events.js:95:17)\n    at HTTPParser.parserOnIncomingClient (http.js:1692:21)\n    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23)\n    at Socket.socketOnData (http.js:1587:20)\n    at TCP.onread (net.js:527:27)"},"msg":"\"Key not found in database [latest!my-registry]\"","time":"2015-05-14T15:20:23.043Z","src":{"file":"/usr/src/app/resources/service/controller.js","line":108},"v":0}
May 14 19:19:00 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"req":{"method":"GET","url":"/cluster/hosts"},"res":{"statusCode":200,"header":""},"uuid":"1x6y28","msg":"","time":"2015-05-14T19:19:00.231Z","src":{"file":"/usr/src/app/middleware/logger.js","line":4,"func":"module.exports"},"v":0}
May 14 19:19:00 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"hosts.list":"*","uuid":"1x6y28","msg":"","time":"2015-05-14T19:19:00.231Z","src":{"file":"/usr/src/app/resources/host/controller.js","line":40,"func":"controller.list"},"v":0}
May 14 19:19:03 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"req":{"method":"GET","url":"/services?noEmit=true"},"res":{"statusCode":200,"header":""},"uuid":"3zdjh9","msg":"","time":"2015-05-14T19:19:03.834Z","src":{"file":"/usr/src/app/middleware/logger.js","line":4,"func":"module.exports"},"v":0}
May 14 19:19:03 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"service.get":"*","uuid":"3zdjh9","msg":"","time":"2015-05-14T19:19:03.834Z","src":{"file":"/usr/src/app/resources/service/controller.js","line":27,"func":"controller.list"},"v":0}
May 14 19:19:05 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"req":{"method":"GET","url":"/services/demo-api/config/last"},"res":{"statusCode":200,"header":""},"uuid":"8o4rmd","msg":"","time":"2015-05-14T19:19:05.745Z","src":{"file":"/usr/src/app/middleware/logger.js","line":4,"func":"module.exports"},"v":0}
May 14 19:19:05 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"service.configLast":"demo-api","uuid":"8o4rmd","msg":"","time":"2015-05-14T19:19:05.745Z","src":{"file":"/usr/src/app/resources/service/controller.js","line":100,"func":"controller.configLast"},"v":0}
May 14 19:19:05 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"req":{"method":"GET","url":"/services/demo-api/config/next"},"res":{"statusCode":200,"header":""},"uuid":"5ewkis","msg":"","time":"2015-05-14T19:19:05.751Z","src":{"file":"/usr/src/app/middleware/logger.js","line":4,"func":"module.exports"},"v":0}
May 14 19:19:05 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":30,"service.configNext":"demo-api","uuid":"5ewkis","msg":"","time":"2015-05-14T19:19:05.752Z","src":{"file":"/usr/src/app/resources/service/controller.js","line":117,"func":"controller.configNext"},"v":0}
May 14 19:19:05 core-01 bash[22546]: {"name":"paz-orchestrator_log","hostname":"327559062d4f","pid":6,"level":50,"err":{"message":"\"Key not found in database [latest!demo-api]\"","name":"NotFoundError","stack":"NotFoundError: \"Key not found in database [latest!demo-api]\"\n    at ClientRequest.onResponse (/usr/src/app/node_modules/restify/lib/clients/http_client.js:161:26)\n    at ClientRequest.g (events.js:180:16)\n    at ClientRequest.emit (events.js:95:17)\n    at HTTPParser.parserOnIncomingClient (http.js:1692:21)\n    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23)\n    at Socket.socketOnData (http.js:1587:20)\n    at TCP.onread (net.js:527:27)"},"msg":"\"Key not found in database [latest!demo-api]\"","time":"2015-05-14T19:19:05.757Z","src":{"file":"/usr/src/app/resources/service/controller.js","line":108},"v":0}

$ fleetctl -strict-host-key-checking=false -endpoint=http://172.17.8.101:4001 journal paz-scheduler
-- Logs begin at Wed 2015-05-13 20:56:22 UTC, end at Thu 2015-05-14 19:22:01 UTC. --
May 13 22:16:16 core-02 systemd[1]: Started paz-scheduler: Takes apps from your paz service directory and runs them on a CoreOS cluster using fleet.
May 13 22:16:17 core-02 bash[1507]: {"port":9002,"loglevel":"info","dbname":"db","svcdir":"paz-service-directory.paz","ssh-host":"172.17.8.103","ssh-key":null,"ssh-port":null,"gen-key":true,"etcd-pubkey-key":"/paz/config/scheduler/_pubkey","etcd-endpoint":"172.17.8.103:4001","cors":true,"nodeploy":false}
May 13 22:16:17 core-02 bash[1507]: Starting server
May 13 22:16:17 core-02 bash[1507]: {"name":"paz-scheduler_log","hostname":"f3e13096e59d","pid":5,"level":30,"msg":"paz-scheduler now running on port 9002","time":"2015-05-13T22:16:17.607Z","src":{"file":"/usr/src/app/server.js","line":170},"v":0}
May 14 15:17:23 core-02 bash[1507]: {"name":"paz-scheduler_log","hostname":"f3e13096e59d","pid":5,"level":30,"req":{"method":"GET","url":"/config/my-registry/version/latest"},"res":{"statusCode":200,"header":""},"uuid":"aauuqo","msg":"","time":"2015-05-14T15:17:23.290Z","src":{"file":"/usr/src/app/middleware/logger.js","line":2,"func":"module.exports"},"v":0}
May 14 15:17:23 core-02 bash[1507]: {"name":"paz-scheduler_log","hostname":"f3e13096e59d","pid":5,"level":30,"config.getLatest":"my-registry","uuid":"aauuqo","msg":"","time":"2015-05-14T15:17:23.293Z","src":{"file":"/usr/src/app/resources/config/controller.js","line":6,"func":"controller.getLatest"},"v":0}
May 14 15:20:23 core-02 bash[1507]: {"name":"paz-scheduler_log","hostname":"f3e13096e59d","pid":5,"level":30,"req":{"method":"GET","url":"/config/my-registry/version/latest"},"res":{"statusCode":200,"header":""},"uuid":"2yimp","msg":"","time":"2015-05-14T15:20:23.039Z","src":{"file":"/usr/src/app/middleware/logger.js","line":2,"func":"module.exports"},"v":0}
May 14 15:20:23 core-02 bash[1507]: {"name":"paz-scheduler_log","hostname":"f3e13096e59d","pid":5,"level":30,"config.getLatest":"my-registry","uuid":"2yimp","msg":"","time":"2015-05-14T15:20:23.040Z","src":{"file":"/usr/src/app/resources/config/controller.js","line":6,"func":"controller.getLatest"},"v":0}
May 14 19:19:05 core-02 bash[1507]: {"name":"paz-scheduler_log","hostname":"f3e13096e59d","pid":5,"level":30,"req":{"method":"GET","url":"/config/demo-api/version/latest"},"res":{"statusCode":200,"header":""},"uuid":"as7lvi","msg":"","time":"2015-05-14T19:19:05.749Z","src":{"file":"/usr/src/app/middleware/logger.js","line":2,"func":"module.exports"},"v":0}
May 14 19:19:05 core-02 bash[1507]: {"name":"paz-scheduler_log","hostname":"f3e13096e59d","pid":5,"level":30,"config.getLatest":"demo-api","uuid":"as7lvi","msg":"","time":"2015-05-14T19:19:05.749Z","src":{"file":"/usr/src/app/resources/config/controller.js","line":6,"func":"controller.getLatest"},"v":0}

The "localhost" may be a likely culprit... as to why there is an error in a GET to http://paz-orchestrator.paz/services/demo-api/config/last I'm not sure... the return from that get is:

"Key not found in database [latest!demo-api]"

(I beieve this is reflected in the log)

[lukebond]

Thanks @RVN-BR!

The first 404 (and related first fleetctl journal output) looks like paz-web is trying to get the latest deployed config from a service that has not yet been deployed, and therefore has no latest config. This may not be a big deal but it's messy that you see it.

The second one is interesting. It looks like paz-web is not being configured with the correct URL for the scheduler. It should be http://paz-scheduler.paz...

@jacyzon got any thoughts on this? might be related to https://github.com/paz-sh/paz-web/pull/26

[roynasser]

I think it may be related (please see my edit on my previous comment, I did it just as you replied)...

the browser is attempting to call localhost:9002

[roynasser]

Is there anyway to override the localhost?

[lukebond]

yes i did see that before posting, that's why i was saying it isn't being configured with the correct URL.

i've just found the issue, fix incoming!

[roynasser]

Great! Let me know so I can re-clone the repo, and if possible which command I should run :)

Will it need to redownload the images? (my connection isnt the best and it takes >1hr to install the cluster ...) no biggy, but if there is a way to upgrade instead it would save me some time and let me play quicker :D

[lukebond]

It will get resolved when I push and fix this: https://github.com/paz-sh/paz-web/issues/31

We'll then need to wait for CI to rebuild the Docker image and push to quay.io. Once that's done you can do fleetctl ssh paz-web to SSH into the box that runs paz-web, and then run the following:

$ fleetctl stop paz-web{,-announce} && sleep 5 && fleetctl start paz-web{,-announce}

...and then watch fleetctl list-units until you see that it's running again.

stay tuned...

[roynasser]

On the edge of the chair! :)

[lukebond]

okay try now :)

bear in mind you may currently fall afoul of this: https://github.com/paz-sh/paz-scheduler/issues/12

[roynasser]

Still getting exactly the same error... trying to connect on localhost:9002...

$ fleetctl -strict-host-key-checking=false -endpoint=http://172.17.8.101:4001 ssh paz-web
Last login: Thu May 14 20:22:17 2015 from 172.17.8.1
CoreOS beta (668.3.0)
Update Strategy: No Reboots
core@core-03 ~ $ fleetctl stop paz-web{,-announce} && sleep 5 && fleetctl start paz-web{,-announce}
Unit paz-web-announce.service loaded on 51e62345.../172.17.8.103
Unit paz-web.service loaded on 51e62345.../172.17.8.103
Unit paz-web-announce.service launched on 51e62345.../172.17.8.103
Unit paz-web.service launched on 51e62345.../172.17.8.103

[lukebond]

Can you ensure you have the docker image on that host as the latest one on quay.io?

It might be faster if we jump on Gitter and chat about it: https://gitter.im/paz-sh/paz

[roynasser]

Sounds good. on gitter right now.

[lukebond]

@RVN-BR try now that az-sh/paz-web#31 is fixed (again)

[roynasser]

Waiting for it to spin up my vms... will respond asap (need to leave soon worst case ill get back to you by monday, sorry)

[lukebond]

We've been working with this all weekend and I'm certain it's fixed. Reopen if it still happens for you.

Thanks again!

[roynasser]

Apparently its working for me! :) Waiting for the container to download and start...but deploy step has gone through.