Rancher Installation Issue

[lenovouser]

I did install rancher with:

docker run -d --restart=always -p 8000:8080 --name=rancher-server \
-e "CATTLE_API_ALLOW_CLIENT_OVERRIDE=true" \
-e "CATTLE_HOST_API_PROXY_SCHEME=wss" rancher/server

and then proxied it from another NGINX server with:

upstream rancher {
    server rancher-server:8000;
}

server {
    listen 443 ssl;
    server_name rancher.domain.tld;
    ssl_certificate /data/ssl/domain.tld.crt;
    ssl_certificate_key /data/ssl/domain.tld.key;

    proxy_buffering off;
    proxy_buffer_size 512;

    location / {
        proxy_set_header X-API-request-url $scheme://<host>$request_uri;
        proxy_pass http://rancher;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

server {
    listen 80;
    server_name rancher.domain.tld;
    return 301 https://$server_name$request_uri;
}

I am getting this error in the chrome console when I now access https://rancher.domain.tld:

OPTIONS https://%3Chost%3E/v1/projects?all=true&limit=1000 net::ERR_NAME_NOT_RESOLVED

I already accessed Rancher per http://myipaddress:8000 and changed the Host Registration value to my domain.

Cheers,

ApfelUser

[lenovouser]

I also just see that the logs are spamming:

time="2015-07-18T11:38:33Z" level=info msg="Starting go-machine-service..." gitcommit=102d311 
time="2015-07-18T11:38:33Z" level=error msg="Unable to create EventRouter" Err="Get https://<host>/v1/schemas: dial tcp: lookup <host>: no such host" 
time="2015-07-18T11:38:33Z" level=info msg="Exiting go-machine-service..." 

[lenovouser]

I just re-installed and got 2 different issues now. I am getting this error in the rancher-server console when I select the standard "rancher.mydomain.tld" as Host Registration:

time="2015-07-18T14:50:58Z" level=error msg="Unable to start EventRouter" Err="websocket: bad handshake" 
time="2015-07-18T14:50:58Z" level=info msg="Exiting go-machine-service..." 
time="2015-07-18T14:50:59Z" level=info msg="Setting log level" logLevel=info 
time="2015-07-18T14:50:59Z" level=info msg="Starting go-machine-service..." gitcommit=102d311 
time="2015-07-18T14:50:59Z" level=info msg="Initializing event router" workerCount=10 
time="2015-07-18T14:51:00Z" level=error msg="Failed to subscribe to events." error="websocket: bad handshake" subscribeUrl="ws://rancher.mydomain.tld/v1/subscribe?eventNames=physicalhost.create%3Bhandler%3DgoMachineService&eventNames=physicalhost.bootstrap%3Bhandler%3DgoMachineService&eventNames=physicalhost.remove%3Bhandler%3DgoMachineService&eventNames=ping%3Bhandler%3DgoMachineService"

This happens about 2 times every second.

This error doesn't come up when I don't select the standard rancher.mydomain.tld and instead manually put https://rancher.mydomain.tld into the Host Registration. Instead this one is showing in the rancher-agent console:

time="2015-07-18T14:59:43Z" level="info" msg="Starting event router." 
time="2015-07-18T14:59:43Z" level="info" msg="Watching state directory: /var/lib/rancher/state/containers" 
time="2015-07-18T14:59:43Z" level="info" msg="Processing event: &docker.APIEvents{Status:\"start\", ID:\"2a6baa8257e30382bb21022e0904cb1dbb4dabe512b6ba0d464a344351c3e92e\", From:\"-simulated-\", Time:0}" 
time="2015-07-18T14:59:43Z" level="info" msg="Connecting to proxy." url="wss://https://rancher.mydomain.tld/v1/connectbackend?token=eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE0MzcyMzE4ODQsInN1YiI6ImNhdHRsZSIsImlzcyI6Imh0dHA6XC9cL2NhdHRsZS5pbyIsInJlcG9ydGVkVXVpZCI6IjI1MzA0ZGNmLWMzOWEtNDcwNS1iMDU5LTUxMTIyYjJhZTJhZSIsImlhdCI6MTQzNzIzMTU4NCwia2lkIjoiZGVmYXVsdCJ9.cnr734y-o5youdnFr9xYd2L7vLjxp6T8WuA9YlEsY-oU9dQzyC0BYsCY2k9ln5LX--hSf79nxH4qPPBhkuHekQidH2TBuIwYBFW8Izm396HfMlGm3qE2tT-sYyJ2ifIsqSN6KhlrO8vtrRpqGTbI3XwtvEiXgu9DxjbiB0QKtkkOfmNtZCW2FhQ7vhIg83hckjAYrH4AkflkE0rTgDrZC_mYu7iIljdQZMLmoAZPoyc4hPtQe7Z1HVQBcIfZdxNOgLu3E-eJPeOsYliXG4UJcGo1Wjp_P_mprkdNVt2f4dMTLAXeADonruc_RAfWURBuk4wE--kHGrJ9_nTu5wwKnw" 
time="2015-07-18T14:59:43Z" level="fatal" msg="Failed to connect to proxy." error="dial tcp 46.101.xxx.xxx:0: connection refused"

That is why I am kind of in a dilemma because no matter what I do, there will occur an error on one of the both sides rancher-server and rancher-agent

[ebuildy]

Hmm, what is the value of <host> ?

[ebuildy]

Try to comment the line

proxy_set_header X-API-request-url $scheme://<host>$request_uri;

[lenovouser]

@ebuildy I fixed the first error by using proxy_set_header X-API-request-url $scheme://rancher.domain.tld$request_uri;

Now I have the two errors mentioned in my last comment.

Thanks for helping though!

You can find a more complete report here, I did that one after the last comment on this issue.

[ebuildy]

Maybe you need to play with HTTP header Nginx sends to Rancher:

proxy_connect_timeout   59s;
proxy_send_timeout      600;
proxy_read_timeout      600;
proxy_buffer_size       64k;
proxy_buffers           16 32k;
proxy_pass_header       Set-Cookie;
proxy_redirect          off;
proxy_hide_header       Vary;

proxy_busy_buffers_size         64k;
proxy_temp_file_write_size      64k;

proxy_set_header        Accept-Encoding         '';
proxy_ignore_headers    Cache-Control           Expires;
proxy_set_header        Referer                 $http_referer;
proxy_set_header        Host                    $host;
proxy_set_header        Cookie                  $http_cookie;
proxy_set_header        X-Forwarded-Proto       $scheme;
proxy_set_header        X-Real-IP               $remote_addr;
proxy_set_header        X-Forwarded-For         $proxy_add_x_forwarded_for;
proxy_set_header        X-Forwarded-Host        $host;

I tried also to run rancher behind a Nginx proxy, but w/o success ;-( So please, keep me posted if you do!

[flaccid]

See https://github.com/rancher/rancher/issues/1151. I'm very interested in a 100% working nginx .conf for this and then we can have it documented.

[ebuildy]

Finally I moved to haproxy, it works like a charm without anything special:

backend service_docker  
    balance roundrobin
    option httpclose
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    http-request add-header X-Forwarded-Host docker.admin.*****.com

    server docker-1 172.17.0.56:8080 maxconn 500 id 1 
    acl    server-1 srv_id 1 
    rspadd X-Served-By:\ docker-1 if server-1 

[lenovouser]

What do the rancher-server and rancher-agent logs say? Are there no errors?

[cjellick]

We'll have a fix in today. However, in the mean time, @ApfelUser have you tried setting the api.host setting to a space as indicated at the bottom of these instructions: http://docs.rancher.com/rancher/installing-rancher/installing-server/basic-ssl-config/

[cjellick]

@ApfelUser just a quick update to properly set expectations:

1. We decided on a more comprehensive fix than the one in the PR that I referenced above. This fix will have an impact on the configuration. You will not be able to set the protocol in host registration setting.
2. Just FYI: I realized I may have been misleading say that the fix would be in "today." Getting it onto master today means it wont be in the official release of the rancher/server image until next week.

Finally, a question for you @ApfelUser: In the forum issue, you mentioned that you said:

This error doesn't come up when I don't select the standard rancher.mydomain.tld and instead manually put https://rancher.mydomain.tld into the Host Registration.

Where did you find the instructions/hint for adding https:// to that setting? I didn't think it was officially documented anywhere and I want to make sure we update that reference after we make these changes?

[lenovouser]

@cjellick I did not get it from anywhere. I just saw the ws error and thought it might work when I force HTTPS as host. So no need to worry. With the api.host setting, do I just set up Rancher and don't change the Host Registration value, or do I need to change that do something else too?

[cjellick]

@ApfelUser A couple notes so that we're on the same page:

1. Because of this bug, there is currently no out-of-the-box SSL configuration that will work.
2. When things are working, all you'll need to do is set the Host Registration value in the API.

[lenovouser]

@cjellick Okay. So I just wait or do you want me to try the api.host thing?

[cjellick]

@ApfelUser unfortunately, the best course of action is to wait at this point.

[lenovouser]

Okay, no problem. Just wanted to ask :)

[cjellick]

Thanks for you patience and help in reporting the issue @ApfelUser.

[flaccid]

Any chance we could also have the haproxy solution in https://github.com/rancher/rancher/issues/1580#issuecomment-122876135 documented and supported? It would be good to have configs that are also not https e.g. TLS is terminated in front of the reverse proxy by an ELB.

[lenovouser]

@flaccid I am not sure if his configuration really works, as he has not responded to my question whether there are errors in either the rancher-server or rancher-agent console or not. Because e.g the interface etc. does indeed work even if there are ws / wss communication errors.

[cjellick]

@flaccid we are working on ELB support now. #1309 represents the primary work that needs done in order to support it. Once we iron out the remaining details for HAProxy and ELB, we'll revisit our documentation to make sure it covers the major use cases.

[flaccid]

Understood guys and thank you. I'm excited!

[lenovouser]

@cjellick It seems like this fixed with v0.30.0 now. Can I close this, or am I wrong?

[cjellick]

You're correct @ApfelUser. We'll have one of our QA close it though, if you don't mind. I had a whole bucket of ssl/proxy related bugs that I was holding on to until I wrapped up one other related feature. That hit master yesterday, so I'll be assigning this and the other SSL issues over to QA today.

cc @deniseschannon

[lenovouser]

Okay, won't close this then. I can confirm that this issue #1580 got fixed with release v0.30.0 by the way. Just installed fresh today.

[sangeethah]

Tested with rancher/server:v0.31.0-rc1

Able to bring up a rancher-server with SSL behind Nginx as https reverse proxy. Following are the steps followed:

docker run -d --restart=always --name=rancher-server -e "CATTLE_API_ALLOW_CLIENT_OVERRIDE=true" rancher/server:v0.31.0-rc1
mkdir /root/nginx

docker run -d -p 80:80 -p 443:443 -v /root/nginx/ssl:/etc/nginx/ssl/:ro -v /root/nginx/rancher.io.conf:/etc/nginx/conf.d/rancher.io.conf:ro --link='rancher-server' nginx

nginx confifuration used:

upstream rancher {
    server rancher-server:8080;
}

server {
    listen 443 ssl;
    server_name <servername>;
    ssl_certificate /etc/nginx/ssl/rancher.io.bundled.crt;
    ssl_certificate_key /etc/nginx/ssl/rancher.io.key;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://rancher;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

server {
    listen 80;
    server_name <servername>;
    return 301 https://$server_name$request_uri;
}

Able to add custom hosts , Digital Ocean hosts to rancher server successfully. Able to view host stats. Able to view container logs and also exec shell. Basic sanity checks to make sure that the connectivity between containers from different hosts work.