503 - Service Unavailable

[ersinpw]

I followed the following docs page very precisely: https://www.balena.io/open/docs/getting-started/ This is my third try, however I stuck every time at the step where the openBalena server is tested. The services that runs on port 80 and 443 returns:

   503 Service Unavailable
   No server is available to handle this request.

Therefore the ping with curl fails also and returns the same. I am running everything on a brand new Ubuntu 18.04 x64 server.

Edit:

I see lots of people with the same issue. I did a deeper investigation and it seems that the "openbalena_haproxy" container is not working properly. When I open a bash to the container:

docker exec -it e7eae59bbe09 /bin/sh

and fetch the page at 0.0.0.0

wget 0.0.0.0

it returns the same 503 Service unavailable page

[luisgs7]

I also have the same problem, did you manage to solve it, @ersinpw how did you solve it?

Thanks.

[matiasAS]

I have the same problem, I am testing everything on localhost, that is, within a virtual machine, the haproxy.conf file I have changed the name of the service to the ip of the vm 192.168.10.14, leaving it like this:

global tune.ssl.default-dh-param 1024

defaults timeout connect 5s timeout client 50s timeout server 50s

frontend http-in mode http option forwardfor bind *:80 reqadd X-Forwarded-Proto:\ http

acl is_cert_validation path -i -m beg "/.well-known/acme-challenge/" use_backend cert-provider if is_cert_validation

acl host_api hdr_dom(host) -i "api.${HAPROXY_HOSTNAME}" use_backend backend_api if host_api

acl host_registry hdr_dom(host) -i "registry.${HAPROXY_HOSTNAME}" use_backend backend_registry if host_registry

acl host_vpn hdr_dom(host) -i "vpn.${HAPROXY_HOSTNAME}" use_backend backend_vpn if host_vpn

acl host_s3 hdr_dom(host) -i "s3.${HAPROXY_HOSTNAME}" use_backend backend_s3 if host_s3

frontend ssl-in mode tcp bind *:443 tcp-request inspect-delay 2s tcp-request content accept if { req.ssl_hello_type 1 }

acl is_ssl req.ssl_ver 2:3.4

acl host_tunnel req_ssl_sni -i "tunnel.${HAPROXY_HOSTNAME}" use_backend redirect-to-tunnel-in if host_tunnel

use_backend redirect-to-https-in if is_ssl use_backend vpn-devices if !is_ssl

backend redirect-to-https-in mode tcp balance roundrobin server localhost 127.0.0.1:444 send-proxy-v2

backend redirect-to-tunnel-in mode tcp balance roundrobin server localhost 127.0.0.1:3129

frontend https-in mode http option forwardfor bind 127.0.0.1:444 ssl crt /etc/ssl/private/open-balena.pem accept-proxy reqadd X-Forwarded-Proto:\ https

acl host_api hdr_dom(host) -i "api.${HAPROXY_HOSTNAME}" use_backend backend_api if host_api

acl host_registry hdr_dom(host) -i "registry.${HAPROXY_HOSTNAME}" use_backend backend_registry if host_registry

acl host_vpn hdr_dom(host) -i "vpn.${HAPROXY_HOSTNAME}" use_backend backend_vpn if host_vpn

acl host_s3 hdr_dom(host) -i "s3.${HAPROXY_HOSTNAME}" use_backend backend_s3 if host_s3

backend backend_api mode http option forwardfor balance roundrobin server balena_api_1 192.168.10.14:80 check port 80

backend backend_registry mode http option forwardfor balance roundrobin server balena_registry_1 192.168.10.14:80 check port 80

backend backend_vpn mode http option forwardfor balance roundrobin server balena_vpn_1 192.168.10.14:80 check port 80

backend backend_s3 mode http option forwardfor balance roundrobin server balena_s3_1 192.168.10.14:80 check port 80

backend cert-provider mode http option forwardfor balance roundrobin server balena_cert-provider_1 192.168.10.14:80 no-check

backend vpn-devices mode tcp server balena_vpn_1 192.168.10.14:443 send-proxy-v2 check-send-proxy port 443

frontend db mode tcp bind *:5432 default_backend backend_db timeout client 1h

backend backend_db mode tcp server balena_db_1 db:5432 check port 5432

frontend redis mode tcp bind *:6379 default_backend backend_redis timeout client 1h

backend backend_redis mode tcp server balena_redis_1 redis:6379 check port 6379

listen vpn-tunnel mode tcp bind *:3128 server balena_vpn 192.168.10.14:3128 check port 3128

listen vpn-tunnel-tls mode tcp bind *:3129 ssl crt /etc/ssl/private/open-balena.pem server balena_vpn 192.168.10.14:3128 check port 3128

being the solution to these errors:

Building certificate from environment variables... Setting up watches. Beware: since -r was given, this may take a while! Watches established. [ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:76] : 'server balena_api_1' : could not resolve address 'api'. [ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:82] : 'server balena_registry_1' : could not resolve address 'registry'. [ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:88] : 'server balena_vpn_1' : could not resolve address 'vpn'. [ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:94] : 'server balena_s3_1' : could not resolve address 's3'. [ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:104] : 'server balena_vpn_1' : could not resolve address 'vpn'. [ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:129] : 'server balena_vpn' : could not resolve address 'vpn'. [ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:134] : 'server balena_vpn' : could not resolve address 'vpn'. [ALERT] 315/000032 (16) : Failed to initialize server(s) addr.

and now the errors I have left to resolve are:

(input): docker logs -f openbalena_haproxy_1

(output): Building certificate from environment variables... Setting up watches. Beware: since -r was given, this may take a while! [NOTICE] 315/024440 (16) : New worker #1 (18) forked Watches established. [WARNING] 315/024442 (18) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [ALERT] 315/024442 (18) : proxy 'vpn-tunnel' has no server available! [WARNING] 315/024442 (18) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [ALERT] 315/024442 (18) : proxy 'vpn-tunnel-tls' has no server available!

The local domain configured is openbalena.local, if I enter www.openbalena.local configuring that address in /etc/hosts, the error is the following:

503 Service Unavailable No server is available to handle this request.

I would appreciate your help in solving this problem

Greetings Matias

@ersinpw @luisgs7 (speak spanish?, i from chile) @wrboyce @relaxdiego @dfunckt

[ersinpw]

@luisgs7 no I didn't managed to get it work.. @matiasAS no I don't speak spanish

Looks like everyone is having the same issue. @fisehara can you prioritise this issue?

[fisehara]

@ersinpw Thanks for the mentioning.

If you have a working setup which results into a haproxy service running in a container can you please share the logs from the failing HAproxy service? As all services are initialised from a central docker-compose.yml you should be able to run the command manually which should give you the logs or all running services.

Moreover, are we more than happy for contributions to maintain this project, as the current pinned HAproxy version itself is fairly outdated.

Thanks and best regards Harald

[fisehara]

@matiasAS

This message:

Building certificate from environment variables...
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:76] : 'server balena_api_1' : could not resolve address 'api'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:82] : 'server balena_registry_1' : could not resolve address 'registry'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:88] : 'server balena_vpn_1' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:94] : 'server balena_s3_1' : could not resolve address 's3'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:104] : 'server balena_vpn_1' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:129] : 'server balena_vpn' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : parsing [/usr/local/etc/haproxy/haproxy.cfg:134] : 'server balena_vpn' : could not resolve address 'vpn'.
[ALERT] 315/000032 (16) : Failed to initialize server(s) addr.

shows that the docker-compose environment isn't able to resolve the service names for the inter-service communication. As seen here: https://github.com/balena-io/open-balena/blob/ea07d43c5c7e98bb8b11c808b12428a46019c8d5/src/haproxy/haproxy.cfg#L76 The haproxy container only knows about name api and the underlying docker / docker-compose environment has to resolve the api to the actual other service running as container.

Please try to narrow down if the docker / docker-compose environment is able to communicate with each other and resolve services names. eg. with two services and nc the one which listens and the other that tries to connect.