Failing instructions for enabling SSL

[candlerb]

At https://zato.io/docs/admin/guide/tls/client-lb.html#accepting-ssl-tls-connections-client-certificates-optional it gives some text to append to the lb config to listen on SSL with incoming certificates.

I am trying this with zato 2.0.3

Adding the stanza and clicking "Validate and save" gives the following exception (reformatted here for clarity by converting \n to newline):

Caught an exception while invoking the load-balancer agent, e:[Traceback (most recent call last):
  File "/opt/zato/2.0.3/zato-web-admin/src/zato/admin/web/views/load_balancer.py", line 88, in _client_validate_save
    func(*args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "/usr/lib/python2.7/xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: '<type 'exceptions.Exception'>:Caught an exception, e:[Traceback (most recent call last):
  File "/opt/zato/2.0.3/zato-agent/src/zato/agent/load_balancer/server.py", line 174, in _validate
    self._popen(command, HAPROXY_VALIDATE_TIMEOUT, timeout_msg, rc_non_zero_msg, common_msg)
  File "/opt/zato/2.0.3/zato-agent/src/zato/agent/load_balancer/server.py", line 104, in _popen
    raise Exception(msg)
Exception: Failed to validate the config file using HAProxy. config_file:[# ##############################################################################

global
    log 127.0.0.1:514 local0 debug # ZATO global:log
    stats socket /opt/zato/cluster1/load-balancer/haproxy-stat.sock # ZATO global:stats_socket

# ##############################################################################

defaults
    log global
    option httpclose

    stats uri /zato-lb-stats # ZATO defaults:stats uri

    timeout connect 15000 # ZATO defaults:timeout connect
    timeout client 15000 # ZATO defaults:timeout client
    timeout server 15000 # ZATO defaults:timeout server

    errorfile 503 /opt/zato/cluster1/load-balancer/config/repo/503.http

    stats enable
    stats realm   Haproxy\ Statistics

    # Note: The password below is a UUID4 written in plain-text.
    stats auth    admin1:9c8d2d400a6d43bab7b10952773f0b51

    stats refresh 5s

# ##############################################################################

backend bck_http_plain
    mode http
    balance roundrobin

# ZATO begin backend bck_http_plain

    server http_plain--server1 127.0.0.1:17010 check inter 2s rise 2 fall 2 # ZATO backend bck_http_plain:server--server1
    server http_plain--server2 127.0.0.1:17011 check inter 2s rise 2 fall 2 # ZATO backend bck_http_plain:server--server2

# ZATO end backend bck_http_plain

# ##############################################################################

frontend front_http_plain

    mode http
    default_backend bck_http_plain

    option forwardfor
    option httplog # ZATO frontend front_http_plain:option log-http-requests
    bind 0.0.0.0:11223 # ZATO frontend front_http_plain:bind
    maxconn 200 # ZATO frontend front_http_plain:maxconn

    monitor-uri /zato-lb-alive # ZATO frontend front_http_plain:monitor-uri

frontend front_tls_optional_client_certs

  mode http
  default_backend bck_http_plain
  option forwardfor
  reqadd X-Forwarded-Proto:\ https

  acl has_x_forwarded_proto req.fhdr(X-Forwarded-Proto) -m found
  http-request deny if has_x_forwarded_proto

  bind 0.0.0.0:31223 ssl crt /key-and-cert.pem verify optional ca-file /ca-certs.pem
]command:[[u'haproxy', u'-c', u'-f', u'/opt/zato/cluster1/load-balancer/config/zato-tmpjPCAis']], return code:[1], stdout:[], stderr:[[ALERT] 118/122204 (27072) : parsing [/opt/zato/cluster1/load-balancer/config/zato-tmpjPCAis:65] : error detected while parsing ACL 'has_x_forwarded_proto'.
[ALERT] 118/122204 (27072) : parsing [/opt/zato/cluster1/load-balancer/config/zato-tmpjPCAis:66] : error detected while parsing an 'http-request deny' condition.
[ALERT] 118/122204 (27072) : Error(s) found in configuration file : /opt/zato/cluster1/load-balancer/config/zato-tmpjPCAis
]
]'>
]

It seems that the ACL line is rejected. I checked at http://www.haproxy.org/download/1.4/doc/configuration.txt and I don't see any examples of req.fhdr

After commenting out the reqadd, acl and http-request deny lines, I get instead:

\'bind\' only supports the \'transparent\', \'defer-accept\', \'name\', \'id\', \'mss\' and \'interface\' options

Maybe the version of haproxy which zato pulled in as a dependency is not SSL-aware? zato does not seem to depend on any specific version.

$ apt-cache show haproxy | grep Version:
Version: 1.4.24-2
$ apt-cache show zato | grep -i Depends
Depends: git, bzr, gfortran, haproxy, libatlas3gf-base, libblas3gf, libevent-2.0-5, libgfortran3, liblapack3gf, libpq5, libyaml-0-2, libxml2, libxslt1.1, libumfpack5.6.2, openssl, python2.7, python-m2crypto, python-numpy, python-pip, python-scipy, python-zdaemon, swig, uuid, uuid-runtime, bash-completion

Aha: according to this article SSL support was only included in haproxy 1.5 :-(

However I have not found any mention in the zato docs of needing to pull in a newer haproxy from a third-party repo.

UPDATE: I installed haproxy 1.5 from the repo linked in the above article. The error changes to:

\'bind 0.0.0.0:31223\' : unable to load SSL private key from PEM file \'/key-and-cert.pem\'

So this does seem to be the problem. I therefore suggest:

• document dependency on haproxy 1.5
• document the need to use zato ca create server (?) (with example)

[candlerb]

In case it was any help, I was able to make it work by doing

cd /opt/zato/cluster1/load-balancer/config/repo
cat zato-lba-priv-key.pem zato-lba-cert.pem >zato-lba-key-and-cert.pem

and putting either the absolute paths in haproxy config, or these relative ones:

  bind 0.0.0.0:31223 ssl crt config/repo/zato-lba-key-and-cert.pem verify optional ca-file config/repo/zato-lba-ca-certs.pem

[candlerb]

... and if there are intermediate CA certificates, their certificates also need to be appended to zato-lba-key-and-cert.pem - this is in the haproxy documentation