Hitch exit(1) on reload

xdecock commented 3 years ago

Expected Behavior

On hitch reload, if an error occurs, the current process continue running

Current Behavior

We're currently encountering this behavior, which is strange:


Nov 30 01:30:21 host hitch[20345]: Received SIGHUP: Initiating configuration reload.
Nov 30 01:30:25 host hitch[20345]: {core} Config reloaded in 3.69 seconds. Starting new child processes.
Nov 30 01:30:25 host systemd[1]: hitch.service: Main process exited, code=exited, status=1/FAILURE
Nov 30 01:30:25 host systemd[1]: hitch.service: Failed with result 'exit-code'.



## Possible Solution
From what i can pinpoint, it looks like https://github.com/varnish/hitch/blob/master/src/hitch.c#L3431 The verify_privileges in the start_ocsp is the likely culprit, altough i have no clue how to confirm this or how it can fail.
So if you have any info i can provide I'll gladly help or anything to change.

## Steps to Reproduce (for bugs)
Currently, wait 1 week.

## Context
Once a week we got one of our 6 hitch instances getting stopped, with for the first time one of our production frontend stopped.

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: 1.6.1
* Operating System and version: 4.19.0-8-cloud-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux
* Source of binary packages used (if any) : package cloud : hi  hitch                                1.6.1-1~buster              amd64        Hitch TLS proxy

daghf commented 3 years ago

Hi @xdecock

Could you post your hitch.conf and your hitch command line configuration?

xdecock commented 3 years ago

Configuration:

#
# Example configuration file for hitch(8).
#
# NOTE: all config file parameters can be overriden
#       from command line!

# Listening address. REQUIRED.
# Can be specified multiple times for multiple listen endpoints.
# type: string
# syntax: [HOST]:PORT[+CERT]
frontend = "[*]:443"

# Upstream server address. REQUIRED.
#
# type: string
# syntax: [HOST]:PORT.
backend = "[127.0.0.1]:6081"

# SSL protocol.
#
# tls = on
# ssl = off
# tls-protos = TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
tls-protos = TLSv1.2 TLSv1.3

# List of allowed SSL ciphers.
#
# Run openssl ciphers for list of available ciphers.
# type: string

# Future ciphers (when TLSv1.1 is dropped)
ciphers = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"

# Enforce server cipher list order
#
# type: boolean
prefer-server-ciphers = on

# Use specified SSL engine
#
# type: string
ssl-engine = ""

# Number of worker processes
#
# type: integer
workers = 8

# Listen backlog size
#
# type: integer
backlog = 250

# TCP socket keepalive interval in seconds
#
# type: integer
keepalive = 600

# Chroot directory
#
# type: string
chroot = ""

# Set uid after binding a socket
#
# type: string
user = "hitch"

# Set gid after binding a socket
#
# type: string
group = "hitch"

# Quiet execution, report only error messages
#
# type: boolean
quiet = off

# Use syslog for logging
#
# type: boolean
syslog = on

# Log chattiness. 0=silence, 1=errors, 2=info/debug.
#
# type: integer
log-level = 0

# Syslog facility to use
#
# type: string
syslog-facility = "daemon"

# Run as daemon
#
# type: boolean
daemon = on

# Report client address by writing IP before sending data
#
# NOTE: This option is mutually exclusive with option write-proxy-v2, write-proxy and proxy-proxy.
#
# type: boolean
write-ip = off

# Report client address using SENDPROXY protocol, see
# http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt
# for details.
#
# NOTE: This option is mutually exclusive with option write-proxy-v2, write-ip and proxy-proxy.
#
# type: boolean
write-proxy-v1 = off

# Report client address using SENDPROXY v2 binary protocol, see
# http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt
# for details.
#
# NOTE: This option is mutually exclusive with option write-ip, write-proxy-v1 and proxy-proxy.
#
# type: boolean
write-proxy-v2 = on

# Report the chosen cipher and protocol as part of the PROXYv2 header.
#
# NOTE: This option is disabled due to the following varnish bug: https://github.com/varnishcache/varnish-cache/issues/3180
# type: boolean
proxy-tlv = off

# Proxy an existing SENDPROXY protocol header through this request.
#
# NOTE: This option is mutually exclusive with option write-proxy-v2, write-ip and write-proxy-v1.
#
# type: boolean
proxy-proxy = off

# Abort handshake when the client submits an unrecognized SNI server name.
#
# type: boolean
sni-nomatch-abort = on

# SSL x509 certificate file. REQUIRED.
# List multiple certs to use SNI. Certs are used in the order they
# are listed; the last cert listed will be used if none of the others match
#
# type: string
# pem-file=
pem-dir = "{{ hitch_directory_certificates }}"

# Available proto post-tls
# List the http/2 availability, comma separated values
#
# type:string
# values: h2, http/1.1
alpn-protos = "http/1.1"

# TCP Fast Open
#
# type: boolen
tcp-fastopen = on

# OCSP settings
#
# Directory where Hitch will store and read OCSP responses for
# stapling. Directory must be readable and writable for the configured
# hitch user. Setting this option enables automatic retrieval and
# updating of OCSP responses.
#
ocsp-dir = "/var/lib/hitch-ocsp"

service override:

/usr/sbin/hitch --config /etc/hitch/hitch.conf

[Service]
OOMScoreAdjust=-600
LimitNOFILE=40000
Restart=on-failure

xdecock commented 3 years ago

FYI we've got currently 2600 certificates in the directory

varnish / hitch

Hitch exit(1) on reload #343

Expected Behavior

Current Behavior