search

dlundquist / sniproxy

Proxies incoming HTTP and TLS connections based on the hostname contained in the initial request of the TCP session.
BSD 2-Clause "Simplified" License
2.55k stars 397 forks source link

Service stops, but daemon remains active until KILL #307

Open CptLeeChuck opened 6 years ago

CptLeeChuck commented 6 years ago

Hi,

I noticed another issue, from time to time, I need to adjust IP addresses in the config file and restart the service. I noticed a restart does not take effect on the new config file because it is actually not restarting. I can't stop the old service, and so the new one will fail. Any clue what could be the reason for this?

Here is the example on a fres rebootet OS:

Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

Last login: Sat Aug 25 20:46:31 2018 from 123.123.123.123
root@myserver:~# netstat -tlanp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      351/sshd            
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      561/master          
tcp        0      0 11.22.33.44:443      0.0.0.0:*               LISTEN      346/sniproxy        
tcp        0    268 11.22.33.44:22       123.123.123.123:56840     ESTABLISHED 565/sshd: root@pts/ 
tcp        0    137 11.22.33.44:443      188.174.114.171:60043    ESTABLISHED 346/sniproxy        
tcp        0      0 11.22.33.44:51450    123.123.123.123:447       ESTABLISHED 346/sniproxy        
tcp6       0      0 :::80                   :::*                    LISTEN      374/apache2         
tcp6       0      0 :::22                   :::*                    LISTEN      351/sshd            
tcp6       0      0 :::25                   :::*                    LISTEN      561/master          
tcp6       0      0 2aaa:238:4333:cccc::443 :::*                    LISTEN      346/sniproxy        
tcp6       0      0 11.22.33.44:80       123.123.123.123:29150     TIME_WAIT   -                   

root@myserver:~# /etc/init.d/sniproxy status
● sniproxy.service - LSB: HTTPS SNI Proxy
   Loaded: loaded (/etc/init.d/sniproxy; generated)
   Active: active (running) since Sat 2018-08-25 21:24:21 CEST; 36s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 318 ExecStart=/etc/init.d/sniproxy start (code=exited, status=0/SUCCESS)
    Tasks: 2 (limit: 105)
   CGroup: /system.slice/sniproxy.service
           ├─346 /usr/sbin/sniproxy -c /etc/sniproxy.conf
           └─350 /usr/sbin/sniproxy -c /etc/sniproxy.conf

Aug 25 20:57:22 myserver.mydomain.net systemd[1]: Starting LSB: HTTPS SNI Proxy...
Aug 25 20:57:22 myserver.mydomain.net systemd[1]: Started LSB: HTTPS SNI Proxy.
Aug 25 20:57:52 myserver.mydomain.net systemd[1]: Stopping LSB: HTTPS SNI Proxy...
Aug 25 20:57:52 myserver.mydomain.net systemd[1]: Stopped LSB: HTTPS SNI Proxy.
Aug 25 20:59:04 myserver.mydomain.net systemd[1]: Starting LSB: HTTPS SNI Proxy...
Aug 25 20:59:04 myserver.mydomain.net systemd[1]: Started LSB: HTTPS SNI Proxy.
Aug 25 21:20:38 myserver.mydomain.net systemd[1]: Stopping LSB: HTTPS SNI Proxy...
Aug 25 21:20:38 myserver.mydomain.net systemd[1]: Stopped LSB: HTTPS SNI Proxy.
Aug 25 21:24:20 myserver.mydomain.net systemd[1]: Starting LSB: HTTPS SNI Proxy...
Aug 25 21:24:21 myserver.mydomain.net systemd[1]: Started LSB: HTTPS SNI Proxy.

After fresh OS boot the server is running and listening on 443 with PID 346 as expected, but now let's stop the service (let's say because we'd like to adjusted the config) and check the status again:

root@myserver:~# /etc/init.d/sniproxy stop
[ ok ] Stopping sniproxy (via systemctl): sniproxy.service.
root@myserver:~# /etc/init.d/sniproxy status
● sniproxy.service - LSB: HTTPS SNI Proxy
   Loaded: loaded (/etc/init.d/sniproxy; generated)
   Active: inactive (dead) since Sat 2018-08-25 21:25:17 CEST; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 612 ExecStop=/etc/init.d/sniproxy stop (code=exited, status=0/SUCCESS)
  Process: 318 ExecStart=/etc/init.d/sniproxy start (code=exited, status=0/SUCCESS)
    Tasks: 2 (limit: 105)
   CGroup: /system.slice/sniproxy.service
           ├─346 /usr/sbin/sniproxy -c /etc/sniproxy.conf
           └─350 /usr/sbin/sniproxy -c /etc/sniproxy.conf

Aug 25 20:57:52 myserver.mydomain.net systemd[1]: Stopping LSB: HTTPS SNI Proxy...
Aug 25 20:57:52 myserver.mydomain.net systemd[1]: Stopped LSB: HTTPS SNI Proxy.
Aug 25 20:59:04 myserver.mydomain.net systemd[1]: Starting LSB: HTTPS SNI Proxy...
Aug 25 20:59:04 myserver.mydomain.net systemd[1]: Started LSB: HTTPS SNI Proxy.
Aug 25 21:20:38 myserver.mydomain.net systemd[1]: Stopping LSB: HTTPS SNI Proxy...
Aug 25 21:20:38 myserver.mydomain.net systemd[1]: Stopped LSB: HTTPS SNI Proxy.
Aug 25 21:24:20 myserver.mydomain.net systemd[1]: Starting LSB: HTTPS SNI Proxy...
Aug 25 21:24:21 myserver.mydomain.net systemd[1]: Started LSB: HTTPS SNI Proxy.
Aug 25 21:25:17 myserver.mydomain.net systemd[1]: Stopping LSB: HTTPS SNI Proxy...
Aug 25 21:25:17 myserver.mydomain.net systemd[1]: Stopped LSB: HTTPS SNI Proxy.

root@myserver:~# netstat -tlanp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      351/sshd            
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      561/master          
tcp        0      0 11.22.33.44:443      0.0.0.0:*               LISTEN      346/sniproxy        
tcp        0    268 11.22.33.44:22       123.123.123.123:56840     ESTABLISHED 565/sshd: root@pts/ 
tcp        0      0 11.22.33.44:443      188.174.114.171:60045    TIME_WAIT   -                   
tcp        0      0 11.22.33.44:443      188.174.114.171:60047    TIME_WAIT   -                   
tcp6       0      0 :::80                   :::*                    LISTEN      374/apache2         
tcp6       0      0 :::22                   :::*                    LISTEN      351/sshd            
tcp6       0      0 :::25                   :::*                    LISTEN      561/master          
tcp6       0      0 2aaa:238:4333:cccc::443 :::*                    LISTEN      346/sniproxy        
tcp6       0      0 11.22.33.44:80       123.123.123.123:29164     TIME_WAIT   -                   
tcp6       0      0 2aaa:238:4333:cccc::443 2003:71:83ee:9550:53272 TIME_WAIT   -                   
tcp6       0      0 2aaa:238:4333:cccc::443 2003:71:83ee:9550:53271 TIME_WAIT   -                   
root@myserver:~#

The service is in state inactive (dead) with two remaining tasks. And netstat shows it is still listening. This is the reason why a restart does not take effect, as it can't start again because the old service with PID 346 is still listening. This is why the sniproxy error log contains also this error:

2018-08-25 21:24:13 waitpid: No child processes
2018-08-25 21:26:29 bind 11.22.33.44:443 failed: Address already in use
2018-08-25 21:26:29 Failed to initialize listener 11.22.33.44:443

I hope you can point me again in the right direction to get this sorted out.

BTW: The behavior is the same after I KILL PID 346 (350) and start the service again. Now the new config file is loaded, but I can't stop it without KILL.

oakaigh commented 5 years ago

@CptLeeChuck Kind of wondering why you wrapped dirty old init.d service script inside system service. Take a look at the sample sniproxy.service file

[Unit] 
Description=SNI Proxy Service 
After=network.target 

[Service] 
Type=forking
ExecStart=/usr/sbin/sniproxy -c /etc/sniproxy/sniproxy.conf

[Install] 
WantedBy=multi-user.target

@dlundquist Should we add this file to the repository so that sniproxy service is shipped along with the compiled package?