Intermittent errors when serving VestaCP through nginx reverse proxy

SteenSchutt commented 6 years ago

Operating System (OS/VERSION):

Debian 9.5 (Kernel 4.9.0-7)

VestaCP Version:

0.9.8-22

Installed Software (what you got with the installer):

nginx, apache, mysql, iptables, fail2ban

Steps to Reproduce:

Install VestaCP on a fresh Debian 9 (nginx + apache version)

curl -sO http://vestacp.com/pub/vst-install.sh
bash vst-install.sh \
    --force --nginx yes --apache yes --phpfpm no --vsftpd no --proftpd no \
    --named no --mysql yes --postgresql no --exim no --dovecot no --clamav no \
    --spamassassin no --iptables yes --fail2ban yes --quota no --lang en \
    --interactive no --hostname $(hostname --fqdn) --email sysop@example.com

Download my templates

wget -O $VESTA/data/templates/web/nginx/vestaproxy.stpl https://pastebin.com/raw/e6vYerDC
wget -O $VESTA/data/templates/web/nginx/vestaproxy.tpl https://pastebin.com/raw/RAmuUrCe

Create a new vhost through VestaCP to access it from in the future

v-add-web-domain admin vesta.example.com
v-change-web-domain-proxy-tpl admin vesta.exaple.com vestaproxy

Add SSL of whatever kind makes you happy
Visit the panel through the new domain, e.g. https://vesta.example.com:443
Create a new website with whatever bogus settings you want and try to delete it again. Result should be an error from the browser because the server returned an empty response. Sometimes it also shows as connection refused or even a VestaCP error saying that it doesn't exist.

Related Issues/Forum Threads:

My own thread on the issue: https://forum.vestacp.com/viewtopic.php?f=11&t=17639&p=73136#p73136

Other Notes:

If you cannot reproduce, I have a blank server that shows this exact behaviour.

Skamasle commented 6 years ago

Hi

You have any logs from vesta server or proxy template you make ?

SteenSchutt commented 6 years ago

@Skamasle The error doesn't generate any log entries. I can find an excerpt of the log tomorrow if you'd like, but it just says that everything works as it should. The vesta-nginx error log is empty, and vesta's system.log logs the Vesta actions correctly.

SteenSchutt commented 6 years ago

Okay. Here's some of what I get.

When creating a new domain (I have ensured that it does not exist on the server already), the error log says this:

2018-10-17 10:19:01 v-add-web-domain  'user' 'domain.dk' '159.69.42.206' 'no' 'www.domain.dk' 'jpeg,jpg,png,gif,bmp,ico,svg,tif,tiff,css,js,htm,html,ttf,otf,webp,woff,txt,csv,rtf,doc,docx,xls,xlsx,ppt,pptx,odf,odp,ods,odt,pdf,psd,ai,eot,eps,ps,zip,tar,tgz,gz,rar,bz2,7z,aac,m4a,mp3,mp4,ogg,wav,wma,3gp,avi,flv,m4v,mkv,mov,mpeg,mpg,wmv,exe,iso,dmg,swf' [Error 4]

I got a "domain already exists" error, which I presume is error 4. It didn't already exist and was created successfully.

When trying to delete it again, I got the following:

2018-10-17 10:22:02 v-restart-proxy  'yes' [Error 20]
2018-10-17 10:22:02 v-delete-domain  'user' 'domain.dk' [Error 20]

nginx failed to start back up (I haven't experienced this before) but manually restarting it from the terminal made it work again and I can see that the domain was deleted correctly.

/var/log/vesta/nginx-error.log spams the following, but I don't see how that would be related as I am not using Softaculous.

2018/10/17 10:25:01 [error] 6785#0: *4176 FastCGI sent in stderr: "PHP message: PHP Notice:  Undefined variable: _SESSION in /usr/local/vesta/web/softaculous/index.php on line 27
PHP message: PHP Notice:  Undefined variable: _SESSION in /usr/local/vesta/web/softaculous/index.php on line 32
PHP message: PHP Notice:  Undefined variable: _SESSION in /usr/local/vesta/web/softaculous/index.php on line 35
PHP message: PHP Notice:  Undefined variable: _SESSION in /usr/local/vesta/web/softaculous/index.php on line 60
PHP message: PHP Notice:  Use of undefined constant VESTA_CMD - assumed 'VESTA_CMD' in /usr/local/vesta/web/softaculous/index.php on line 65
PHP message: PHP Notice:  Use of undefined constant VESTA_CMD - assumed 'VESTA_CMD' in /usr/local/vesta/web/softaculous/index.php on line 76
PHP message: PHP Warning:  array_keys() expects parameter 1 to be array, null given in /usr/local/vesta/web/softaculous/index.php on line 85" while reading response header from upstream, client: 127.0.0.1, server: _, request: "POST /softaculous/index.php?act=cp_api&api=serialize HTTP/1.1", upstream: "fastcgi://unix:/var/run/vesta-php.sock:", host: "127.0.0.1:8083"

Again, feel free to send me a PM on the forum if you'd like access to the server to try it out for yourself.

madeITBelgium commented 6 years ago

Can you check /var/log/nginx log files?

SteenSchutt commented 6 years ago

@madeITBelgium Everything in that directory is empty.

troels-athenas commented 5 years ago

Hi, as I have access to the server @SteenSchutt , I was asked to give my input on this issue.

The problem is the POST causing a proxy restart and it runs on the same server. So before the response is made, nginx is shut down and the session thrown away.

I tried changing it to reload instead of restart, as that should be sufficient. Seems to fix the issue.

https://github.com/serghey-rodin/vesta/blob/0d85c88d181a314b22ec1d0489bdfb3269e53fd6/bin/v-restart-proxy#L53 Changed to: service $PROXY_SYSTEM reload >/dev/null 2>&1

This will also keep it running if the new config is faulty.

Don't know why a restart should be necessary, but it might be different for non nginx proxy?

SteenSchutt commented 5 years ago

I think it's worth adding that nginx has service nginx check-reload which will run nginx -t before attempting to reload. This way you can catch errors before reloading, to prevent unexpected downtime. Apache can do something similar, and not checking before reloading or restarting is one of my major gripes with VestaCP at the moment.

In most cases a reload should be sufficient, and I see how the restart would mess with the proxy setup we did. It would probably make sense to go through everything and change restarts to reloads where it makes sense, and add in a config check before doing so.

outroll / vesta