Trylok commented 1 year ago

After starting a HTTP farm the farms list ist empty.

l4xnat farms are working but not displayed anymore. Only stopping the corresponding Virtual Interface and rebooting the system brings the List back. For testing i did a fresh iso install, only creating a new HTTP farm. With the same result.

HTTP/HTTPS Farms do not work anymore, no matter on upgrade or fresh install.

Thanks, Michael

Rendone-zevenet commented 1 year ago

Hi, Could you attach a supportsave file from the system?

The HTTP/HTTPS Farms work as expected. Maybe something in the environment where the ISO is installed is causing that behavior.

Thanks in advance! Regards.

Trylok commented 1 year ago

Its a Hyper-V environment. The supportsave is from version 5.12. Because i did a rollback. Hope this helps. Here is the file: supportsave_ZEVENET_20230301_1409.tar.gz

Thanks, Michael

Sysadminfromhell commented 1 year ago

Same issue here, I upgraded and all my backends seem to run into a Timeout most of the times. Sometimes it works but mostly it doesnt.

Mar 1 15:15:02 HMLRPX02 zproxy: timeout request to backend 172.16.24.112:8080 Mar 1 15:15:02 HMLRPX02 zproxy: timeout request for client x.x.x.x:49186

It feels super weird. Can I go back to 5.12? I do not have a snapshot :(

Trylok commented 1 year ago

Just for fun i installed version 5.13 manually on an fresh debian 11 (bullseye). Webgui does not work out of the box. But after manually installing zevenet-gui-ce the webgui works fine. However, the problem remains the same. On the dashboard i get a endless rotating circle after starting http farms.

Here the supportsave from this version. supportsave_zevenet1_20230301_1509.tar.gz

Thanks, Michael

Sysadminfromhell commented 1 year ago

I cannot access the admin gui to create a supportfile. asap when im on the premises tomorrow I will create one. But it looks like that the backend is the problem, the access from there to the backend system always runs into a timeout for some reason. No matter how much seconds I setup to wait for the timeout. I even put 60 seconds via cli into but still after 60 seconds I only got timeout to the backend. Its not the backend system itself, theyre many different backend systems and all have the same issue.

Rendone-zevenet commented 1 year ago

Hi, # supportsave can be generated via the command line too.

/usr/local/zevenet/bin/supportsave

It will generate the file in /tmp directory.

Best regards.

Sysadminfromhell commented 1 year ago

Here u go :) supportsave_HMLRPX02_20230301_1930.tar.gz

Sysadminfromhell commented 1 year ago

Some update?

Trylok commented 1 year ago

@Sysadminfromhell stay calm ;-) I'm sure it's not that easy.

Rendone-zevenet commented 1 year ago

Hi, There are some "resurrected" backends, so it is caused by a reconnected communication between the system and the backend: Mar 1 10:15:17 HOSTNAME zproxy: [bk:BACKEN_IP:BACKEND_PORT] The backend resurrected

If the the other services experience timeouts they are caused by the backend communication

Mar 1 15:11:26 HOSTNAME zproxy: timeout request for client YYYY:65486 Mar 1 15:12:13 HOSTNAME zproxy: timeout request to backend XXXXX:8006 Mar 1 15:12:13 HOSTNAME zproxy: timeout request for client YYYY:65514 Mar 1 15:12:13 HOSTNAME zproxy: timeout request to backend XXXXX:8006 Mar 1 15:12:13 HOSTNAME zproxy: timeout request for client YYYY:65513 Mar 1 15:14:35 HOSTNAME zproxy: error reading from client YYYY:25120 (Connection reset by peer) Mar 1 15:14:35 HOSTNAME zproxy: [7f18dc2b5700] connection release for client YYYY:25120 (result=Connection reset by peer) Mar 1 15:14:35 HOSTNAME zproxy: error reading from client YYYY:26623 (Connection reset by peer) Mar 1 15:14:35 HOSTNAME zproxy: [7f18da2b1700] connection release for client YYYY:26623 (result=Connection reset by peer) Mar 1 15:14:35 HOSTNAME zproxy: error reading from client YYYY:29355 (Connection reset by peer) Mar 1 15:14:35 HOSTNAME zproxy: [7f18da2b1700] connection release for client YYYY:29355 (result=Connection reset by peer)

All the config files seems to be well-configured. The routing tables are correct. The Zevenet system and the backends are in the same network so there is no extra devices implied. It seems the backend is not able to response due to the timeout reached.

Could you test the connection to the backends? _telnet BACKEND_IP BACKENDPORT

Could you enable the farm logs to view more info in /var/log/syslog and test a connection to the service farm? Change param LogLevel in /usr/local/zevenet/config/FARMNAME_proxy.cfg and restart the farm. LogLevel 7

Could you paste the state of the farm after the test service farm?

_curl --unix-socket /tmp/FARMNAMEproxy.socket http://localhost/listener/0

Thanks in advance. Best regards.

Sysadminfromhell commented 1 year ago

Telnet works:

The curl doesnt work:, because my farm isnt existing.

And the log says:

Mar  2 14:46:10 HMLRPX02 zproxy: [7f5b8b16a700] accepting client connection from 192.168.178.12:63627
Mar  2 14:46:10 HMLRPX02 zproxy: Found cert for pve.safh.de
Mar  2 14:46:10 HMLRPX02 zproxy: [7f5b8a168700] accepting client connection from 192.168.178.12:63628
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:-][bk:-][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:702} GET / HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012Cache-Control: max-age=0#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012sec-ch-ua-platform: "Windows"#015#012Upgrade-Insecure-Requests: 1#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7#015#012Sec-Fetch-Site: none#015#012Sec-Fetch-Mode: navigate#015#012Sec-Fetch-User: ?1#015#012Sec-Fetch-Dest: document#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:-][bk:-][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 0 -> 1
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:-][bk:-][cl:192.168.178.12](debug)[caller/clearStats:121]The stream stats are not defined
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:735/735} GET / HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012Cache-Control: max-age=0#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012sec-ch-ua-platform: "Windows"#015#012Upgrade-Insecure-Requests: 1#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7#015#012Sec-Fetch-Site: none#015#012Sec-Fetch-Mode: navigate#015#012Sec-Fetch-User: ?1#015#012Sec-Fetch-Dest: document#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 2
Mar  2 14:46:10 HMLRPX02 zproxy: Found cert for pve.safh.de
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 2 -> 3
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:1101} HTTP/1.1 200 OK#015#012Cache-Control: max-age=0#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Pragma: no-cache#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 824#015#012Content-Type: text/html; charset=utf-8#015#012Expires: Thu, 02 Mar 2023 13:46:10 GMT#015#012#015#012#037#010
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](established) host:pve.safh.de - "GET / HTTP/1" "HTTP/1.1 200 OK" 824 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:1146/1146} HTTP/1.1 200 OK#015#012Cache-Control: max-age=0#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Pragma: no-cache#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 824#015#012Content-Type: text/html; charset=utf-8#015#012Expires: Thu, 02 Mar 2023 13:46:10 GMT#015#012Strict-Transport-Security: max-age=21600000#015#012#015#012#037#010
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:559} GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://pve.safh.de/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 3 -> 1
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:592/592} GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://pve.safh.de/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:16384} HTTP/1.1 200 OK#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 683505#015#012Content-Type: application/javascript#015#012Last-Modified: Tue, 14 Jan 2020 06:08:10 GMT#015#012#015#012#037#010
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](established) host:pve.safh.de - "GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1" "HTTP/1.1 200 OK" 683505 "https://pve.safh.de/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: resp_head_rcv -> resp_body_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:16429/683790} HTTP/1.1 200 OK#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 683505#015#012Content-Type: application/javascript#015#012Last-Modified: Tue, 14 Jan 2020 06:08:10 GMT#015#012Strict-Transport-Security: max-age=21600000#015#012#015#012#037#010
Mar  2 14:46:40 HMLRPX02 zproxy: timeout request for client 192.168.178.12:63628
Mar  2 14:46:40 HMLRPX02 zproxy: [7f5b8a168700] connection release for client 192.168.178.12:63628 (result=ok)
Mar  2 14:46:40 HMLRPX02 zproxy: timeout request to backend 172.16.24.102:8006
Mar  2 14:46:40 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](error) e504 Gateway timeout "Host:pve.safh.de" "GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1.1"
Mar  2 14:46:40 HMLRPX02 zproxy: timeout request for client 192.168.178.12:63627
Mar  2 14:46:40 HMLRPX02 zproxy: [7f5b8b16a700] connection release for client 192.168.178.12:63627 (result=ok)

Different access, different backend:

Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:-][bk:-][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 0 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:-][bk:-][cl:192.168.178.12](debug)[caller/clearStats:121]The stream stats are not defined
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] Stream 13 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:687/687} GET /guacamole/angular.min.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"177368-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 2
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 2 -> 3
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:673} GET /guacamole/guacamole.22d162f5cef4ad7db069.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"368162-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 3 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] Stream 11 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:706/706} GET /guacamole/guacamole.22d162f5cef4ad7db069.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"368162-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:133} HTTP/1.1 304 #015#012ETag: W/"16386-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/1.guacamole.bc6250d982db6462ff59.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] Stream 10 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:178/178} HTTP/1.1 304 #015#012ETag: W/"16386-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:622} GET /guacamole/app.js?b=20230227005223 HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-Modified-Since: Mon, 27 Feb 2023 08:54:22 GMT#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] Stream 8 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:655/655} GET /guacamole/app.js?b=20230227005223 HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-Modified-Since: Mon, 27 Feb 2023 08:54:22 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:134} HTTP/1.1 304 #015#012ETag: W/"177368-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/angular.min.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] Stream 13 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:179/179} HTTP/1.1 304 #015#012ETag: W/"177368-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:191} HTTP/1.1 304 #015#012Cache-Control: no-cache#015#012Pragma: no-cache#015#012Last-Modified: Mon, 27 Feb 2023 08:54:22 GMT#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/app.js?b=20230227005223 HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] Stream 8 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:236/236} HTTP/1.1 304 #015#012Cache-Control: no-cache#015#012Pragma: no-cache#015#012Last-Modified: Mon, 27 Feb 2023 08:54:22 GMT#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:660} GET /guacamole/1.guacamole.8c18237a4a94ee9845d9.css HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: text/css,*/*;q=0.1#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: style#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012Range: bytes=65373-79743#015#012If-Range: W/"79744-1677459432000"#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 3 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] Stream 12 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:693/693} GET /guacamole/1.guacamole.8c18237a4a94ee9845d9.css HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: text/css,*/*;q=0.1#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: style#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012Range: bytes=65373-79743#015#012If-Range: W/"79744-1677459432000"#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:134} HTTP/1.1 304 #015#012ETag: W/"368162-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/guacamole.22d162f5cef4ad7db069.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] Stream 11 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:179/179} HTTP/1.1 304 #015#012ETag: W/"368162-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:8192} HTTP/1.1 206 #015#012Accept-Ranges: bytes#015#012ETag: W/"79744-1677459432000"#015#012Last-Modified: Mon, 27 Feb 2023 00:57:12 GMT#015#012Content-Range: bytes 65373-79743/79744#015#012Content-Type: text/css#015#012Content-Length: 14371#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015#012{background:#822;border-color:#d44}.guac-keyboard .guac-keyboard-group{line-height:0}.guac-keyboard .guac-keyboard-group.guac-keyboard-alpha,.guac-keyboard .guac-keyboard-group.guac-keyboard-function,.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{display:inline-block;text-align:center;vertical-align:top}.guac-keyboard .guac-keyboard-group.guac-keyboard-main,.guac-keyboard .guac-keyboard-group.guac-keyboard-top{display:-ms-flexbox;-ms-flex-align:stretch;-ms-flex-direction:row;display:-moz-box;-moz-box-align:stretch;-moz-box-orient:horizontal;display:-webkit-box;-webkit-box-align:stretch;-webkit-box-orient:horizontal;display:-webkit-flex;-webkit-align-items:stretch;-webkit-flex-direction:row;display:flex;align-items:stretch;flex-direction:row}.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{-ms-flex:1 1 auto;-moz-box-flex:1;-webkit-box-flex:1;-webkit-flex:1 1 auto;flex:1 1 auto}.guac-keyboard .guac-keyboard-gap{display:inline-block}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key.guac-keyboard-uses-alt-gr .guac-keyboard-cap:not(.guac-keyboard-requires-alt-gr),.guac-keyboard.guac-keyboard-modifier-caps .guac-keyboard-key.guac-keyboard-uses-caps .guac-keyboard-cap:not(.guac-keyboard-requires-caps),.guac-keyboard.guac-keyboard-modifier-lat .guac-keyboard-key.guac-keyboard-uses-lat .guac-keyboard-cap:not(.guac-keyboard-requires-lat),.guac-keyboard.guac-keyboard-modifier-shift .guac-keyboard-key.guac-keyboard-uses-shift .guac-keyboard-cap:not(.guac-keyboard-requires-shift),.guac-keyboard:not(.guac-keyboard-modifier-alt-gr) .guac-keyboard-cap.guac-keyboard-requires-alt-gr,.guac-keyboard:not(.guac-keyboard-modifier-caps) .guac-keyboard-cap.guac-keyboard-requires-caps,.guac-keyboard:not(.guac-keyboard-modifier-lat) .guac-keyboard-cap.guac-keyboard-requires-lat,.guac-keyboard:not(.guac-keyboard-modifier-shift) .guac-keyboard-cap.guac-keyboard-requires-shift{display:none}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key:not(.guac-keyboard-uses-alt-gr):not(.guac-keyboard-key-alt-gr){opacity:.5}guac-player{display:inline-block;position:relative}guac-player .guac-player-display{position:absolute;top:0;left:0;width:100%;height:100%}guac-player .guac-player-controls{position:absolute;left:0;bottom:0;width:100%}.guac-player-controls .guac-player-seek{display:block;width:100%}.guac-player-controls .guac-player-pause,.guac-player-controls .guac-player-play{color:#fff;background:transparent;border:none;width:2em;height:2em;min-width:0;padding:0;margin:0}.guac-player-controls .guac-player-pause:hover,.guac-player-controls .guac-player-play:hover{background:hsla(0,0%,100%,.5)}.guac-player-controls .pause-icon,.guac-player-controls .play-icon{display:inline-block;width:2em;height:2em;background-size:contain;background-position:50%;background-repeat:no-repeat;vertical-align:middle}.guac-player-controls .play-icon{background-image:url(images/action-icons/guac-play.svg)}.guac-player-controls .pause-icon{background-image:url(images/action-icons/guac-pause.svg)}guac-player .guac-player-status{position:fixed;top:0;left:0;width:100%;height:100%;background:rgba(0,0,0,.5);z-index:1;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-moz-box-orient:vertical;-moz-box-direction:normal;-ms-flex-direction:column;flex-direction:column}.guac-player-display,guac-player .guac-player-status{display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}.guac-player-display{position:relative}.guac-player-display .guac-player-display-container{-webkit-box-flex:0;-webkit-flex:0 0 auto;-moz-box-flex:0;-ms-flex:0 0 auto;flex:0 0 auto}guac-player-progress-indicator{width:128px;height:128px;position:relative;display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}guac-player-progress-indicator .guac-player-progress-text{font-size:2em;font-weight:700}guac-player-progress-indicator .guac-player-progress-bar-container{position:absolute;right:0;top:0;width:50%;height:100%;overflow:hidden}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway{overflow:visible}guac-player-progress-indicator .guac-player-progress-bar,guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{position:absolute;left:-64px;top:0;width:128px;height:128px;-webkit-border-radius:128px;-moz-border-radius:128px;border-radius:128px;border-color:#5af transparent transparent #5af;border-style:solid;border-width:12px}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{content:" ";display:block;box-sizing:border-box;-webkit-transform:rotate(135deg);-moz-transform:rotate(135deg);-ms-transform:rotate(135deg);-o-transform:rotate(135deg);transform:rotate(135deg)}input[type=range]{background:transparent;width:100%;margin:0}input[type=range]:focus{outline:none}input[type=range]{-webkit-appearance:none}input[type=range]::-webkit-slider-runnable-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-webkit-slider-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;-webkit-appearance:none;cursor:pointer}input[type=range]:focus::-webkit-slider-runnable-track{background:#5af}input[type=range]::-moz-range-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-moz-range-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;cursor:pointer}input[type=range]::-ms-track{width:100%;background:transparent;color:transparent}input[type=range]::-ms-thumb,input[type=range]::-ms-track{height:.5em;margin:0;border:none;border-radius:0;cursor:pointer}input[type=range]::-ms-thumb{background:#fff;width:3px}input[type=range]::-ms-fill-lower,input[type=range]::-ms-fill-upper,input[type=range]:focus::-ms-fill-lower,input[type=range]:focus::-ms-fill-upper{border:none;border-radius:0;background:#5af}a.button.add-connection,a.button.add-connection-group,a.button.add-user,a.button.add-user-group{font-size:.8em;padding-left:1.8em;position:relative}a.button.add-connection-group:before,a.button.add-connection:before,a.button.add-user-group:before,a.button.add-user:before{content:" ";position:absolute;width:1.8em;top:0;bottom:0;left:0;background-repeat:no-repeat;background-size:1em;background-position:.5em .45em}a.button.add-user:before{background-image:url(images/action-icons/guac-user-add.svg)}a.button.add-user-group:before{background-image:url(images/action-icons/guac-user-group-add.svg)}a.button.add-connection:before{background-image:url(images/action-icons/guac-monitor-add.svg)}a.button.add-connection-group:before{background-image:url(images/action-icons/guac-group-add.svg)}.settings.connections .connection-list .new-connection,.settings.connections .connection-list .new-connection-group,.settings.connections .connection-list .new-sharing-profile{opacity:.5;font-style:italic}.settings.connect
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/1.guacamole.8c18237a4a94ee9845d9.css HTTP/1" "HTTP/1.1 206 " 14371 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] Stream 12 changing status: resp_head_rcv -> resp_body_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:8237/14704} HTTP/1.1 206 #015#012Accept-Ranges: bytes#015#012ETag: W/"79744-1677459432000"#015#012Last-Modified: Mon, 27 Feb 2023 00:57:12 GMT#015#012Content-Range: bytes 65373-79743/79744#015#012Content-Type: text/css#015#012Content-Length: 14371#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015#012{background:#822;border-color:#d44}.guac-keyboard .guac-keyboard-group{line-height:0}.guac-keyboard .guac-keyboard-group.guac-keyboard-alpha,.guac-keyboard .guac-keyboard-group.guac-keyboard-function,.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{display:inline-block;text-align:center;vertical-align:top}.guac-keyboard .guac-keyboard-group.guac-keyboard-main,.guac-keyboard .guac-keyboard-group.guac-keyboard-top{display:-ms-flexbox;-ms-flex-align:stretch;-ms-flex-direction:row;display:-moz-box;-moz-box-align:stretch;-moz-box-orient:horizontal;display:-webkit-box;-webkit-box-align:stretch;-webkit-box-orient:horizontal;display:-webkit-flex;-webkit-align-items:stretch;-webkit-flex-direction:row;display:flex;align-items:stretch;flex-direction:row}.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{-ms-flex:1 1 auto;-moz-box-flex:1;-webkit-box-flex:1;-webkit-flex:1 1 auto;flex:1 1 auto}.guac-keyboard .guac-keyboard-gap{display:inline-block}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key.guac-keyboard-uses-alt-gr .guac-keyboard-cap:not(.guac-keyboard-requires-alt-gr),.guac-keyboard.guac-keyboard-modifier-caps .guac-keyboard-key.guac-keyboard-uses-caps .guac-keyboard-cap:not(.guac-keyboard-requires-caps),.guac-keyboard.guac-keyboard-modifier-lat .guac-keyboard-key.guac-keyboard-uses-lat .guac-keyboard-cap:not(.guac-keyboard-requires-lat),.guac-keyboard.guac-keyboard-modifier-shift .guac-keyboard-key.guac-keyboard-uses-shift .guac-keyboard-cap:not(.guac-keyboard-requires-shift),.guac-keyboard:not(.guac-keyboard-modifier-alt-gr) .guac-keyboard-cap.guac-keyboard-requires-alt-gr,.guac-keyboard:not(.guac-keyboard-modifier-caps) .guac-keyboard-cap.guac-keyboard-requires-caps,.guac-keyboard:not(.guac-keyboard-modifier-lat) .guac-keyboard-cap.guac-keyboard-requires-lat,.guac-keyboard:not(.guac-keyboard-modifier-shift) .guac-keyboard-cap.guac-keyboard-requires-shift{display:none}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key:not(.guac-keyboard-uses-alt-gr):not(.guac-keyboard-key-alt-gr){opacity:.5}guac-player{display:inline-block;position:relative}guac-player .guac-player-display{position:absolute;top:0;left:0;width:100%;height:100%}guac-player .guac-player-controls{position:absolute;left:0;bottom:0;width:100%}.guac-player-controls .guac-player-seek{display:block;width:100%}.guac-player-controls .guac-player-pause,.guac-player-controls .guac-player-play{color:#fff;background:transparent;border:none;width:2em;height:2em;min-width:0;padding:0;margin:0}.guac-player-controls .guac-player-pause:hover,.guac-player-controls .guac-player-play:hover{background:hsla(0,0%,100%,.5)}.guac-player-controls .pause-icon,.guac-player-controls .play-icon{display:inline-block;width:2em;height:2em;background-size:contain;background-position:50%;background-repeat:no-repeat;vertical-align:middle}.guac-player-controls .play-icon{background-image:url(images/action-icons/guac-play.svg)}.guac-player-controls .pause-icon{background-image:url(images/action-icons/guac-pause.svg)}guac-player .guac-player-status{position:fixed;top:0;left:0;width:100%;height:100%;background:rgba(0,0,0,.5);z-index:1;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-moz-box-orient:vertical;-moz-box-direction:normal;-ms-flex-direction:column;flex-direction:column}.guac-player-display,guac-player .guac-player-status{display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}.guac-player-display{position:relative}.guac-player-display .guac-player-display-container{-webkit-box-flex:0;-webkit-flex:0 0 auto;-moz-box-flex:0;-ms-flex:0 0 auto;flex:0 0 auto}guac-player-progress-indicator{width:128px;height:128px;position:relative;display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}guac-player-progress-indicator .guac-player-progress-text{font-size:2em;font-weight:700}guac-player-progress-indicator .guac-player-progress-bar-container{position:absolute;right:0;top:0;width:50%;height:100%;overflow:hidden}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway{overflow:visible}guac-player-progress-indicator .guac-player-progress-bar,guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{position:absolute;left:-64px;top:0;width:128px;height:128px;-webkit-border-radius:128px;-moz-border-radius:128px;border-radius:128px;border-color:#5af transparent transparent #5af;border-style:solid;border-width:12px}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{content:" ";display:block;box-sizing:border-box;-webkit-transform:rotate(135deg);-moz-transform:rotate(135deg);-ms-transform:rotate(135deg);-o-transform:rotate(135deg);transform:rotate(135deg)}input[type=range]{background:transparent;width:100%;margin:0}input[type=range]:focus{outline:none}input[type=range]{-webkit-appearance:none}input[type=range]::-webkit-slider-runnable-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-webkit-slider-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;-webkit-appearance:none;cursor:pointer}input[type=range]:focus::-webkit-slider-runnable-track{background:#5af}input[type=range]::-moz-range-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-moz-range-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;cursor:pointer}input[type=range]::-ms-track{width:100%;background:transparent;color:transparent}input[type=range]::-ms-thumb,input[type=range]::-ms-track{height:.5em;margin:0;border:none;border-radius:0;cursor:pointer}input[type=range]::-ms-thumb{background:#fff;width:3px}input[type=range]::-ms-fill-lower,input[type=range]::-ms-fill-upper,input[type=range]:focus::-ms-fill-lower,input[type=range]:focus::-ms-fill-upper{border:none;border-radius:0;background:#5af}a.button.add-connection,a.button.add-connection-group,a.button.add-user,a.button.add-user-group{font-size:.8em;padding-left:1.8em;position:relative}a.button.add-connection-group:before,a.button.add-connection:before,a.button.add-user-group:before,a.button.add-user:before{content:" ";position:absolute;width:1.8em;top:0;bottom:0;left:0;background-repeat:no-repeat;background-size:1em;background-position:.5em .45em}a.button.add-user:before{background-image:url(images/action-icons/guac-user-add.svg)}a.button.add-user-group:before{background-image:url(images/action-icons/guac-user-group-add.svg)}a.button.add-connection:before{background-image:url(images/action-icons/guac-monitor-add.svg)}a.button.add-connection-group:before{background-image:url(images/action-icons/guac-group-add.svg)}.settings.connections .connection-list .new-connection,.settings.connections .connection-list .new-connection-group,.settings.connections .connection-list .new-shari
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:651} GET /guacamole/templates.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"97681-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] Stream 9 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:684/684} GET /guacamole/templates.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"97681-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:133} HTTP/1.1 304 #015#012ETag: W/"97681-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/templates.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] Stream 9 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:178/178} HTTP/1.1 304 #015#012ETag: W/"97681-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015

Rendone-zevenet commented 1 year ago

Hi, the curl command did not work because you use "/tmp/FARMNAME_proxy.socket" as a socket file without changing it to the real farm name. Anyway, we will analyze the debug log in order to find the issue. We guess there is enough information.

We will keep you updated.

Thanks!

nevola commented 1 year ago

Hi @Sysadminfromhell could you please try to increase the timeouts? Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues.

https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/

In addition, which kind of application is running in the backend side?

Thanks!

nevola commented 1 year ago

Hi again @Sysadminfromhell Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes:

Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

nevola commented 1 year ago

Hi again @Sysadminfromhell

Also, some "error reading from client" detected. Are those legitimated clients?

Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer) Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer) Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

Trylok commented 1 year ago

@nevola Just to be clear. As soon as you start an http/https farm, the farmlist is empty and the dashboard chrashes, as I have already described above. Even if no backends are configured. l4xnat farms to not show this beavior. So, it has nothing to do with the backends, Changing the timing makes no difference. Not even a simple url redirect works without crashing the farm.

Best regards.

emiliocampos-zevenet commented 1 year ago

We need to figure out what is happening with the web GUI,so please do the following:

Press 12 the development console will be loaded Go to network tab where all the requests will be shown Reproduce the problem, a line in red colour will be shown or I guess. Finally export all the data in the dev console to a HAR, here explained how to export a HAR file. https://support.google.com/admanager/answer/10358597?hl=en

Finally attach the file to this issue for our review

Thanks

El vie., 3 mar. 2023 15:50, Trylok @.***> escribió:

@nevola https://github.com/nevola Just to be clear. As soon as you start an http/https the farmlist is empty and the dashboard chrashes, as I have already described above. Even if no backends are configured. l4xnat farms to not show this beavior. So, it has nothing to do with the backends, Changing the timing makes no difference. Not even a simple url redirect works without crashing the farm.

Best regards.

— Reply to this email directly, view it on GitHub https://github.com/zevenet/zlb/issues/123#issuecomment-1453643791, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBQEPBQRE2JGM3MEQJLZG3W2IAMLANCNFSM6AAAAAAVKR3R5Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Trylok commented 1 year ago

Here you go. This is from the dashboard/home section, where i get the endless spinning cycle after starting the farm. 192.168.0.32.zip This is my test installation with just one farm configured.

The farms section looks like this before i start the farm. And this after starting it.

Sysadminfromhell commented 1 year ago

@nevola,

Hi @Sysadminfromhell could you please try to increase the timeouts? Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues.

https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/

In addition, which kind of application is running in the backend side?

Thanks!

I tried, this fixed the "error" partial. If I force refresh or access a different backend its not working. supportsave_HMLRPX02_20230303_1630.tar.gz

Hi again @Sysadminfromhell

Also, some "error reading from client" detected. Are those legitimated clients?

Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer) Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer) Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

not legit, but nothing really to be worried about, Happen all the time.

Hi again @Sysadminfromhell Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes:

Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

The guardian check on this backend is only "ping" this should work all the time.

Farm explenation:

pve => Proxmox VE Backend Guacamole => Apache Guacamole backend Grafana => Grafana Dashboard Backend PBS => Proxmox Backup Server Backend

Update: The pve and pbs works sometimes, the grafana and Guacamole one is always just white and empty. Nothing comes up not even a error page.

emiliocampos-zevenet commented 1 year ago

It seems like the request: https://192.168.0.32:444/zapi/v4.0/zapi.cgi/farms

is generating in the load balancer a 500 Error.

Please send us again a fresh supportsave of the home server 192.168.0.32 for our review?

Thanks!

i/v4.0/zapi.cgi/farms

El vie, 3 mar 2023 a las 16:13, Trylok @.***>) escribió:

Here you go. This is from the dashboard/home section. 192.168.0.32.zip https://github.com/zevenet/zlb/files/10883189/192.168.0.32.zip This is my test installation with just one farm configured.

The farms section looks like this before i start the farm. [image: image] https://user-images.githubusercontent.com/50360497/222755287-780b4566-20d8-4c01-a6a0-ebc92c58d375.png And this after starting it. [image: image] https://user-images.githubusercontent.com/50360497/222754874-6a1b87a3-f6fe-44b1-8177-d4f5d03b57bf.png

— Reply to this email directly, view it on GitHub https://github.com/zevenet/zlb/issues/123#issuecomment-1453678815, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBQEPH4NKHFZDF5MG7K22DW2IDBJANCNFSM6AAAAAAVKR3R5Y . You are receiving this because you commented.Message ID: @.***>

-- Emilio CamposZEVENET Teamwww.zevenet.com

Review ZEVENET Product at Gartner Insights https://gtnr.io/LfXtqnsSr https://www.linkedin.com/company/zevenet https://twitter.com/zevenet https://www.facebook.com/zevenet https://github.com/zevenet [image: ZEVENET] https://www.zevenet.com/signature/

DISCLAIMER: This message contains confidential information and is intended only for the individual named. If you are not the named addressee please notify the sender immediately by email if you have received it by mistake and delete it from your system, you should not disseminate, distribute or copy this email in whole or in part.

Trylok commented 1 year ago

Here you go. supportsave_zevenet1_20230303_1639.tar.gz

Rendone-zevenet commented 1 year ago

Hi Trylok, ZEVENET CE 5.13 is based on Debian Buster. The har file and the supportsave says the system is using a kernel 5.10. ... "compression": 67, "text": "{\n \"description\" : \"Get version\",\n \"params\" : {\n \"appliance_version\" : \"ZCE 5, hypervisor: HyperV\",\n \"hostname\" : \"zevenet1\",\n \"kernel_version\" : \"5.10.0-21-amd64\",\n \"system_date\" : \"Fri Mar 3 16:01:02 2023\",\n \"zevenet_version\" : \"5.13.0\"\n }\n}\n" ....

Please, try to install ZEVENET CE 5.13 in a Debian Buster. If the issue is experienced again, attach the supportsave and the har file again.

Thanks!

Trylok commented 1 year ago

@Rendone-zevenet Fresh install from the 5.13 iso. With and without package updates. The only diverence is, now the Dashboard does not hang anymore, and the farms remain listed. But the same end result. They are still not working. I run into the same timeouts on http/https farms as Adminfromhell from above. The l4xnat farm for MS-Exchange also no longer works correctly as I just found out. I get asked endlessly for credentials. Same configuration on 5.12.2 works just perfect. Here a supportsave from this try. supportsave_zevenet1_20230304_0110.tar.gz

nevola commented 1 year ago

Hi @Sysadminfromhell, it seems that it is related to network issues, please try to leave a ping from ZEVENET command line to your backends and check the lost packets.

If you need us to review something else, please open a new issue. Thank you.

@nevola,

Hi @Sysadminfromhell could you please try to increase the timeouts? Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues. https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/ In addition, which kind of application is running in the backend side? Thanks!

I tried, this fixed the "error" partial. If I force refresh or access a different backend its not working. supportsave_HMLRPX02_20230303_1630.tar.gz

Hi again @Sysadminfromhell Also, some "error reading from client" detected. Are those legitimated clients? Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer) Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer) Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

not legit, but nothing really to be worried about, Happen all the time.

Hi again @Sysadminfromhell Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes: Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

The guardian check on this backend is only "ping" this should work all the time.

Farm explenation:

pve => Proxmox VE Backend Guacamole => Apache Guacamole backend Grafana => Grafana Dashboard Backend PBS => Proxmox Backup Server Backend

Update: The pve and pbs works sometimes, the grafana and Guacamole one is always just white and empty. Nothing comes up not even a error page.

Sysadminfromhell commented 1 year ago

Hi @Sysadminfromhell, it seems that it is related to network issues, please try to leave a ping from ZEVENET command line to your backends and check the lost packets.

If you need us to review something else, please open a new issue. Thank you.

@nevola,

Hi @Sysadminfromhell could you please try to increase the timeouts? Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues. https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/ In addition, which kind of application is running in the backend side? Thanks!

I tried, this fixed the "error" partial. If I force refresh or access a different backend its not working. supportsave_HMLRPX02_20230303_1630.tar.gz

Hi again @Sysadminfromhell Also, some "error reading from client" detected. Are those legitimated clients? Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer) Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer) Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer) Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

not legit, but nothing really to be worried about, Happen all the time.

Hi again @Sysadminfromhell Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes: Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard' Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

The guardian check on this backend is only "ping" this should work all the time. Farm explenation: pve => Proxmox VE Backend Guacamole => Apache Guacamole backend Grafana => Grafana Dashboard Backend PBS => Proxmox Backup Server Backend Update: The pve and pbs works sometimes, the grafana and Guacamole one is always just white and empty. Nothing comes up not even a error page.

I cannot really see that this should be a problem. The Backend of Guacamole is perfectly available and the monitoring system never has a ping problem. A ping command also is perfectly in shape. Time between 0.050 - 0.300 ms. No lost packets:

72 packets transmitted, 72 received, 0% packet loss, time 751ms
rtt min/avg/max/mdev = 0.067/0.117/0.166/0.022 ms

Trylok commented 1 year ago

I can confirm that there are no losts packets to the backends. All of them perfectly available.

ct-utel commented 1 year ago

For what it's worth I can confirm that since the upgrade to 5.13 my HTTP/HTTPS Farms are no longer working properly either. My dashboard is fine but even though backends show as ok pages are not loading, some of the sites work again if I disable the farm guardian for the site but are still experience lag before the upgrade they did not and worked fine.

nevola commented 1 year ago

For what it's worth I can confirm that since the upgrade to 5.13 my HTTP/HTTPS Farms are no longer working properly either. My dashboard is fine but even though backends show as ok pages are not loading, some of the sites work again if I disable the farm guardian for the site but are still experience lag before the upgrade they did not and worked fine.

Dear @ct-utel could you please enable the logs in the affected farm, then reproduce the problem and finally, generate a support save file to be shared with us? Also, a wireshark pcap file would help to analyze what is happening.

Thank you!

ct-utel commented 1 year ago

Hi nevola,

Please find a support save I generated earlier today. The farm I've been trying to diagnose is Web-Redirects-HTTPS.

I'm about to head home for the day but I will try and generate the pcap file in the morning.

supportsave_svr-lb-01_20230306_1311.tar.gz

nevola commented 1 year ago

Hi @ct-utel

I'm seeing:

Mar 5 13:34:38 svr-lb-01 zproxy: [f:][th:61c5f700] Malformed chunk, size was not found Mar 5 13:34:38 svr-lb-01 zproxy: [f:][th:61c5f700] Malformed chunk, size was not found Mar 5 13:34:40 svr-lb-01 zproxy: unexpected state in zproxy_client_read: 3

If you could export an HAR file from your browser with the malformed request would help.

Please send the HAR file to support@zevenet.com

Thank you!

Hi nevola,

Please find a support save I generated earlier today. The farm I've been trying to diagnose is Web-Redirects-HTTPS.

I'm about to head home for the day but I will try and generate the pcap file in the morning.

ct-utel commented 1 year ago

Hi @nevola

I've sent a zip of a few example HAR files to the support address, please let me know if you require any further info.

Sysadminfromhell commented 1 year ago

So I could fix my issue.

The Guacamole was "broken" I had to recreate the dockers on the server and allow the procy explicit to access the page now its working.
Grafana was broken after an upgrade of the Dockers. I had to add som environment variables so the proxy can access it. Also the Dashboard ID got deleted from the website of Grafana so this was broken.
The Timeout for pve up to 45 seconds (instead of 40) fixed that. I also had to change the backend check from Ping to http or tcp check. after this and a little bit of wait now everything looks back to normal again.

Trylok commented 1 year ago

@Sysadminfromhell I am not using docker nor Grafana. Problem on my side remains unchanged. Dashboard and farmguardian says all ok. But its not. No single farm is working correctly. I restored 5.12.2 image from a backup. All is working now again. 5.13 is at this time not usable for me.

odedonato commented 1 year ago

Hi @ct-utel

I'm seeing:

Mar 5 13:34:38 svr-lb-01 zproxy: [f:][th:61c5f700] Malformed chunk, size was not found Mar 5 13:34:38 svr-lb-01 zproxy: [f:][th:61c5f700] Malformed chunk, size was not found Mar 5 13:34:40 svr-lb-01 zproxy: unexpected state in zproxy_client_read: 3

If you could export an HAR file from your browser with the malformed request would help.

Please send the HAR file to support@zevenet.com

Thank you!

Hi nevola, Please find a support save I generated earlier today. The farm I've been trying to diagnose is Web-Redirects-HTTPS. I'm about to head home for the day but I will try and generate the pcap file in the morning.

I had the same exact error in 3 separate installations, I had to revert, no problems at all with 5.12.2

ct-utel commented 1 year ago

Hi @odedonato

I don't suppose there's an easy downgrade option? Unfortunately I've fallen foul of not having a viable backup...

odedonato commented 1 year ago

Hi @odedonato

I don't suppose there's an easy downgrade option? Unfortunately I've fallen foul of not having a viable backup...

Luckily I made a snapshot on all 3 installations before upgrading, so I reverted back to the snapshot on 2 machine, the third one was restored from veeam because unfortunately I deleted the snapshot too early :)

nevola commented 1 year ago

Hi @odedonato

Could you please generate a support save and a HAR with the problematic request, and share it with us by email to support@zevenet.com ?

Could you also verify if this happens requesting with big files?

Thank you.

I had the same exact error in 3 separate installations, I had to revert, no problems at all with 5.12.2

odedonato commented 1 year ago

Hi @odedonato

Could you please generate a support save and a HAR with the problematic request, and share it with us by email to support@zevenet.com ?

Could you also verify if this happens requesting with big files?

Thank you.

I had the same exact error in 3 separate installations, I had to revert, no problems at all with 5.12.2

Sorry but I can't, there are sensitive informations.

We have a different services running, I don't know if zproxy: [f:][th:a3865700] Malformed chunk, size was not found zproxy: unexpected state in zproxy_client_read: 3 is related to an our mobile app that send json, but also we have this kind of log zproxy: [f:][th:a2062700] [st:6618][svc:MYSVC][bk:-][cl:aaa.bbb.ccc.ddd] Request body is not expected, ignore 2140 extra bytes:<json> and also I tried some of our web apps and those are very very very slow, around 60 seconds to load, the only log is zproxy: timeout request for client aaa.bbb.ccc.ddd]:61111 zproxy: timeout request for client aaa.bbb.ccc.ddd]:61114 zproxy: timeout request for client aaa.bbb.ccc.ddd]:61113 zproxy: timeout request for client aaa.bbb.ccc.ddd]:61115

ZEVENET Version: 5.13.0 WebGUI Version: 1.0.25-5.13.0 Appliance Version: ZCE 5 Kernel Version: 4.19.0-23-amd64 Debian Version: 10.13

nevola commented 1 year ago

Hi @odedonato

We'll require at least the requests that produces these errors and the proxy configuration file.

Kind Regards.

odedonato commented 1 year ago

Hi @odedonato

We'll require at least the requests that produces these errors and the proxy configuration file.

Kind Regards.

Mail sent

Trylok commented 1 year ago

Quick Update. Just tried version 5.13.1 No difference. Still non functional farms with Microsoft Exchange backends. Timeout on zproxy with https farms for OWA. Outlook endless asking for credentials on l4xnat farms.

naortega commented 1 year ago

Quick Update. Just tried version 5.13.1 No difference. Still non functional farms with Microsoft Exchange backends. Timeout on zproxy with https farms for OWA. Outlook endless asking for credentials on l4xnat farms.

Good day, @Trylok,

We're currently applying patches to fix this issue, and we seem to have narrowed down the issue and found a patch that appears to resolve it. To be sure we've narrowed it down correctly, can you tell me the properties of the request? Primarily I'd like to know if the Transfer-Encoding is chunked or not.

Thank you.

Trylok commented 1 year ago

@naortega Microsoft Exchange uses chunked Transfer-Encoding.

naortega commented 1 year ago

@naortega Microsoft Exchange uses chunked Transfer-Encoding.

Okay. We have a patch ready for this issue which we hope to be releasing soon.

Thank you for your patience and cooperation.

naortega commented 1 year ago

I'm posting to notify that a new version of zproxy (0.9.2) has been release that should fix the issues with chunking and timeouts.

ct-utel commented 1 year ago

@naortega thanks for the update.

With the latest updates we are still having issues.

We have HTTPS redirect for a TeamCity server, the GUI doesn't load at all via the load balancer but going to IP:port directly its fine.

We have a Home Assistance instance that uses the same farm, the initial login page loads but times out trying to use the dashboard. Again going directly to it, loads fine.

There is also a Project Send instance on the same farm. It loads fine internally but externally it's very slow to respond.

I'd be happy to send another support save file it that would help? so you might be able to compare the before and after upgrade.

naortega commented 1 year ago

@naortega thanks for the update.

With the latest updates we are still having issues.

We have HTTPS redirect for a TeamCity server, the GUI doesn't load at all via the load balancer but going to IP:port directly its fine.

We have a Home Assistance instance that uses the same farm, the initial login page loads but times out trying to use the dashboard. Again going directly to it, loads fine.

There is also a Project Send instance on the same farm. It loads fine internally but externally it's very slow to respond.

I'd be happy to send another support save file it that would help? so you might be able to compare the before and after upgrade.

A support save would be nice, but it would also be helpful to know better what problems you are experiencing. Regarding the first issue you experience (HTTPS redirect for TeamCity server): what exactly doesn't load? What's the request, what's the expected result, and what's the erroneous result you're getting?

In regards to the second issue (Home Assistance): Are you sure this isn't due to a network issue or a malconfiguration of the Zevenet machine? Could you show me the logs pertaining to this problem?

Thank you for your cooperation.

Trylok commented 1 year ago

@naortega I am sorry to inform you that nothing has changed after update zproxy to 0.9.2. Same problems as before. Here a new supportsave. supportsave_zevenet1_20230317_2126.tar.gz

ct-utel commented 1 year ago

A support save would be nice, but it would also be helpful to know better what problems you are experiencing. Regarding the first issue you experience (HTTPS redirect for TeamCity server): what exactly doesn't load? What's the request, what's the expected result, and what's the erroneous result you're getting?

In regards to the second issue (Home Assistance): Are you sure this isn't due to a network issue or a malconfiguration of the Zevenet machine? Could you show me the logs pertaining to this problem?

Thank you for your cooperation.

@naortega I have just e-mailed a new support save to the existing e-mail chain, I believe you are in it. The TeamCity server is getting "504 Gateway timeout - zproxy" just trying to get to it's web gui.

I'm pretty certain none of the issues are network related as the behavior started the moment we upgraded zevenet to 5.13, also getting to services directly without the load balance in the middle is fine.

emiliocampos-zevenet commented 1 year ago

one question, do you use websocket in this service?

Thanks!

El lun, 20 mar 2023 a las 14:22, ct-utel @.***>) escribió:

A support save would be nice, but it would also be helpful to know better what problems you are experiencing. Regarding the first issue you experience (HTTPS redirect for TeamCity server): what exactly doesn't load? What's the request, what's the expected result, and what's the erroneous result you're getting?

In regards to the second issue (Home Assistance): Are you sure this isn't due to a network issue or a malconfiguration of the Zevenet machine? Could you show me the logs pertaining to this problem?

Thank you for your cooperation.

@naortega https://github.com/naortega I have just e-mailed a new support save to the existing e-mail chain, I believe you are in it. The TeamCity server is getting "504 Gateway timeout - zproxy" just trying to get to it's web gui. [image: image] https://user-images.githubusercontent.com/127128354/226350446-4db4d74b-0d64-41da-8aad-a2978f63f7ad.png

I'm pretty certain none of the issues are network related as the behavior started the moment we upgraded zevenet to 5.13, also getting to services directly without the load balance in the middle is fine.

— Reply to this email directly, view it on GitHub https://github.com/zevenet/zlb/issues/123#issuecomment-1476225530, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBQEPHHT5K2EGLQBY7Y6ADW5BKY3ANCNFSM6AAAAAAVKR3R5Y . You are receiving this because you commented.Message ID: @.***>

-- Emilio CamposZEVENET Teamwww.zevenet.com

Review ZEVENET Product at Gartner Insights https://gtnr.io/LfXtqnsSr https://www.linkedin.com/company/zevenet https://twitter.com/zevenet https://www.facebook.com/zevenet https://github.com/zevenet [image: ZEVENET] https://www.zevenet.com/signature/

DISCLAIMER: This message contains confidential information and is intended only for the individual named. If you are not the named addressee please notify the sender immediately by email if you have received it by mistake and delete it from your system, you should not disseminate, distribute or copy this email in whole or in part.

zevenet / zlb

After Upgrade to 5.13 HTTP/HTTPS Farms not working anymore. #123

/usr/local/zevenet/bin/supportsave