After Upgrade to v5.13.1 HTTPS Farms not working correctly

[MakcaR]

when trying to access a farm with an https Load Balancer, the connection is terminated.

the load balancer and users are in different subnets. The architecture is like this: The pfsense firewall has a subnet for application servers 10.10.0.0/25 and a subnet for users (workstations) 10.10.1.0/27. An https-type company has an address of 10.10.0.13, users open the Web address of the 1C accounting system application. The web interface of the accounting system is organized through 2 web servers, which are just the backends of the https farm.

An unrecoverable error Error when executing a GET request to the /e1cib/metadata/splash resource: due to: HTTP error when accessing the server: https://xxxx.xxxxxxxs.com Transferred a partial file

before the upgrade to v5.13.1 (on version 5.12), the problem was not observed supportsave_nlb-0_20230316_1913.tar.gz

[MakcaR]

An update became available today: zproxy/buster 0.9.2-5.13.1 amd64 [upgradable from: 0.9.1-5.13.1] zproxy/now 0.9.1-5.13.1 amd64 [installed,upgradable to: 0.9.2-5.13.1]

the problem is gone. Thanks to the developers for their promptness.

[MakcaR]

I hurried to rejoice. When contacting via the web, the problem is gone. But, when contacting through the client part of the program, the problem was not solved, the https company immediately closes the connection.... when processing via l4xnat farm, there is no breakdown... supportsave_nlb-0_20230318_1812.tar.gz

[naortega]

I hurried to rejoice. When contacting via the web, the problem is gone. But, when contacting through the client part of the program, the problem was not solved, the https company immediately closes the connection.... when processing via l4xnat farm, there is no breakdown... supportsave_nlb-0_20230318_1812.tar.gz

Could you clarify what you mean by "contacting via the web" and "contacting through the client"? If I had to guess, perhaps you are trying to connect to the service via HTTP while using an HTTPS listener on the zproxy. Could you try connecting explicitly using HTTP and also explicitly using HTTPS, tell me the results, and send in a supportsave with the results? Meanwhile I'll try to reproduce the issue.

Thank you for your cooperation.

[MakcaR]

Yes, I apologize for the confusing description of the problem. I will try to explain, and so: Our accounting system is able to work through a web browser i.e. users go to the address in the browser https://xxxxxx.domain.com /"database name" (picture1.PNG) and through its own (supplied by the developers of the accounting system itself) client application, the so-called "thin client" in which the same string is also specified for the connection https://xxxxxx.domain.com /"database name" (picture2.PNG). So in both cases, after upgrading from v5.12 to 5.13.1, there was a problem that I described in the first post. But then the update of the zproxy/buster 0.9.2-5.13.1 packages became available after its update: in the first case (when accessing through any browser), the problem went away, and in the second case (when using the client part of the accounting system, the problem remained.

[naortega]

Could you show me the error that the client is receiving? Particularly I'd like to see what the web console (in the browser) has to say when the client fails to connect.

[MakcaR]

if we use a browser, there is no problem, I can't reproduce it In the case of a "Thin Client", the break occurs immediately after the connection to the database begins. in the screenshot, what the "Thin client" writes when trying to open the database - The connection is lost

I can give logs from backends, I attach part of the log this is at the moment when the connection loss message crashes

10.10.0.121, -, 3/20/2023, 12:33:17, W3SVC1, 1c-lic-0, 10.10.0.26, 1, 102, 792, 200, 0, GET, /thinclient/, -, 10.10.0.121, -, 3/20/2023, 12:33:17, W3SVC1, 1c-lic-0, 10.10.0.26, 20, 1086, 305, 200, 0, GET, /E51020301/ru/e1cib/ping, -, 10.10.0.121, -, 3/20/2023, 12:33:21, W3SVC1, 1c-lic-0, 10.10.0.26, 1, 102, 792, 200, 0, GET, /thinclient/, -, 10.10.0.121, -, 3/20/2023, 12:33:24, W3SVC1, 1c-lic-0, 10.10.0.26, 10, 654, 8686, 200, 0, GET, /E51020301/e1cib/modules/src, sysver=8.3.22.1750&confver=ac11d27d6003d1233fbcf886cdeb124b4a516f5a&lm=2&id=urn%3Amodule%3Amd%3A44b8472b-0f70-49cc-8f71-9c9d3e9cfbcc%40property%3D%27d5963243-262e-4398-b4d7-fb16d06484f6%27%3Bversion%3D%27d5fe9dc1426d7d4d8e430f33c6f7f12f00000000%27, 10.10.0.121, -, 3/20/2023, 12:33:24, W3SVC1, 1c-lic-0, 10.10.0.26, 4, 654, 2634, 200, 0, GET, /E51020301/e1cib/modules/src, sysver=8.3.22.1750&confver=ac11d27d6003d1233fbcf886cdeb124b4a516f5a&lm=2&id=urn%3Amodule%3Amd%3A5a2d5d8a-afd9-4954-9868-7c82bad00426%40property%3D%27d5963243-262e-4398-b4d7-fb16d06484f6%27%3Bversion%3D%279eca85de077d58418f0e69844ea05b5400000000%27, 10.10.0.121, -, 3/20/2023, 12:33:24, W3SVC1, 1c-lic-0, 10.10.0.26, 5, 654, 1686, 200, 0, GET, /E51020301/e1cib/modules/src, sysver=8.3.22.1750&confver=ac11d27d6003d1233fbcf886cdeb124b4a516f5a&lm=2&id=urn%3Amodule%3Amd%3A13fcf1e8-730e-436f-b316-6bcb8301f894%40property%3D%27d5963243-262e-4398-b4d7-fb16d06484f6%27%3Bversion%3D%27919c4cc465b6c347878c476a37ce66fc00000000%27, 10.10.0.121, -, 3/20/2023, 12:33:27, W3SVC1, 1c-lic-0, 10.10.0.26, 9, 654, 2549, 200, 0, GET, /E51020301/e1cib/modules/src, sysver=8.3.22.1750&confver=ac11d27d6003d1233fbcf886cdeb124b4a516f5a&lm=2&id=urn%3Amodule%3Amd%3A67d39bce-be9f-48af-bab3-b84e65c8336f%40property%3D%27d5963243-262e-4398-b4d7-fb16d06484f6%27%3Bversion%3D%2784d72e492cd28040b56db279a6c2288f00000000%27, 10.10.0.121, -, 3/20/2023, 12:33:27, W3SVC1, 1c-lic-0, 10.10.0.26, 4, 654, 2047, 200, 0, GET, /E51020301/e1cib/modules/src, sysver=8.3.22.1750&confver=ac11d27d6003d1233fbcf886cdeb124b4a516f5a&lm=2&id=urn%3Amodule%3Amd%3Ad10f646d-6a08-47b6-a2b9-a83434642170%40property%3D%27d5963243-262e-4398-b4d7-fb16d06484f6%27%3Bversion%3D%27f3f5eb86e2809b45b123919467b1484c00000000%27, 10.10.0.121, -, 3/20/2023, 12:33:27, W3SVC1, 1c-lic-0, 10.10.0.26, 4, 654, 1713, 200, 0, GET, /E51020301/e1cib/modules/src, sysver=8.3.22.1750&confver=ac11d27d6003d1233fbcf886cdeb124b4a516f5a&lm=2&id=urn%3Amodule%3Amd%3A698ec00f-bf75-420e-81ab-a98808daf3f7%40property%3D%27d5963243-262e-4398-b4d7-fb16d06484f6%27%3Bversion%3D%279a6bac9a8836634f9ef10c84f065fe4300000000%27,

[naortega]

@MakcaR I'm unsure as to where those logs are coming from. Could you clarify?

Also, I'm not sure exactly how the "Thin Client" is trying to "open the database". Do you mean it's trying to connect to the database directly via some protocol other than HTTP/HTTPS, and through the load balancer?

[MakcaR]

He is trying to connect over https, and only over it. On 5.12 everything worked and the problem started only after updating from 5.12 to 5.13. and further in 5.13.1 My load balancer is a virtual machine, and if I take a copy of the VM with version 5.12 from the backup, then I have no problems accessing the databases through the load balancer. Something has changed in 5.13.1

[MakcaR]

my load balancer is a cluster of 2 nodes, can it affect? as far as I understand, there was no update of the cluster package.

[naortega]

my load balancer is a cluster of 2 nodes, can it affect? as far as I understand, there was no update of the cluster package.

In 5.13 we've upgraded to a new zproxy implementation that's supposed to be an improvement upon the older zproxy. Though we seem to still be filling out the edges.

Could you try seeing what SSL/TLS versions are being used by your thin client and ensuring that the load balancer is configured to work with those versions?

[MakcaR]

good question!, I will try to look / search for information about this from the developers of the accounting system. And which SSL/TLS does the load balancer use? I have such settings, and they have not changed since 5.12

[naortega]

good question!, I will try to look / search for information about this from the developers of the accounting system. And which SSL/TLS does the load balancer use? I have such settings, and they have not changed since 5.12

Considering the HTTPS parameters you've just displayed, my guess is that the only protocols enabled are TLSv1.2 and TLSv1.3. Something you could try, if your able, is enabling the other protocols available to see if perhaps it is simply an issue of the protocols you're using (that is, to untoggle the disabled options you have for HTTPS parameters). Please report your findings, thank you.

[MakcaR]

I would agree with you if I was just setting up the work of the load balancer and backends on IIS for the first time, but!!!, the configuration of the load balancer and HTTPS farm, which in the screenshot did not change and was configured more than 6 months ago on v5.11, then there was an update to 5.12 and there were also no problems with either the load balancer or with backends... and only after the update to 5.13, the rpoblema appeared... then updating to 5.13.1 did not solve the problem. the removal of the zproxy/buster 0.9.2-5.13.1 package solved the problem for the case of working through the browser (I wrote above that users work in 2 ways with the application), but the problem remained for working with the "Thin Client" method.

[MakcaR]

as for SSL/TLS that the application backend uses, I looked in the documentation of my application and the developers confirmed that everything is supported, and those that are allowed on IIS that are on the backend are used for data transmission. in my case, these are the ones in the screenshot. For clarity, I will show you through IIS Crypto

[naortega]

@MakcaR I agree, this is not an issue with your configuration. My intention with enabling (for testing purposes) the other protocols would be to see, not if your configuration is a problem, but if the internal logic of zproxy SSL protocol handling needs to be revised.

What I'd need to know isn't what is supported by your backends, but what is supported by your thin client which sends the request to zproxy.

[MakcaR]

on clients, the situation is the same, SSL/TLS uses those that are allowed in the OS on which the "Thin Client" is running. We have it all starting with TLS 1.0 and higher. As a test, I turned on all TLS 1.0 and higher on the farm... but this did not help, communication interruptions will continue to be observed

[MakcaR]

in the documentation for the application 1C:Enterprise I found this:

In order for the system "1C:Enterprise" could correctly determine the HTTP request of the client application and some parameters of the client application, it is necessary to configure the reverse proxy in such a way that:

● To "restore" an HTTP request: when forwarding an HTTP request, the request headers X-Forwarded-Port, X-Forwarded-Host and X-Forwarded-Proto were formed accordingly.

● To correctly determine the IP address of an external client application: when forwarding an HTTP request, the X-Forwarded-For request headers were formed accordingly.

But I don't understand yet where to look at it or configure it on the farm

[MakcaR]

just in case, the network scheme is like this. And users who come from remote sites have no problems, there is a problem only with local network users 10.10.1.0/27

[naortega]

@MakcaR Is the issue with any user in the 10.10.1.0 subnetwork, or just with the thin client? If at all possible, could you give me a client log of the error, such as a 4xx/5xx response or something of the sort?

[MakcaR]

We turned on the log on the client machine and that's what we see, every time the information window falls out that the connection is lost:

15:58.703000-0,EXCP,1,process=1cv8c,OSThread=13968,Exception=580392e6-ba49-4280-ac67-fcd6f2180121,Descr='src\vrscore\src\VResourceConnectionImpl.cpp(608) : 580392e6-ba49-4280-ac67-fcd6f2180121: Connection setup error Error when executing a POST request to the /e1cib/dlist resource: ed776789-afce-4ed9-8983-93ae0ace6e3c: HTTP error when accessing the server: https://1caas.u-clouds.com Failed sending data to the peer' 15:58.703001-0,EXCP,1,process=1cv8c,OSThread=13968,Exception=580392e6-ba49-4280-ac67-fcd6f2180121,Descr='src\vrscore\src\VResourceSessionImpl.cpp(534): 580392e6-ba49-4280-ac67-fcd6f2180121: Connection setup error Error when executing a POST request to the /e1cib/dlist resource: ed776789-afce-4ed9-8983-93ae0ace6e3c: HTTP error when accessing the server: https://1caas.u-clouds.com Failed sending data to the peer' 16:00.953000-0,EXCP,1,process=1cv8c,OSThread=13968,Exception=580392e6-ba49-4280-ac67-fcd6f2180121,Descr='src\vrscore\src\VResourceConnectionImpl.cpp(608) : 580392e6-ba49-4280-ac67-fcd6f2180121: Connection setup error Error when executing a POST request to the /e1cib/modules/call resource: ed776789-afce-4ed9-8983-93ae0ace6e3c: HTTP error when accessing the server: https://1caas.u-clouds.com Failed sending data to the peer',Context='

fully assembled log file attached 23032119.log

[MakcaR]

We tested the connection from users' machines from the "Thin Client" directly to IIS on backends, bypassing the load balancer. There is no problem described above, the test was performed on each of the two backends to make sure that both work fine

[naortega]

We tested the connection from users' machines from the "Thin Client" directly to IIS on backends, bypassing the load balancer. There is no problem described above, the test was performed on each of the two backends to make sure that both work fine

Is it possible that your thin client is utilizing websocket to connect? This is an issue that has been reported by other users (e.g. #123).

[MakcaR]

no the websocket is not used, and users who work via the Internet have no problem. there are only those who work on the same site where the load balancer, only in another subnet, according to the scheme above

[naortega]

Would it be possible for you to generate a HAR file containing the failing requests? I'd like to see the requests and responses.

[MakcaR]

HAR can be collected only by means of MS EDGE, but there are just no problems when working through the browser. But so as to understand what requests are coming, during the session 1caas.u-clouds.com.har.zip

I'll try to collect something via Wireshark, I'll post it by the evening

[naortega]

I'll try to collect something via Wireshark, I'll post it by the evening

That would be very helpful. Thank you very much. And thank you for the HAR file.

[MakcaR]

And so I spent a little time to conduct tests and capture through Wireshark. It turned out 2 files:

1. 1CApp_via_Zevenet_Connection_Losst.pcapng this is through the Event + 1CApp_via_Zevenet_Connection_Losst.PNG screenshot on the feed, you can see which abnormal packets during the information message from the application
2. 1CApp_wo_NLB_to_Backend.pcapng without a balancer directly to one of the Backends 10.10.0.26 I didn't see any anomalies. Desktop.zip

IP of the test user machine 10.10.1.10 DNS 10.10.02, 10.10.0.1 yes, and through the load balancer, the connection is established via TLSv1.3 and without it via TLSv1.2

[nevola]

Dear @MakcaR ,

1. Could you please enable the farm logs? Once activated, please reproduce the error again and please share supportsave via email at support@zevenet.com
2. Did you tried is the problem is reproduced with just 1 backend?
3. Could you please deactivate the StrictTransportSecurity option and try again?
4. I noticed in your client logs that the errors are only with POST requests, is that correct? Do you experience problems only with such kind of requests?

Looking forward to your response, Kind Regards.

[MakcaR]

Yes, mostly errors with POST requests Is the magazine included in the Edit farm guardian section? I haven't found the inclusion of logs in the web interface anywhere else. I have 2 backends 10.10.0.26 and 10.10.0.6 There are no problems with both if you work without a load balancer, and there is a problem on both, there is traffic going through the load balancer. In general, backends are 2 virtual machines that are configured exactly the same

supportsave supportsave_nlb-1_20230322_1818.tar.gz sent to support@zevenet.com

according to point 3, I will conduct the test tomorrow during the day

[nevola]

Hi @MakcaR , Good news.

We have a new binary with several fixes that might resolve your issues. Please download it from here:

https://drive.google.com/file/d/1s9iA28TsD07kcvr5OSIiHW5l8J562KsV/view?usp=share_link

You've to upload to the appliance and rename it in the path /usr/local/zevenet/app/zproxy/bin/zproxy (Please rename the current one in case of rollback)

After that, please restart the farms and let us know how it goes, Thank you!

[MakcaR]

Hi, I replaced the file by the link, and restarted the farm, but this did not solve the problem.

I think the problem is that the application cannot work on TLS 1.3, and the farm raises this protocol, not TLS 1.2. and in the farm settings there is no way to disable the use of TLS 1.3.

If I connect directly to the Backends, the connection is raised via protocol 1.2 and there is no problem.

[nevola]

Hi @MakcaR could you please try this binary out? https://drive.google.com/file/d/1Yx7rXgrljgLVvhVFTUbKptJbUWiGkajk/view?usp=share_link

The SSL options in the proxy have not been changed, isn't it?

Thanks!

[MakcaR]

No, I haven't changed anything, I haven't touched anything there for a long time.

[nevola]

Ok thank you for the confirmation @MakcaR Please provide some feedback after using the latest binary I sent.

https://drive.google.com/file/d/1Yx7rXgrljgLVvhVFTUbKptJbUWiGkajk/view?usp=share_link

Thank you.

[MakcaR]

with the zproxy_force_ssl_ps option, the problem was not solved, although the number of breaks decreased

now it is already possible to login to the application, before the breaks were at the stage of the login password prompt

[nevola]

@MakcaR could you please try to add in the configuration file the directive:

Disable TLSv1_3

Just below the Ciphers directive. Then, restart the farm to apply the changes.

Thank you, Kind Regards.

[MakcaR]

added to config file /usr/local/zevenet/config/1CService_proxy.cfg Disable TLSv1_3 rebooted the farm - did not help rebooted the entire VM machine of the load balancer - did not help

the connection, anyway, uses protocol TLS 1.3

[nevola]

Hi @MakcaR, please download the zproxy binary from here and apply it in zevenet. https://drive.google.com/file/d/1nAFf3D0AYSwM496MEiWXLA71d4PMJZCJ/view?usp=share_link

Please let me know if it's working now with the thin clients.

Thank you.

[MakcaR]

With the latest version of the zproxy_tls file, the connection goes over TLS 1.2 protocol, but connection losses are still present. I attach a screenshot with a capture file from Wireshark. where 10.10.0.14 is the interface of the load balancer with the HTTPS farm, and 10.10.1.1 is the machine with which the connection was tested. LostConn_log.zip supportsave_nlb-0_20230404_1659.tar.gz

[nevola]

Hi @MakcaR, please enable the farm logs, then reproduce the problem and finally, please regenerate a fresh support save (send it via support@zevenet.com ).

Thanks.

[MakcaR]

Hi, I reproduced the problem, generated a new support save and send to support@zevenet.com.

[bcnet-dev]

Hi, Any update about this issue ? I updated my cluster to 5.13.3 and the HTTPS farm issue is still present as descibed in this post. I also had the issue on the 5.13.2 version. For me, the only working version is the 5.12.2 Thanks

[naortega]

Hi, Any update about this issue ? I updated my cluster to 5.13.3 and the HTTPS farm issue is still present as descibed in this post. I also had the issue on the 5.13.2 version. For me, the only working version is the 5.12.2 Thanks

Good day, @bcnet-dev

Are you referring to that there are certain chunked responses/requests that are not being sent in full? If so, we have issued a patch for this issue in version 5.13.2, though this version does not seem to work for you. Please the issue you seem to be having, and send a support save, and if possible a HAR, to support@zevenet.com.

[MakcaR]

Over the past couple of weeks, there have been several update packages, but my problem has not been solved. In general, while we are sitting through a workaround TCP farm. This of course creates some problems, but in this mode I have only 15 users, so it is tolerable for now. But to the developers, we are very much waiting for a solution to the problem. Because the use of ZEVENET CE was conceived as testing before buying the version, and in general, the situation is not in favor of this software yet. In general, we are waiting for a solution to the problem.

[nevola]

Dear @MakcaR , in regards the TLS 1.2 connection please update, ensure you've only "Disable TLS1.3" enabled and try again. It should work, otherwise, please share again a fresh supportsave and HAR (or wireshark trace) via support@zevenet.com . Thank you.

[bcnet-dev]

Hello, After the update, our HTTPS farms serving Remote Desktop 2019 gateways stop working. We can only access the RDS farm through the HTML5 webinterface but it doesn't work through the .rdp file or via the URL configured in the "Work Resources" control panel of the client computer. We also have a L4XNAT farm for Exchange 2019 on the same Zevenet Cluster and this seems to be working normally after the update.

[nevola]

Hi @bcnet-dev , the connection is be performed? or you're experiencing outages? could you provide a support save via support@zevenet.com ? Also, a wireshark trace could be useful. Thank you.

[MakcaR]

Dear @MakcaR , in regards the TLS 1.2 connection please update, ensure you've only "Disable TLS1.3" enabled and try again. It should work, otherwise, please share again a fresh supportsave and HAR (or wireshark trace) via support@zevenet.com . Thank you.

hi, latest updates are all that have been installed, but I don't have such an option "Disable TLSv1.3"

[naortega]

@MakcaR Sorry for taking a while to get back to you.

Would it be possible for you to add the following directive manually to your Zproxy configuration?

Disable TLSv1_3

This should be placed inside the ListenerHTTPS block.

[MakcaR]

Made changes to the config, but it didn't solve the problem. TLS 1.3 is disabled, but the problem persists. screenshot and packet captures from wireshark are attached. S1.zip

and supportsave_nlb-0_20230513_1817.tar.gz sent to support@zevenet.com