Requests much slower than expected through IIS/AJP connector

[bhartsfield]

I have a Windows Server 2016 VM with IIS and a couple of instances of Lucee 5.3.5+92 (using Commandbox). I am using the 1.0.41 connector to connect two IIS sites to their respective commandbox/Lucee instances.

What I am seeing with that setup is that going directly to the underlying commandbox instances is much faster than sending the request through IIS and the AJP connector. On average, a request directly to the underlying server is about 300ms while, on average through IIS/AJP is about 1.4 seconds.

The boncode settings are minimal.

<Settings>
    <Server>localhost</Server>
    <Port>8009</Port>
    <EnableRemoteAdmin>False</EnableRemoteAdmin>
    <EnableHeaderDataSupport>False</EnableHeaderDataSupport>
    <ForceSecureSession>False</ForceSecureSession>
    <AllowEmptyHeaders>False</AllowEmptyHeaders>
    <ResolveRemoteAddrFrom>HTTP_X_FORWARDED_FOR</ResolveRemoteAddrFrom>
    <PacketSize>65531</PacketSize>
</Settings>

As are the web.config settings.

<handlers>
    <add name="BonCode-Tomcat-CFC-Handler" path="*.cfc" verb="*" type="BonCodeIIS.BonCodeCallHandler" preCondition="integratedMode" />
    <add name="BonCode-Tomcat-CFM-Handler" path="*.cfm" verb="*" type="BonCodeIIS.BonCodeCallHandler" preCondition="integratedMode" />
</handlers>

I have zero proof the the slowness is being caused by the connector or that it is being caused by IIS but it seems to be one of them (or a combination of both).

Any thoughts on how to speed this up or track down the actual cause?

[Bilal-S]

This may be related to Tomcat resolution bug. This adds about 1s to the roundtrip. Please try to add the address attribute to your AJP definition in server.xml file of tomcat

address="::1"

[bhartsfield]

This was commandbox/lucee and the default address is already 127.0.0.1. I changed that to ::1 in server.json and it did solve the issue. I tried manually setting it to 127.0.0.1 as well but the issue came back.

Some other odd values that solved the problem were "0.0.0.0" and simply "0".

That all seems very odd to me but it is definitely working now. And, you were right, it was almost exactly 1 second.

Thanks for your quick assistance.

[Bilal-S]

👍

[bdw429s]

I'd feel better if the actual root cause of this slowness was identified as it seems to affect a lot of people. in the ase above, Tomcat isn't even in use, it's JBoss Undertow, so blaming this on a Tomcat bug doesn't really work. Knowing that this affects more than one servlet container makes me question if Boncode isn't doing something wrong and the "all-port" binding is just a workaround and not an actual fix. Users should be able to bind to localhost (which may be required for security reasons) and still have Boncode perform properly.

[Bilal-S]

Brad: (@bdw429s) thanks for the feedback. I would like to know the reason as well. Any insight is welcome.

Tomcat isn't even in use, it's JBoss Undertow: --> AFAIK they use the tomcat engine and add their own wrappers. So tomcat is still at play.

Boncode isn't doing something wrong and the "all-port" binding --> BonCode does not do any type of all-port-bingdings only specific IP and Port all the time using the Windows .net managed TCP/IP controls. We can't be doing anything wrong on that level as we are not allowed to make the changes needed. AJP works on top of TCP/IP. If there is a binding issue that is lower level a Windows bug should be reported to Microsoft.

Users should be able to bind to localhost --> of course. Without a specific host, boncode will not start. The default is to use Windows localhost definition which may differ based on your machine setting and IpV4 vs Ipv6.

# localhost name resolution is handled within DNS itself.
# 127.0.0.1       localhost
# ::1             localhost

You can provide a specific IP address instead during setup.

[bdw429s]

AFAIK they use the tomcat engine and add their own wrappers. So tomcat is still at play.

Absolutely not. Undertow has nothing at all to do with Tomcat and does not use it in anyway. In fact, Tomcat is an Apache Commons product, and Undertow comes from JBoss/Redhat. Undertow was written from scratch and based on XNIO. I've pointed out in the past how the Boncode docs incorrectly refer to Tomcat a lot, but Tomcat is just one of many servlet contains that may sport an AJP listener.

BonCode does not do any type of all-port-bingdings

And I never said it did. I was referring to the workaround of using 0.0.0.0 for the servlet's AJP listener which binds to all addresses. (All ports was a typo, I mean all network adapters)

We can't be doing anything wrong on that level as we are not allowed to make the changes needed.

I'm not sure on what grounds you're saying that Boncode can't be doing anything wrong. Perhaps it isn't, but until we know the root cause of the connection delay, we can't be sure. I'm also not sure what you mean by "not allowed to make the changes needed". What changes are you not allowed to make?

AJP works on top of TCP/IP. If there is a binding issue that is lower level a Windows bug should be reported to Microsoft.

On what grounds? First you say this is a Tomcat bug and now it's a Windows bug? Without root cause analysis, we don't know where the issue lies. The delay is likely in some sort of host name resolution and unrelated to the binding itself.

of course. Without a specific host, boncode will not start

Who said anything about not using a specific host? I understand how localhost works and I also understand that a specific IP can be provided. But none of that is really relevant since I'm saying configuring Boncode to connect to localhost and configuring the AJP listener in your servlet to bind to localhost should be a correctly-working scenario and we should try to figure out why it doesn't work correctly for everyone.

I have a feeling it's related to the host name resolution of localhost but that's what we need to prove. is there a way to "profile" the code in boncode in a similar fashion to the way FusionReactor allows me to profile CFML code and see what lines are taking the longest to execute?

[Bilal-S]

Brad Brother more power to you. I am not trying to be right or create total enlightenment. If you wish to make a contribution whether its for docs or any other thing please submit a PR or point to the exact issue that you are suggesting we change.

The problem started occurring with Tomcat changes to fix Ghostcat vulnerability. This without changes on MS stack or BonCode.

Here an observation from my buddy Andreas:

Just an additional note: Apache shipped a server.xml with an active AJP connector tag and no address attribute until v.9.0.30. From Tomcat v.9.0.31 until release v9.0.35, server.xml AJP connector tag is inactive, but it has a predefined attribute address="::1" defined. The ajp-configuration docs states:
"address: For servers with more than one IP address, this attribute specifies which address will be used for listening on the specified port. By default, the connector will listen on the loopback address. Unless the JVM is configured otherwise using system properties, the Java based connectors (NIO, NIO2) will listen on both IPv4 and IPv6 addresses when configured with either 0.0.0.0 or ::. The APR/native connector will only listen on IPv4 addresses if configured with 0.0.0.0 and will listen on IPv6 addresses (and optionally IPv4 addresses depending on the setting of ipv6v6only) if configured with ::."

Cheers, B