Performance issue with Lucee, BonCode, and IIS

[hilsojunior]

I'm migrating my system from ACF to Lucee. I'm finding it amazing to have some new possibilities, considering I've been on ACF 2016. However, I've been facing an issue that I can't solve for more than 15 days. Currently, my ACF server runs on a Windows server using IIS. We have individual connectors for each site, and for over 4 years, I haven't encountered any performance problems; it works perfectly. As ACF has a very high cost, we decided to switch to Lucee. We've been making small adjustments to our code for about 6 months to make it run 100%, and we succeeded! =]

The problem is that we are unable to get the Lucee server to respond at the same speed as we do with ACF. I'll include performance screenshots below to help you understand what I'm talking about.

I'm running a test with a CFM file containing:

<cfset = “test”>
<cfoutput>
    #variables.test#
</cfoutput>

Conducting a test through K6 simulating 1000 VUs for 1 minute.

Windows - ACF 2016 - IIS

Checks: 217.386 http_req_waiting: 276.37ms

Windows - Lucee - IIS

Checks: 21.464 http_req_waiting: 2.81s

Windows - IIS (HTML text)

Checks: 337.681 http_req_waiting: 177.74ms

Windows - Lucee - without the IIS, consequently without the connector

Checks: 204.170 http_req_waiting: 293.63ms

For me, as a beginner with Lucee, it becomes evident that when there is a connection from Lucee to IIS, the bottleneck causing delays is in the connector. Below are the server.xml configurations

<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector port="8888" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />
    <Connector 
        protocol="AJP/1.3" 
        port="8009" 
        packetSize="65536" 
        secret="xxxx" 
        secretRequired="true" 
        redirectPort="8443" 
        address="::1" 
        maxConnections="10000" 
        MaxThreads="5000" 
    />
    <Engine name="Catalina" defaultHost="127.0.0.1">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>
      </Realm>
      <Host name="127.0.0.1"  appBase="webapps" unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b" />
        <Valve className="mod_cfml.core" loggingEnabled="false" maxContexts="200" timeBetweenContexts="2000" scanClassPaths="false" responseCode="307" sharedKey="xxxx" />
      </Host>
    </Engine>
  </Service>
</Server>

BonCodeAJP13.settings

<Settings>
<Server>localhost</Server>
<Port>8009</Port>
<EnableRemoteAdmin>True</EnableRemoteAdmin>
<EnableHeaderDataSupport>True</EnableHeaderDataSupport>
<ForceSecureSession>False</ForceSecureSession>
<AllowEmptyHeaders>False</AllowEmptyHeaders>
<ModCFMLSecret>xxx</ModCFMLSecret>
<RequestSecret>xxxx</RequestSecret>
<MaxConnections>1000</MaxConnections>
<PacketSize>65536</PacketSize>
<EnableAggressiveGC>true</EnableAggressiveGC>
<FlushThreshold>1</FlushThreshold>
</Settings>

"The question is, what should be done to achieve results similar to ACF?"

Thanks

OS: Windows Server 2022 Standard Java Version: 11.0.20.1 Tomcat Version: 9.0.80 Lucee Version: 6.0.0.585

[mdcmos]

Not sure this is your issue, just chiming in as we've had issues in the past with the AJP connector when using ipv6 localhost, so you may want to try replacing address="::1" in your server.xml AJP connector statement with address="127.0.0.1".

[Bilal-S]

@hilsojunior on the surface the setup looks Ok even if some more than common flags are used. Are you running lucee on the same machines that had previously ACF installed ?

Can you also try to adjust the server.xml address to "127.0.0.1" and Boncode setting file to this:

<Settings>
<Server>127.0.0.1</Server>
<Port>8009</Port>
<EnableRemoteAdmin>True</EnableRemoteAdmin>
<EnableHeaderDataSupport>True</EnableHeaderDataSupport>
<ModCFMLSecret>xxx</ModCFMLSecret>
<RequestSecret>xxxx</RequestSecret>
<PacketSize>65536</PacketSize>
</Settings>

[hilsojunior]

When I leave it this way I get the following error

Error connecting to Apache Tomcat instance. Please check that a Tomcat server is running at given location and port. Details: No connection could be made because the target machine actively refused it 127.0.0.1:8009

You can change this message by changing TomcatConnectErrorURL setting in setting file.

[hilsojunior]

Atualizando: I substituted the value "127.0.0.1" with "::1" because that is what responds when I ping localhost. The speed improved, but it didn't reach the performance of ACF. Not satisfied, I continued researching and I think I managed to solve it. I'll leave the BonCode file here as a record for future reference for anyone who needs it.

<Settings>
<Server>::1</Server>
<Port>8009</Port>
<EnableRemoteAdmin>True</EnableRemoteAdmin>
<EnableHeaderDataSupport>True</EnableHeaderDataSupport>
<ModCFMLSecret>xxxx</ModCFMLSecret>
<RequestSecret>xxxx</RequestSecret>
<PacketSize>65536</PacketSize>
<MaxConnections>4000</MaxConnections>
</Settings>

Server.xml

<Connector 
        protocol="AJP/1.3" 
        port="8009" 
        packetSize="65536" 
        secret="xxx" 
        secretRequired="true" 
        redirectPort="8443" 
        address="::1"
        enableLookups="false"
        keepAliveTimeout="-1"
        maxThreads="4000"
    />

The information that made all the difference is that the "MaxConnections" in BonCode must be the same value as the "maxThreads" in the server.xml. Along with keepAliveTimeout="-1". These adjustments completely changed the performance. I will include a screenshot of the test.

Thanks @Bilal-S and @mdcmos

[Bilal-S]

I am glad to hear you figured this out. I was going to double check the address attribute. If I interpret this correctly the Lucee instance is serving your request at twice the speed of ACF with shorter overall connection times but the iteration value is different? Though some mystery remains, that would be good progress right? I guess, this is the fun you get to have while changing engines 😄

Cheers, Bilal

[hilsojunior]

Yes, with just a few adjustments, the result doubled. I continue reading to see if more adjustments are possible, but I'm already quite satisfied with the outcome.

Now, a question: In ACF, it's possible to configure separate connectors for each site configured in IIS through the "Web server configuration," so if one site has any issues, it doesn't affect the others. Is this also possible in Lucee? I'm trying to gather information on this, but so far, I haven't found anything quite clear on this matter, and if it exists, how should I proceed.

I'm finding this engine switch quite enjoyable. =]

[Bilal-S]

The short answer is yes. You can do per-site connections. The user-manual has some info on how to do this. You have to uninstall the global connector and use per site connectors. You can connect to different tomcat instances running on different ports or to same one.
You do this by running the installer multiple times (once per site). Not sure how much an advantage segregating connectors is is when you only run one tomcat and one IIS engine even with many sites. Best place to get feedback is on lucee support forums https://dev.lucee.org

[hilsojunior]

Okay, thank you very much once again.

Success for you! Hugs