search

Bilal-S / iis2tomcat

AJP Connector between Internet Information Services (IIS) and Apache Tomcat
http://www.boncode.net/boncode-connector
49 stars 32 forks source link

Generic Communication Error #66

Closed srconklin closed 6 years ago

srconklin commented 6 years ago

I am having the exact same issues as referenced in the now closed issue nr 47.

Browsing the Lucee Server admin, every few clicks I get the error:

Please check and adjust your setup: Ensure that Tomcat is running on given host and port. If this is a timeout error consider adjusting IIS timeout by changing executionTimeout attribute in web.config (see manual). Connection between Tomcat and IIS experienced error. If you restarted Tomcat this is expected. at BonCodeAJP13.BonCodeAJP13ServerConnection.ConnectionError() at BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend) at BonCodeIIS.BonCodeCallHandler.ProcessRequest(HttpContext context)

ERROR LOGGED IS:

Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat() 2018-06-29 11:30:56 failed Server Connection was aborted 2018-06-29 11:30:56 1.0.36 ERROR TCP Client level -- Server/Port:localhost/8009 Connection between Tomcat and IIS experienced error. If you restarted Tomcat this is expected. failed:Server Connection was aborted at BonCodeAJP13.BonCodeAJP13ServerConnection.ConnectionError(String message, String messageType) at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat() at BonCodeAJP13.BonCodeAJP13ServerConnection.HandleConnection() at BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend)

I thought the solution for this was to add keepaliveTimeout='-1' to the connector port in the server.xml setting, but that does not seem to help.

These are my BoncodeAJP13.settings 

<Settings>
<Server>localhost</Server>
<Port>8009</Port>
<EnableRemoteAdmin>False</EnableRemoteAdmin>
<EnableHeaderDataSupport>True</EnableHeaderDataSupport>
<ForceSecureSession>False</ForceSecureSession>
<AllowEmptyHeaders>False</AllowEmptyHeaders>
<MaxConnections>1000</MaxConnections>
<PacketSize>65536</PacketSize>
<LogLevel>1</LogLevel>
<LogDir>c:\temp</LogDir>
<FlushThreshold>0</FlushThreshold>
<ModCFMLSecret>99f00ff4667bd09d636438c007090a696ebec2c1a14ca9fc004c6771faf1b02d</ModCFMLSecret>
</Settings>

Here is my connector port settings:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"  keepAliveTimeout="-1" enableLookups="false" maxThreads="1000" connectionTimeout="121000" packetSize="65536" />

Here are my Lucee/OS/Java versions Lucee 5.2.7.63 OS Windows Server 2016 (10.0) 64 bit IIS 10.0.14393.0 Tomcat: 8.5.31 Java 1.8.0_172 (oracle Corp) 64 bit Boncode v 1.0.36

NOTE: I reset the IIS timeout back to 20 mins from 2 mins as per the recommendation of the poster in ISSUE nr 47

Do you know what the issue could be? and how I can fix it?

Bilal-S commented 6 years ago

You should not get an error every few clicks. Did you do a default install first?

srconklin commented 6 years ago

yes, This is fresh install.. ISP just spun the Windows instance up a few days ago. There is only one application running on it and I am the single user testing a newly migrated ADCF application.

It happens on localhost browsing the admin and on the web application site with it's testing domain url.

srconklin commented 6 years ago

OR did you mean did I try it with the default settings first? I typically always use the "performance" recommendations as indicated in the docs which have served me well for years on Lucee 4.0X/IIS7.5-8 installations.

Are you saying I should go back to the out of box settings?

srconklin commented 6 years ago

I guess I am going to try and go back to the out of box settings as this problem is causing the testing to be completely unreliable.

Bilal-S commented 6 years ago

I have setup a fresh lucee 5.2.7 (lucee-5.2.7.063-pl0-windows-installer) and trying your exact settings. I am simulating a constant load of 12 users. It has been two hours and no errors yet. I will keep monitoring.

Bilal-S commented 6 years ago

@srconklin I am able to replicate it when I push enough load on the server. Some connections fail and it causes the display of the generic messages. I will need to see whether I can do anything to prevent connection failure.

When you notice these what is the CPU on the server? Are any other users on it?

srconklin commented 6 years ago

@Bilal-S I am the only one on the server so the load is trivial. I actually reverted back to the out of box settings and have not had the issue since.

(i.e. got rid of all the performance settings recommended in the docs that I have always used with Lucee 4 + Windows 2012)

I am not sure of the CPU was on the server at the time of the failure. I can only imagine it was not much as the machine is running only Lucee..

How are you simulating user load? I would like to do the same thing and report my findings to you.

Bilal-S commented 6 years ago

I think I have this one resolved. Just testing some more. The simple mechanism for load is just a cfm page that does a few tasks and refreshes itself at a random interval. The number of browser windows that I use is the number of users I simulate. For more complex testing I use apache JMeter.

srconklin commented 6 years ago

ah. I have used JMeter before, but I like the simple mechanism you employ for testing...nice approach. Will the solution be a new version or just some configuration setting changes?

Bilal-S commented 6 years ago

@srconklin Can you try this build and let me know if this resolves your issue. This is not a full install pacakge. You can take your downloaded zip of 1.0.36 and replace the same files. https://drive.google.com/file/d/1FCW8aK6S20oQH5kgO8WN91Y3ziagFzCA/view?usp=sharing

Bilal-S commented 6 years ago

Should be fixed with 1.0.37 https://github.com/Bilal-S/iis2tomcat/releases/tag/1.0.37

srconklin commented 6 years ago

Thanks. Sorry I never got to try out what you suggested. Got busy with other things.

srconklin commented 5 years ago

Hi Bilal -

I am having this issue again after about a week of putting this machine/site into production. After several reports by users that they were getting the Generic Communication Error, I reviewed what you said in this thread a while back and decided to upgrade to the latest version.

1) I downloaded and replaced the Boncode Connector with the latest version: 1.0.41 2) followed these instructions for the installation http://www.boncode.net/boncode-connector/upgrading-railo-or-lucee-connector

and I am still experiencing the same issue. ( i get it almost every other click when browsing the CF admin using localhost.. http://localhost/lucee/admin/server.cfm)

A few things have changed since the first time I reported this issue B) Lucee is now version: 5.2.9.31 C) I changed PacketSize in both Tomcat and the .settings file to be 65530 per the recommendation of someone else in another thread. D) I set the Application Pool Defaults (Idle time-out) back to 20 mins from 2 mins per the recommendation of another thread I read.

My connector configuration is:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" keepAliveTimeout="-1" enableLookups="false" maxThreads="1000" connectionTimeout="121000" packetSize="65536" />

my server settings are: Lucee 5.2.9.31 Windows Server 2016 (10.0) 64bit java: 1.8.0_172 (Oracle Corporation) 64bit IIS 10.0.14393.0

bonCodeAJP13.settings are: `

localhost 8009 True True False False 1000 65530 99f00ff4667bd09d636438c007090a696ebec2c1a14ca9fc004c6771faf1b02d

`

I have not tried turning on logging yet. I wanted to check with you first

Do you have any thoughts?

Bilal-S commented 5 years ago

If it is happening while you browse around the admin I should be able to replicate it on my side. I will try again this weekend.

Do you see any errors in the tomcat logs?

srconklin commented 5 years ago

Thanks. I do not see any errors in the tomcat logs.

srconklin commented 5 years ago

Bilal-

Is there anyway you can look at this sooner? The client said their end-users are experiencing on the public facing eCommerce site and are complaining that they cannot place orders. They are saying this is hurting business.

I think I will at least take out all the optimizations in the meantime setting everything back to the out of box settings to see if that helps.

Bilal-S commented 5 years ago

Scott, I am travelling and will not be at home until Friday evening where I have my setup to test this. So, if you can revert to defaults please do so. Can you also verify the version has loaded correctly. On server, run the
http://localhost/a.cfm?BonCodeConnectorVersion=true connector command.

Best, Bilal

srconklin commented 5 years ago

Bilal - Thanks. Reverting back to the defaults seems to have solved the issue. no pressure now on troubleshooting issue

Bilal-S commented 5 years ago

Scott, I am doing tests with your server variables and have not run into issues yet. I am replacing one at a time. I will have to try combinations. However, I did notice that you have a discrepancy in packet sizes:

packetSize="65536" on Tomcat vs <PacketSize>65530</PacketSize> on Boncode.

This will lead to issues. If you could set both to the same value, preferably 65530 and retest.

Best, Bilal

srconklin commented 5 years ago

Hi Blial-

I reset all the settings to their defaults and the issue seems to have solved itself over the last week....so I thought. After 8 days of nothing, one of the users of a back-end tool where he is the single user (closed system), it appears the issue has returned: See what he just reported to me:

Any thoughts on how to troubleshoot this?

tomcat error

So, today I was loading in orders and I loaded about 6 just fine and then on the 7th order I got the Tomcat error again. It then happened on the 8th and final order I loaded. See attached screenshot. Once the error appears it follows me through all of the screens until the end and it also prints on all pages I choose to print.

srconklin commented 5 years ago

Maybe I should enable logging next? https://dev.lucee.org/t/generic-connector-communication-error-with-boncode-ajp/4183

Bilal-S commented 5 years ago

Scott, thanks for showing this. I am assuming you have a few different files/components that you are loading to create this view and one of them is throwing this error? I am also assuming the user is using defaults values?

A generic error may be a valid state depending on the circumstances. Need to get more context information, i.e. logs. So, yes, please enable logging. Start with level 1. Also enable windows event logging please. http://www.boncode.net/connector/webdocs/Tomcat_Connector.htm#_Toc520189761

Best, Bilal

srconklin commented 5 years ago

Bilal- Thanks.

This is a very old legacy CF application just ported over from Win 2003 running CF 9 (Jrun) There are no components being loaded as far as I know and for the most part this is very straigthforward CFML ( I am not the original developer and some of this code is near 15 yrs old)

This user is indeed using the default values. there ar e only two sites on the whole machine: neither of which have a lot of traffic.

1) https://www.poolandspa.com/ - which is showing no more signs of having the error in the last 8 days since going back to default settings.

2) sys.poolandspa.com - which as you saw just happened today after the user "processed" some online orders.

I will enable logging now at level 1 and let you know what we find.

Bilal-S commented 5 years ago

Thanks for clarifying. Don't forget to enable windows event logging since many of the connection errors are at low level (TCP) and will never reach the AJP logic layer.

srconklin commented 5 years ago

Hi Bilal-

Took awhile, but the user of the admin backend piece finally experienced the error again. It looks like the only thing that was logged is below: not sure if this is helpful:

2019-02-07 00:00:02 BonCode AJP Connenctor version 1.0.41 2019-02-07 07:40:27 failed Server Connection was aborted 2019-02-07 07:40:27 1.0.41 ERROR TCP Client level -- Server/Port:localhost/8009 Connection between Tomcat and IIS experienced error. If you restarted Tomcat this is expected. (2)failed:Server Connection was aborted at BonCodeAJP13.BonCodeAJP13ServerConnection.ConnectionError(String message, String messageType) at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat() at BonCodeAJP13.BonCodeAJP13ServerConnection.HandleConnection() at BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend)

Bilal-S commented 5 years ago

Scott, unfortunately not much info here except that the tomcat connection was shutdown and that is why you see the error. Have you enabled and looked into windows event log, it should have more info on the nature of the shutdown.

There are normally only two main reasons for this error: a) Tomcat is busy or overloaded and stops responding or accepting connections b) We have stale connections that have not yet been cleaned up. This can only occur in high volume with connection pools on both sides that need to be synched. This is not enabled by default.

Since you are using default values all connections are specifically created and torn down after each request. Thus, I am not certain why you see this error.

What do you see when you run the connector command for settings http://localhost/a.jsp?BonCodeConfigList=true

Best. Bilal

srconklin commented 5 years ago

Hi Bilal-

Here is the results of the BClist:

BonCode Active Configuration:

BONCODEAJP13_ADOBE_SUPPORT = False BONCODEAJP13_PORT = 8009 BONCODEAJP13_SERVER = localhost BONCODEAJP13_AUTOFLUSHDETECTION_TICKS = 0 BONCODEAJP13_AUTOFLUSHDETECTION_BYTES = 0 MAX_BONCODEAJP13_CONCURRENT_CONNECTIONS = 0 BONCODEAJP13_LOG_LEVEL = 1 BONCODEAJP13_LOG_DIR = c:\temp BONCODEAJP13_LOG_FILE = BonCodeAJP13Connection BONCODEAJP13_ENABLE_REMOTE_MANAGER = True BONCODEAJP13_AUTOCOMPRESS = False BONCODEAJP13_TEXT_MARK = System.String[] BONCODEAJP13_MANAGER_URLS = System.String[] BONCODEAJP13_MANAGER_FLEXURLS = System.String[] BONCODEAJP13_HEADER_SUPPORT = True BonCodeAjp13_DocRoot = BonCodeAjp13_PhysicalFilePath = BONCODEAJP13_BLACKLIST_HEADERS = URL,SERVER_SOFTWARE,SERVER_NAME,SERVER_PROTOCOL BONCODEAJP13_WHITELIST_HEADERS = BONCODEAJP13_FLUSH_TIMEOUT = 30 BONCODEAJP13_SERVER_READ_TIMEOUT = 120000 BONCODEAJP13_SERVER_WRITE_TIMEOUT = 0 BONCODEAJP13_FORCE_SECURE_SESSION = False BONCODEAJP13_REMOTEADDR_FROM = BONCODEAJP13_ALLOW_EMTPY_HEADERS = False BONCODEAJP13_PATHINFO_HEADER = xajp-path-info BONCODEAJP13_ENABLE_HTTPSTATUSCODES = True BONCODEAJP13_TOMCAT_DOWN_URL = BONCODEAJP13_TOMCAT_STREAM_ERRORMSG = BONCODEAJP13_TOMCAT_TCPCLIENT_ERRORMSG = BONCODEAJP13_PATH_PREFIX = MAX_BONCODEAJP13_PACKET_LENGTH = 8192 MAX_BONCODEAJP13_USERDATA_LENGTH = 8186 BONCODEAJP13_ENABLE_CLIENTFINGERPRINT = False BONCODEAJP13_FINGERPRINTHEADERS = REMOTE_ADDR,HTTP_ACCEPT,HTTP_ACCEPT_CHARSET,HTTP_ACCEPT_ENCODING,HTTP_ACCEPT_LANGUAGE,HTTP_USER_AGENT,UA_CPU,REMOTE_HOST BONCODEAJP13_SKIP_IISCUSTOMERRORS = False BONCODEAJP13_LOG_IPFILTER = BONCODEAJP13_REQUEST_SECRET = BONCODEAJP13_FORCE_GC = False BONCODE_MODCFML_SECRET = 99f00ff4667bd09d636438c007090a696ebec2c1a14ca9fc004c6771faf1b02d BONCODE_DOCROOT_OVERRIDE =

srconklin commented 5 years ago

Hi Bilal-

There is nothing in the Windows event viewer except some information notices an one error that is completely unrelated to to anything.

srconklin commented 5 years ago

Hi Bilal-

just about an hour ago while we were talking about all this. The server finally went down. One could not access the site in a browser. It just spun and spun and never connected.

I thought it might be Lucee but cycling it did nothing. I had to cycle IIS to bring the server back and it took quite some tim e to stop IIS. A new error message was dumped into the log BC file file. Does this help you see anything?

2019-02-08 00:00:00 BonCode AJP Connenctor version 1.0.41 2019-02-08 12:51:12 1.0.41 ERROR Stream reading problem (2)(2), we stopped waiting on Tomcat response. You may have shutdown Tomcat unexpectedly The source was not found, but some or all event logs could not be searched. Inaccessible logs: Security. at System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean readOnly, Boolean wantToCreate) at System.Diagnostics.EventLog.SourceExists(String source, String machineName, Boolean wantToCreate) at System.Diagnostics.EventLog.SourceExists(String source) at BonCodeAJP13.BonCodeAJP13Logger.RecordSysEvent(String message, EventLogEntryType eType) at BonCodeAJP13.BonCodeAJP13Logger.LogPacket(BonCodeAJP13Packet packet, Boolean logAllways, Int32 minLogLevel) at BonCodeAJP13.BonCodeAJP13ServerConnection.AnalyzePackage(Boolean& delayWriteIndicator, Byte[] receiveBuffer, Boolean skipFlush, Int32 iOffset) at BonCodeAJP13.BonCodeAJP13ServerConnection.AnalyzePackage(Byte[] receiveBuffer, Boolean skipFlush, Int32 iOffset) at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat() 2019-02-08 12:51:12 failed Server Connection was aborted 2019-02-08 12:51:12 1.0.41 ERROR TCP Client level -- Server/Port:localhost/8009 Connection between Tomcat and IIS experienced error. If you restarted Tomcat this is expected. (2)failed:Server Connection was aborted at BonCodeAJP13.BonCodeAJP13ServerConnection.ConnectionError(String message, String messageType) at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat() at BonCodeAJP13.BonCodeAJP13ServerConnection.HandleConnection() at BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend)

srconklin commented 5 years ago

There are normally only two main reasons for this error: a) Tomcat is busy or overloaded and stops responding or accepting connections b) We have stale connections that have not yet been cleaned up. This can only occur in high volume with connection pools on both sides that need to be synched. This is not enabled by default.

Are you saying since we are using the defaults and thus are creating and tearing down the connections on each request that our problem can only be a) above?

What do we do about it if that is indeed the issue?

Bilal-S commented 5 years ago

This seems to indicate that as we are waiting on a response from Tomcat (and encountered an error during second read cycle from TCP stream since the first one was not fast enough that is what the second (2) means, we experienced an error. We could not log the error because The source was not found, but some or all event logs could not be searched. This means we attempted to write it to the Windows event log but did not have permissions.

Please enable windows event logging (see previous messages on how to do it).

Is there something that would intercept the network streams inside machine? Or something that slows down TCP/IP? We only have to go into a wait cycle if TCP/IP is lagging.

Bilal-S commented 5 years ago

There are normally only two main reasons for this error: a) Tomcat is busy or overloaded and stops responding or accepting connections b) We have stale connections that have not yet been cleaned up. This can only occur in high volume with connection pools on both sides that need to be synched. This is not enabled by default.

Are you saying since we are using the defaults and thus are creating and tearing down the connections on each request that our problem can only be a) above?

What do we do about it if that is indeed the issue?

If Tomcat is not busy, we are not following usual patterns. We either have a very complex bug or something is stopping all bytes from being transmitted.

srconklin commented 5 years ago

Bilal-

Thanks. I must have missed the instructions on enabling windows event logging for BC. sorry about that. I have done so now. I guess we will wait and see the next time the error occurs and then I will report the events.

Thanks for your help

srconklin commented 5 years ago

Hi Bilal-

Ok, it happened again and we have windows event logging activated:

I see 4 repeated entries for event 417 right after the other:

Error flushing data: Index and count must refer to a location within the buffer. Parameter name: bytes at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count) at BonCodeAJP13.BonCodeAJP13Packet.GetString(Byte[] data, String& value, Int32 pos) at BonCodeAJP13.TomcatPackets.TomcatSendHeaders.GetHeaders() at BonCodeIIS.BonCodeCallHandler.PrintFlush(BonCodeAJP13PacketCollection flushCollection)

Bilal-S commented 5 years ago

That would mean that we cannot read one or more of the Tomcat Headers that are being passed. This may be a symptom of not being able read the TCP stream. Does this happen on all machines or just on the specific one?

I will give you a special build that logs out the actual byte values when this happens to see whether this is a protocol error or something else.

srconklin commented 5 years ago

Thank you, I await the build and instructions for installing..

This happens just on this machine. In fact, while I can't be certain... I think it is only happening one of the two applications running on this server. No reports of this happening on the public facing eCommerce site. Seems to be only happening on the back office admin site for order processing.

Bilal-S commented 5 years ago

Scott, please replace the .dll files in install package with the ones contained in this zip file and reinstall connector. BonCodeAJP13.zip

srconklin commented 5 years ago

Hi bilal- How do I replace these dlls in the install package?

Bilal-S commented 5 years ago

You can download the install package from github (https://github.com/Bilal-S/iis2tomcat/releases/download/1.0.41/AJP13_v1041.zip) Then unblock and unzip it. You will find the same library names in the unzipped content. Simply copy the dll files that are in the special build into it, replacing the existing content.

You should then use this new combination to install on the machine that shows symptoms.

Hope this helps, Bilal

srconklin commented 5 years ago

Bilal- Thanks. I have installed this with the new dlls placed into the unpacked installation dir. I assume the next time we get the error the Windows Event Viewer will have more verbose error information?

Bilal-S commented 5 years ago

Yes. It should log the raw tcp packet data

On Thu, Feb 14, 2019, 19:40 Scott Conklin <notifications@github.com wrote:

Bilal- Thanks. I have installed this with the new dlls placed into the unpacked installation dir. I assume the next time we get the error the Windows Event Viewer will have more verbose error information?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/Bilal-S/iis2tomcat/issues/66#issuecomment-463861664, or mute the thread https://github.com/notifications/unsubscribe-auth/AAo05ahDFjWlRjmdZ1311OHu5dpjaqhAks5vNgISgaJpZM4U9Y4i .

srconklin commented 5 years ago

Hi bilal- So the user has gotten quite a few errors already today. The Event Viewer reports 2 events: 1) a warning 2) then the actual error that I have posted above... "error flushing data.." etc...

The Warning Level entries I believe contain the tcp packet data that you are looking for. let me know if I am wrong. I think they are all slightly different so I am giving you all 4. Here they are in order from today

A) Error reading headers: 04-00-C8-00-03-32-30-30-00-00-06-A0-07-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-4C-56-3D-31-35-35-30-35-31-30-30-31-36-33-38-32-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-33-36-20-55-54-43-00-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-54-43-3D-31-35-34-37-32-32-36-31-39-30-32-37-34-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-33-36-20-55-54-43-00-A0-07-00-46-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-48-43-3D-35-38-34-38-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-33-36-20-55-54-43-00-A0-01-00-17-74-65-78-74-2F-68-74-6D-6C-3B-63-68-61-72-73-65-74-3D-55-54-46-2D-38-00-A0-03-00-04-36-31-37-34-00

B) Error reading headers: 04-00-C8-00-03-32-30-30-00-00-06-A0-07-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-4C-56-3D-31-35-35-30-35-31-30-30-32-31-36-33-39-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-34-31-20-55-54-43-00-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-54-43-3D-31-35-34-37-32-32-36-31-39-30-32-37-34-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-34-31-20-55-54-43-00-A0-07-00-46-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-48-43-3D-35-38-35-30-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-34-31-20-55-54-43-00-A0-01-00-17-74-65-78-74-2F-68-74-6D-6C-3B-63-68-61-72-73-65-74-3D-55-54-46-2D-38-00-A0-03-00-04-37-31-39-38-00

C) Error reading headers: 04-00-C8-00-03-32-30-30-00-00-06-A0-07-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-4C-56-3D-31-35-35-30-35-31-30-30-32-35-39-38-37-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-34-36-20-55-54-43-00-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-54-43-3D-31-35-34-37-32-32-36-31-39-30-32-37-34-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-34-36-20-55-54-43-00-A0-07-00-46-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-48-43-3D-35-38-35-32-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-33-3A-34-36-20-55-54-43-00-A0-01-00-17-74-65-78-74-2F-68-74-6D-6C-3B-63-68-61-72-73-65-74-3D-55-54-46-2D-38-00-A0-03-00-05-31-36-36-37-31-00

D) Error reading headers: 04-00-C8-00-03-32-30-30-00-00-06-A0-07-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-4C-56-3D-31-35-35-30-35-31-30-30-34-31-36-34-35-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-34-3A-30-31-20-55-54-43-00-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-54-43-3D-31-35-34-37-32-32-36-31-39-30-32-37-34-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-34-3A-30-31-20-55-54-43-00-A0-07-00-46-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-48-43-3D-35-38-35-34-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-34-3A-30-31-20-55-54-43-00-A0-01-00-17-74-65-78-74-2F-68-74-6D-6C-3B-63-68-61-72-73-65-74-3D-55-54-46-2D-38-00-A0-03-00-04-33-36-39-33-00

srconklin commented 5 years ago

In addition, I am seeing some other entries that seemed to have happen very early this morning before the users even got to the office. Not sure if these are meaningful:

A) Error during log write : The wait completed due to an abandoned mutex.

B) Connection error 2: Connection between Tomcat and IIS experienced error. If you restarted Tomcat this is expected. (1) at BonCodeAJP13.BonCodeAJP13ServerConnection.ConnectionError() at BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend) at BonCodeAJP13.BonCodeAJP13ServerConnection.BeginConnection() at BonCodeIIS.BonCodeCallHandler.ProcessRequest(HttpContext context)

C) ProcessRequest: Thread was being aborted. at BonCodeIIS.BonCodeCallHandler.ProcessRequest(HttpContext context)

These 3 entries repeat themselves over and over almost every minute between 4:00AM and 6:00 AM today. so there are quite a few

srconklin commented 5 years ago

Bilal- Actually there are a lot more TCP data packet dump entries than the 4 I listed above. Let me know if you need them all. one of them is quite large:

Error reading headers: 04-00-C8-00-03-32-30-30-00-00-06-A0-07-1F-70-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-3D-25-37-42-25-32-32-43-4F-4D-50-41-4E-59-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-4F-55-52-43-45-44-41-54-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-41-44-44-52-45-53-53-32-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-49-4E-46-4F-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-4F-52-44-45-52-56-49-41-25-32-32-25-33-41-25-32-32-49-6E-74-65-72-6E-65-74-25-32-32-25-32-43-25-32-32-50-41-59-4D-45-4E-54-54-59-50-45-25-32-32-25-33-41-25-32-32-43-72-65-64-69-74-25-32-30-43-61-72-64-25-32-32-25-32-43-25-32-32-53-48-49-50-4C-41-53-54-4E-41-4D-45-25-32-32-25-33-41-25-32-32-55-45-43-4B-45-52-25-32-32-25-32-43-25-32-32-4F-4D-49-4E-56-4F-49-43-45-25-32-32-25-33-41-25-32-32-31-39-2D-32-31-37-32-36-31-25-32-32-25-32-43-25-32-32-43-43-41-56-53-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-43-4F-4D-4D-45-4E-54-53-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-48-41-56-45-4C-49-53-54-25-32-32-25-33-41-25-32-32-53-70-61-25-32-32-25-32-43-25-32-32-49-44-4E-4F-25-32-32-25-33-41-32-31-37-32-34-37-25-32-43-25-32-32-43-48-45-43-4B-4E-4F-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-43-4F-55-4E-54-52-59-25-32-32-25-33-41-25-32-32-55-6E-69-74-65-64-25-32-30-53-74-61-74-65-73-25-32-32-25-32-43-25-32-32-43-52-45-41-54-45-44-41-54-45-25-32-32-25-33-41-63-72-65-61-74-65-44-61-74-65-54-69-6D-65-25-32-38-32-30-31-39-25-32-43-32-25-32-43-31-38-25-32-43-30-25-32-43-30-25-32-43-30-25-32-43-30-25-32-43-25-32-32-41-6D-65-72-69-63-61-25-32-46-43-68-69-63-61-67-6F-25-32-32-25-32-39-25-32-43-25-32-32-53-48-49-50-52-45-53-49-44-45-4E-54-49-41-4C-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-41-4C-54-50-48-4F-4E-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-4F-55-52-43-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-45-58-50-44-41-54-45-25-32-32-25-33-41-25-32-32-31-31-25-32-46-32-30-32-31-25-32-32-25-32-43-25-32-32-43-41-52-44-48-4F-4C-44-45-52-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-4C-41-53-54-4E-41-4D-45-25-32-32-25-33-41-25-32-32-55-45-43-4B-45-52-25-32-32-25-32-43-25-32-32-43-43-41-55-54-48-4E-4F-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-48-4F-4D-45-50-48-4F-4E-45-25-32-32-25-33-41-25-32-32-32-30-37-2D-36-38-30-2D-30-35-37-38-25-32-32-25-32-43-25-32-32-43-56-56-32-25-32-32-25-33-41-25-32-32-37-33-32-25-32-32-25-32-43-25-32-32-4F-52-44-45-52-54-45-58-54-46-49-4C-45-25-32-32-25-33-41-25-32-32-32-30-31-39-30-32-31-36-2D-31-30-34-32-32-32-2D-2E-74-78-74-25-32-32-25-32-43-25-32-32-41-4C-54-50-48-4F-4E-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-54-4F-41-44-44-52-45-53-53-25-32-32-25-33-41-25-32-32-34-31-31-25-32-30-49-4E-53-50-45-43-54-49-4F-4E-25-32-30-53-45-52-56-49-43-45-53-25-32-43-25-32-30-4D-41-49-4E-54-45-4E-41-4E-43-45-25-32-30-41-4E-44-25-32-30-48-4F-4D-45-25-32-30-49-4D-50-52-4F-56-45-4D-45-25-30-44-25-30-41-53-54-41-4E-4C-45-59-25-32-30-52-4F-53-53-25-32-30-4D-41-43-4C-41-52-45-4E-25-32-30-4A-52-25-30-44-25-30-41-31-30-37-25-32-30-5A-45-4E-49-54-48-25-32-30-54-52-41-49-4C-25-30-44-25-30-41-35-35-39-37-25-30-44-25-30-41-45-4C-4C-49-4A-41-59-25-32-43-25-32-30-47-41-25-32-30-33-30-35-34-30-25-30-44-25-30-41-50-68-6F-6E-65-25-33-41-25-32-30-37-37-30-2D-38-39-31-2D-31-39-38-39-25-32-30-25-32-30-25-32-30-25-32-30-46-61-78-25-33-41-25-32-30-25-30-44-25-30-41-25-32-32-25-32-43-25-32-32-4F-52-44-45-52-54-45-58-54-25-32-32-25-33-41-25-32-32-4F-72-64-65-72-54-79-70-65-25-32-30-25-32-30-25-32-30-25-32-30-43-72-65-64-69-74-25-32-30-43-61-72-64-25-30-44-25-30-41-45-6D-61-69-6C-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-6A-75-65-63-6B-65-72-25-34-30-72-6F-61-64-72-75-6E-6E-65-72-2E-63-6F-6D-25-30-44-25-30-41-43-6F-6D-70-61-6E-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-46-69-72-73-74-4E-61-6D-65-25-32-30-25-32-30-25-32-30-25-32-30-4A-65-66-66-72-65-79-25-30-44-25-30-41-4C-61-73-74-4E-61-6D-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-55-65-63-6B-65-72-25-30-44-25-30-41-41-64-64-72-65-73-73-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-33-30-37-25-32-30-52-6F-63-6B-79-25-32-30-53-68-6F-72-65-25-32-30-4C-61-6E-65-25-30-44-25-30-41-41-70-74-53-75-69-74-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-43-69-74-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-4F-61-6B-6C-61-6E-64-25-30-44-25-30-41-53-74-61-74-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-4D-45-25-30-44-25-30-41-5A-69-70-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-30-34-39-36-33-25-30-44-25-30-41-50-72-6F-76-69-6E-63-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-43-6F-75-6E-74-72-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-55-6E-69-74-65-64-25-32-30-53-74-61-74-65-73-25-30-44-25-30-41-50-68-6F-6E-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-32-30-37-2D-36-38-30-2D-30-35-37-38-25-30-44-25-30-41-46-61-78-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-53-68-69-70-43-6F-6D-70-61-6E-79-25-32-30-25-32-30-25-30-44-25-30-41-53-68-69-70-46-69-72-73-74-4E-61-6D-25-32-30-4A-65-66-66-72-65-79-25-30-44-25-30-41-53-68-69-70-4C-61-73-74-4E-61-6D-65-25-32-30-55-65-63-6B-65-72-25-30-44-25-30-41-53-68-69-70-41-64-64-72-65-73-73-25-32-30-25-32-30-33-30-37-25-32-30-52-6F-63-6B-79-25-32-30-53-68-6F-72-65-25-32-30-4C-61-6E-65-25-30-44-25-30-41-53-68-69-70-41-70-74-53-75-69-74-65-25-32-30-25-30-44-25-30-41-53-68-69-70-43-69-74-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-4F-61-6B-6C-61-6E-64-25-30-44-25-30-41-53-68-69-70-53-74-61-74-65-25-32-30-25-32-30-25-32-30-25-32-30-4D-45-25-30-44-25-30-41-53-68-69-70-5A-69-70-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-30-34-39-36-33-25-30-44-25-30-41-53-68-69-70-50-72-6F-76-69-6E-63-65-25-32-30-25-30-44-25-30-41-53-68-69-70-43-6F-75-6E-74-72-79-25-32-30-25-32-30-55-6E-69-74-65-64-25-32-30-53-74-61-74-65-73-25-30-44-25-30-41-53-68-69-70-50-68-6F-6E-65-25-32-30-25-32-30-25-32-30-25-32-30-32-30-37-2D-36-38-30-2D-30-35-37-38-25-30-44-25-30-41-53-68-69-70-46-61-78-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-49-74-65-6D-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-47-4C-42-2D-35-30-2D-36-37-31-35-25-30-44-25-30-41-44-65-73-63-72-69-70-74-69-6F-6E-25-32-30-25-32-30-52-65-6E-64-65-7A-76-6F-75-73-25-32-30-53-61-6E-69-2D-53-70-61-25-32-30-43-68-6C-6F-72-69-6E-61-74-69-6E-67-25-32-30-47-72-61-6E-75-6C-65-73-25-32-30-53-70-61-25-32-30-53-68-6F-63-6B-25-32-30-2D-31-2E-37-35-25-32-30-6C-62-25-32-30-2D-25-32-30-47-4C-42-2D-35-30-2D-36-37-31-35-25-30-44-25-30-41-4F-70-74-69-6F-6E-73-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-51-74-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-31-25-30-44-25-30-41-50-72-69-63-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-31-36-2E-39-39-25-30-44-25-30-41-4C-69-6E-65-53-68-69-70-70-69-6E-67-25-32-30-30-25-30-44-25-30-41-49-74-65-6D-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-4C-53-54-2D-35-30-2D-39-37-39-25-30-44-25-30-41-44-65-73-63-72-69-70-74-69-6F-6E-25-32-30-25-32-30-4C-65-69-73-75-72-65-25-32-30-54-69-6D-65-25-32-30-49-6E-73-74-61-6E-74-25-32-30-43-61-72-74-72-69-64-67-65-25-32-30-43-6C-65-61-6E-25-32-30-2D-25-32-30-70-69-6E-74-25-32-30-2D-25-32-30-4C-53-54-2D-35-30-2D-39-37-39-25-30-44-25-30-41-4F-70-74-69-6F-6E-73-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-51-74-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-32-25-30-44-25-30-41-50-72-69-63-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-37-2E-39-39-25-30-44-25-30-41-4C-69-6E-65-53-68-69-70-70-69-6E-67-25-32-30-30-25-30-44-25-30-41-49-74-65-6D-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-4C-53-54-2D-35-30-2D-38-38-31-25-30-44-25-30-41-44-65-73-63-72-69-70-74-69-6F-6E-25-32-30-25-32-30-4C-65-69-73-75-72-65-25-32-30-54-69-6D-65-25-32-30-46-6F-61-6D-25-32-30-44-6F-77-6E-25-32-30-2D-25-32-30-53-70-61-25-32-30-57-61-74-65-72-25-32-30-44-65-66-6F-61-6D-65-72-25-32-30-2D-25-32-30-50-69-6E-74-25-32-30-2D-25-32-30-4C-53-54-2D-35-30-2D-38-38-31-25-30-44-25-30-41-4F-70-74-69-6F-6E-73-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-51-74-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-31-25-30-44-25-30-41-50-72-69-63-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-38-2E-39-39-25-30-44-25-30-41-4C-69-6E-65-53-68-69-70-70-69-6E-67-25-32-30-30-25-30-44-25-30-41-49-74-65-6D-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-4C-53-54-2D-35-30-2D-38-38-30-25-30-44-25-30-41-44-65-73-63-72-69-70-74-69-6F-6E-25-32-30-25-32-30-4C-65-69-73-75-72-65-25-32-30-54-69-6D-65-25-32-30-53-70-61-25-32-30-4D-65-74-61-6C-25-32-30-47-6F-6E-65-25-32-30-2D-25-32-30-50-69-6E-74-25-32-30-2D-25-32-30-4C-53-54-2D-35-30-2D-38-38-30-25-30-44-25-30-41-4F-70-74-69-6F-6E-73-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-51-74-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-32-25-30-44-25-30-41-50-72-69-63-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-38-2E-39-39-25-30-44-25-30-41-4C-69-6E-65-53-68-69-70-70-69-6E-67-25-32-30-30-25-30-44-25-30-41-49-74-65-6D-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-32-30-36-58-25-30-44-25-30-41-44-65-73-63-72-69-70-74-69-6F-6E-25-32-30-25-32-30-49-6E-53-50-41-72-61-74-69-6F-6E-25-32-30-52-6F-6D-61-6E-63-65-25-32-30-2D-25-32-30-32-30-36-58-25-30-44-25-30-41-4F-70-74-69-6F-6E-73-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-30-44-25-30-41-51-74-79-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-31-25-30-44-25-30-41-50-72-69-63-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-37-2E-39-39-25-30-44-25-30-41-4C-69-6E-65-53-68-69-70-70-69-6E-67-25-32-30-30-25-30-44-25-30-41-53-68-69-70-70-69-6E-67-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-39-2E-39-35-25-30-44-25-30-41-53-68-69-70-4D-65-74-68-6F-64-25-32-30-25-32-30-25-32-30-55-50-53-25-32-30-47-72-6F-75-6E-64-25-30-44-25-30-41-53-61-6C-65-73-54-61-78-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-30-25-30-44-25-30-41-54-6F-74-61-6C-41-6D-74-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-34-37-37-2E-38-38-25-30-44-25-30-41-43-61-72-64-54-79-70-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-44-69-73-63-6F-76-65-72-25-30-44-25-30-41-43-61-72-64-4E-6F-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-36-30-31-31-2D-32-30-38-38-2D-37-39-34-39-2D-30-38-35-35-25-30-44-25-30-41-45-78-70-44-61-74-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-31-31-25-32-46-32-30-32-31-25-30-44-25-30-41-43-56-56-43-6F-64-65-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-37-33-32-25-30-44-25-30-41-4F-72-64-65-72-4E-6F-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-25-32-30-50-53-2D-39-30-30-36-2D-34-38-37-39-25-30-44-25-30-41-41-64-64-69-74-69-6F-6E-61-6C-25-32-30-25-32-30-25-32-30-49-25-32-30-68-61-76-65-25-32-30-61-25-30-44-25-30-41-53-70-61-25-30-44-25-30-41-25-30-44-25-30-41-57-68-61-74-25-32-30-62-72-6F-75-67-68-74-25-32-30-79-6F-75-25-32-30-74-6F-25-32-30-6F-75-72-25-32-30-77-65-62-25-32-30-73-69-74-65-25-33-46-25-30-44-25-30-41-25-30-44-25-30-41-25-30-44-25-30-41-50-6C-65-61-73-65-25-32-30-65-6E-74-65-72-25-32-30-61-6E-79-25-32-30-63-6F-6D-6D-65-6E-74-73-25-32-30-61-62-6F-75-74-25-32-30-79-6F-75-72-25-32-30-6F-72-64-65-72-2E-25-32-30-50-6C-65-61-73-65-25-32-30-44-4F-25-32-30-4E-4F-54-25-32-30-61-73-6B-25-32-30-54-65-63-68-6E-69-63-61-6C-25-32-30-51-75-65-73-74-69-6F-6E-73-25-32-30-69-6E-25-32-30-74-68-69-73-25-32-30-73-70-61-63-65-2E-25-32-30-45-2D-6D-61-69-6C-25-32-30-43-75-73-74-6F-6D-65-72-25-32-30-53-65-72-76-69-63-65-25-32-30-64-69-72-65-63-74-6C-79-25-32-30-66-6F-72-25-32-30-61-6C-6C-25-32-30-54-65-63-68-6E-69-63-61-6C-25-32-30-51-75-65-73-74-69-6F-6E-73-2E-25-30-44-25-30-41-25-30-44-25-30-41-25-30-44-25-30-41-25-30-44-25-30-41-25-32-32-25-32-43-25-32-32-49-4E-56-4F-49-43-45-25-32-32-25-33-41-25-32-32-31-39-2D-32-31-37-32-36-31-25-32-32-25-32-43-25-32-32-53-48-49-50-53-54-41-54-45-25-32-32-25-33-41-25-32-32-4D-45-25-32-32-25-32-43-25-32-32-53-55-42-54-4F-54-41-4C-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-42-49-4C-4C-54-4F-41-44-44-52-45-53-53-25-32-32-25-33-41-25-32-32-34-31-31-25-32-30-49-4E-53-50-45-43-54-49-4F-4E-25-32-30-53-45-52-56-49-43-45-53-25-32-43-25-32-30-4D-41-49-4E-54-45-4E-41-4E-43-45-25-32-30-41-4E-44-25-32-30-48-4F-4D-45-25-32-30-49-4D-50-52-4F-56-45-4D-45-25-30-44-25-30-41-53-54-41-4E-4C-45-59-25-32-30-52-4F-53-53-25-32-30-4D-41-43-4C-41-52-45-4E-25-32-30-4A-52-25-30-44-25-30-41-31-30-37-25-32-30-5A-45-4E-49-54-48-25-32-30-54-52-41-49-4C-25-30-44-25-30-41-35-35-39-37-25-30-44-25-30-41-45-4C-4C-49-4A-41-59-25-32-43-25-32-30-47-41-25-32-30-33-30-35-34-30-25-30-44-25-30-41-50-68-6F-6E-65-25-33-41-25-32-30-37-37-30-2D-38-39-31-2D-31-39-38-39-25-32-30-25-32-30-25-32-30-25-32-30-46-61-78-25-33-41-25-32-30-25-30-44-25-30-41-25-32-32-25-32-43-25-32-32-4F-52-44-45-52-53-54-41-54-55-53-25-32-32-25-33-41-25-32-32-4F-70-65-6E-25-32-32-25-32-43-25-32-32-43-43-44-41-54-45-54-49-4D-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-48-4F-4C-44-44-41-54-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-41-44-44-52-45-53-53-25-32-32-25-33-41-25-32-32-33-30-37-25-32-30-52-4F-43-4B-59-25-32-30-53-48-4F-52-45-25-32-30-4C-41-4E-45-25-32-32-25-32-43-25-32-32-54-41-58-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-43-43-4E-4F-25-32-32-25-33-41-25-32-32-36-30-31-31-2D-32-30-38-38-2D-37-39-34-39-2D-30-38-35-35-25-32-32-25-32-43-25-32-32-4F-52-44-45-52-4E-4F-25-32-32-25-33-41-25-32-32-50-53-2D-39-30-30-36-2D-34-38-37-39-25-32-32-25-32-43-25-32-32-53-48-49-50-41-44-44-52-45-53-53-33-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-54-41-58-53-54-41-54-45-25-32-32-25-33-41-25-32-32-46-4C-25-32-32-25-32-43-25-32-32-41-44-44-52-45-53-53-33-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-46-41-58-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-50-4F-43-4F-55-4E-54-45-52-25-32-32-25-33-41-36-35-25-32-43-25-32-32-57-4F-52-4B-50-48-4F-4E-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-57-4F-52-4B-50-48-4F-4E-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-4F-52-44-45-52-54-59-50-45-25-32-32-25-33-41-25-32-32-4F-72-64-65-72-25-32-32-25-32-43-25-32-32-53-48-49-50-43-49-54-59-25-32-32-25-33-41-25-32-32-4F-41-4B-4C-41-4E-44-25-32-32-25-32-43-25-32-32-53-48-49-50-50-49-4E-47-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-43-56-56-52-45-53-55-4C-54-25-32-32-25-33-41-25-32-32-43-56-56-25-32-30-4D-41-54-43-48-25-32-32-25-32-43-25-32-32-44-42-4E-41-4D-45-25-32-32-25-33-41-25-32-32-4F-46-53-44-42-25-32-32-25-32-43-25-32-32-4F-4D-49-44-4E-4F-25-32-32-25-33-41-32-31-37-32-36-31-25-32-43-25-32-32-43-41-52-54-49-54-45-4D-53-25-32-32-25-33-41-25-32-32-47-4C-42-2D-35-30-2D-36-37-31-35-25-32-30-25-37-45-52-65-6E-64-65-7A-76-6F-75-73-25-32-30-53-61-6E-69-2D-53-70-61-25-32-30-43-68-6C-6F-72-69-6E-61-74-69-6E-67-25-32-30-47-72-61-6E-75-6C-65-73-25-32-30-53-70-61-25-32-30-53-68-6F-63-6B-25-32-30-2D-31-2E-37-35-25-32-30-6C-62-25-32-30-2D-25-32-30-47-4C-42-2D-35-30-2D-36-37-31-35-25-32-30-25-37-45-25-32-30-25-37-45-31-25-32-30-25-37-45-31-36-2E-39-39-25-32-30-25-37-45-30-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-4C-49-48-54-25-32-30-25-37-45-35-34-38-36-30-36-25-32-30-25-37-45-25-32-30-25-37-43-4C-53-54-2D-35-30-2D-39-37-39-25-32-30-25-37-45-4C-65-69-73-75-72-65-25-32-30-54-69-6D-65-25-32-30-49-6E-73-74-61-6E-74-25-32-30-43-61-72-74-72-69-64-67-65-25-32-30-43-6C-65-61-6E-25-32-30-2D-25-32-30-70-69-6E-74-25-32-30-2D-25-32-30-4C-53-54-2D-35-30-2D-39-37-39-25-32-30-25-37-45-25-32-30-25-37-45-32-25-32-30-25-37-45-37-2E-39-39-25-32-30-25-37-45-30-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-4C-49-48-54-25-32-30-25-37-45-35-35-31-33-35-33-25-32-30-25-37-45-25-32-30-25-37-43-4C-53-54-2D-35-30-2D-38-38-31-25-32-30-25-37-45-4C-65-69-73-75-72-65-25-32-30-54-69-6D-65-25-32-30-46-6F-61-6D-25-32-30-44-6F-77-6E-25-32-30-2D-25-32-30-53-70-61-25-32-30-57-61-74-65-72-25-32-30-44-65-66-6F-61-6D-65-72-25-32-30-2D-25-32-30-50-69-6E-74-25-32-30-2D-25-32-30-4C-53-54-2D-35-30-2D-38-38-31-25-32-30-25-37-45-25-32-30-25-37-45-31-25-32-30-25-37-45-38-2E-39-39-25-32-30-25-37-45-30-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-4C-49-48-54-25-32-30-25-37-45-35-35-31-32-38-39-25-32-30-25-37-45-25-32-30-25-37-43-4C-53-54-2D-35-30-2D-38-38-30-25-32-30-25-37-45-4C-65-69-73-75-72-65-25-32-30-54-69-6D-65-25-32-30-53-70-61-25-32-30-4D-65-74-61-6C-25-32-30-47-6F-6E-65-25-32-30-2D-25-32-30-50-69-6E-74-25-32-30-2D-25-32-30-4C-53-54-2D-35-30-2D-38-38-30-25-32-30-25-37-45-25-32-30-25-37-45-32-25-32-30-25-37-45-38-2E-39-39-25-32-30-25-37-45-30-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-4C-49-48-54-25-32-30-25-37-45-35-34-36-37-39-33-25-32-30-25-37-45-25-32-30-25-37-43-32-30-36-58-25-32-30-25-37-45-49-6E-53-50-41-72-61-74-69-6F-6E-25-32-30-52-6F-6D-61-6E-63-65-25-32-30-2D-25-32-30-32-30-36-58-25-32-30-25-37-45-25-32-30-25-37-45-31-25-32-30-25-37-45-37-2E-39-39-25-32-30-25-37-45-30-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-25-32-30-25-37-45-4C-49-48-54-25-32-30-25-37-45-39-38-31-36-38-36-25-32-30-25-37-45-25-32-32-25-32-43-25-32-32-46-49-52-53-54-4E-41-4D-45-25-32-32-25-33-41-25-32-32-4A-45-46-46-52-45-59-25-32-32-25-32-43-25-32-32-53-54-41-54-45-25-32-32-25-33-41-25-32-32-4D-45-25-32-32-25-32-43-25-32-32-53-48-49-50-41-44-44-52-45-53-53-32-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-54-4F-54-41-4C-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-46-49-52-53-54-4E-41-4D-45-25-32-32-25-33-41-25-32-32-4A-45-46-46-52-45-59-25-32-32-25-32-43-25-32-32-43-49-54-59-25-32-32-25-33-41-25-32-32-4F-41-4B-4C-41-4E-44-25-32-32-25-32-43-25-32-32-43-48-41-52-47-45-44-41-54-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-4C-41-53-54-50-41-47-45-25-32-32-25-33-41-25-32-32-46-69-6E-64-50-72-6F-64-75-63-74-73-2E-63-66-6D-25-32-32-25-32-43-25-32-32-43-4F-4D-50-4C-45-54-45-44-41-54-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-45-4D-41-49-4C-25-32-32-25-33-41-25-32-32-4A-55-45-43-4B-45-52-25-34-30-52-4F-41-44-52-55-4E-4E-45-52-2E-43-4F-4D-25-32-32-25-32-43-25-32-32-4F-41-44-41-54-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-43-48-41-4E-47-45-44-41-54-45-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-4F-4C-44-42-59-25-32-32-25-33-41-25-32-32-77-65-62-73-69-74-65-25-32-32-25-32-43-25-32-32-43-4F-55-4E-54-52-59-25-32-32-25-33-41-25-32-32-55-6E-69-74-65-64-25-32-30-53-74-61-74-65-73-25-32-32-25-32-43-25-32-32-5A-49-50-25-32-32-25-33-41-25-32-32-30-34-39-36-33-25-32-32-25-32-43-25-32-32-53-48-49-50-5A-49-50-25-32-32-25-33-41-25-32-32-30-34-39-36-33-25-32-32-25-32-43-25-32-32-50-41-59-49-4E-46-4F-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-46-41-58-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-4F-55-52-43-45-55-52-4C-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-48-4F-4D-45-50-48-4F-4E-45-25-32-32-25-33-41-25-32-32-32-30-37-2D-36-38-30-2D-30-35-37-38-25-32-32-25-32-43-25-32-32-4F-46-46-49-43-45-43-4F-4D-4D-45-4E-54-53-25-32-32-25-33-41-25-32-32-49-25-32-30-68-61-76-65-25-32-30-61-25-30-44-25-30-41-53-70-61-25-30-44-25-30-41-25-30-44-25-30-41-57-68-61-74-25-32-30-62-72-6F-75-67-68-74-25-32-30-79-6F-75-25-32-30-74-6F-25-32-30-6F-75-72-25-32-30-77-65-62-25-32-30-73-69-74-65-25-33-46-25-30-44-25-30-41-25-30-44-25-30-41-25-30-44-25-30-41-50-6C-65-61-73-65-25-32-30-65-6E-74-65-72-25-32-30-61-6E-79-25-32-30-63-6F-6D-6D-65-6E-74-73-25-32-30-61-62-6F-75-74-25-32-30-79-6F-75-72-25-32-30-6F-72-64-65-72-2E-25-32-30-50-6C-65-61-73-65-25-32-30-44-4F-25-32-30-4E-4F-54-25-32-30-61-73-6B-25-32-30-54-65-63-68-6E-69-63-61-6C-25-32-30-51-75-65-73-74-69-6F-6E-73-25-32-30-69-6E-25-32-30-74-68-69-73-25-32-30-73-70-61-63-65-2E-25-32-30-45-2D-6D-61-69-6C-25-32-30-43-75-73-74-6F-6D-65-72-25-32-30-53-65-72-76-69-63-65-25-32-30-64-69-72-65-63-74-6C-79-25-32-30-66-6F-72-25-32-30-61-6C-6C-25-32-30-54-65-63-68-6E-69-63-61-6C-25-32-30-51-75-65-73-74-69-6F-6E-73-2E-25-32-32-25-32-43-25-32-32-52-45-50-49-44-25-32-32-25-33-41-25-32-32-77-65-62-73-69-74-65-25-32-32-25-32-43-25-32-32-53-48-49-50-43-4F-4D-50-41-4E-59-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-43-43-53-54-41-54-55-53-25-32-32-25-33-41-25-32-32-25-32-32-25-32-43-25-32-32-53-48-49-50-56-49-41-25-32-32-25-33-41-25-32-32-55-50-53-25-32-30-47-72-6F-75-6E-64-25-32-32-25-32-43-25-32-32-4F-52-44-45-52-44-45-54-41-49-4C-53-25-32-32-25-33-41-25-32-32-51-74-79-25-30-39-49-74-65-6D-25-30-39-25-30-39-44-65-73-63-72-69-70-74-69-6F-6E-25-30-39-25-30-39-25-30-39-25-32-30-25-32-30-25-32-30-50-72-69-63-65-25-30-39-25-30-39-25-32-30-25-32-30-25-32-30-54-6F-74-61-6C-25-30-44-25-30-41-31-25-30-39-57-43-39-31-36-2D-34-25-30-39-25-30-39-31-35-2D-59-65-61-72-25-32-30-31-32-25-32-30-58-25-32-30-32-30-25-32-30-4F-76-61-6C-25-32-30-41-72-63-74-69-63-25-32-30-41-72-6D-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-6F-72-25-32-30-43-6F-76-65-72-25-32-30-2D-25-32-30-55-50-43-25-32-30-2D-25-32-30-36-37-32-38-37-35-33-30-32-39-30-31-25-32-30-2D-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-57-43-39-31-36-2D-34-25-30-39-25-30-39-25-30-39-25-30-39-25-32-30-25-32-30-25-32-34-37-30-2E-39-35-25-30-39-25-32-30-25-32-30-25-32-34-37-30-2E-39-35-25-30-44-25-30-41-31-25-30-39-4E-57-31-35-32-25-30-39-25-30-39-34-25-32-37-25-32-30-78-25-32-30-31-35-25-32-37-25-32-30-41-69-72-25-32-30-50-69-6C-6C-6F-77-25-32-30-2D-25-32-30-4E-57-31-35-32-25-32-30-2D-25-32-30-55-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-50-43-25-32-30-2D-25-32-30-37-32-33-38-31-35-30-34-31-35-35-35-25-30-39-25-30-39-25-32-30-25-32-30-25-32-34-33-34-2E-39-35-25-30-39-25-32-30-25-32-30-25-32-34-33-34-2E-39-35-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-53-75-62-54-6F-74-61-6C-25-30-39-25-32-30-25-32-34-31-30-35-2E-39-30-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-53-68-69-70-70-69-6E-67-25-30-39-25-32-30-25-32-30-25-32-30-25-32-34-30-2E-30-30-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-53-61-6C-65-73-25-32-30-54-61-78-25-30-39-25-32-30-25-32-30-25-32-30-25-32-34-37-2E-34-31-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-2D-25-30-44-25-30-41-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-25-30-39-4F-72-64-65-72-25-32-30-54-6F-74-61-6C-25-30-39-25-32-30-25-32-34-31-31-33-2E-33-31-25-30-44-25-30-41-25-32-32-25-32-43-25-32-32-41-44-44-52-45-53-53-25-32-32-25-33-41-25-32-32-33-30-37-25-32-30-52-4F-43-4B-59-25-32-30-53-48-4F-52-45-25-32-30-4C-41-4E-45-25-32-32-25-32-43-25-32-32-54-48-49-52-44-50-41-52-54-59-4F-52-44-45-52-4E-4F-25-32-32-25-33-41-25-32-32-25-32-32-25-37-44-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-35-3A-32-33-20-55-54-43-00-A0-07-00-4F-43-46-5F-43-4C-49-45-4E-54-5F-4C-49-48-54-4F-46-53-5F-4C-56-3D-31-35-35-30-35-31-30-31-32-33-31-33-37-3B-50-61-74-68-3D-2F-3B-45-78-70-69-72-65-73-3D-53-75-6E-2C-20-31-39-2D-4D-61-79-2D-32-30-31-39-20-31-37-3A-31-35-3A-32-33-20-55-54-43-00-A0-07-A0-07-A0-01-00-17-74-65-78-74-2F-68-74-6D-6C-3B-63-68-61-72-73-65-74-3D-55-54-46-2D-38-00-A0-03

srconklin commented 5 years ago

Bilal- we are now getting hundreds and hundreds of these entries.

srconklin commented 5 years ago

abort

Bilal-S commented 5 years ago

This means we cannot do a write log operation when the thread is being killed. This could be normal but unsure without context. Is this showing itself for the user in any way?

Multiple reasons. Are we using Page Redirects? If so this is could be logged when the page redirect header is sent. It should be ignored.

Also to check: Does the IIS worker you are using have permission to write in the log directory? Can you verify that IIS_IUSRS group has permission to read/write in the designated directory for logging and that the directory exists.

Bilal-S commented 5 years ago

Bilal- Actually there are a lot more TCP data packet dump entries than the 4 I listed above. Let me know if you need them all. one of them is quite large:

Error reading headers: 04-00-C

Will look at this later.

srconklin commented 5 years ago

Bilal- I am not sure if the user is seeing anything. Right now the only side effect I can see is that the company admins have reported that now the shopping cart on the public facing site spins and spins https://www.poolandspa.com/cart/showcart.cfm

There could well be cflocations CF redirects in the CFML code. I did not write the application. It is a 20 yr old legacy app that has been running for years on CF 9.

I do see the IIS_IUSRS group has read/write permissions on the the directory that the logging is happening as set in IIS. But this is not the same directory that have the boncode error logging set to? Is this what you mean? I note that the boncode logging seems to be happening.. the directory is full of files like:

BonCodeAJP13Connection_3_VPS-0CE-4B-B4C_20190219.log

srconklin commented 5 years ago

Bilal- Regarding the shopping cart issue above., I reported the issue to the ISP to to get their opinion instead of just cycling IIS myself. This is what they had to say:

Hello,

This does appear to be related to the IIS worker process for the website, as it looks like this was possibly stuck. There was a worker process showing as taking up a large amount of resources and appeared to be stuck (an IIS restart didn't initially take as it was waiting on the worker process to close and it wouldn't). We closed the worker process manually and IIS begin working properly again.

This is most likely related to the IIS > Lucee connector. In most cases this isn't something that continues to occur, however, if you see this happen again please let us know within this ticket (or let us know this is recurring) we'll be able to take a thread dump of Lucee to see which request might be stuck and figure out if there is any possible tuning needed to prevent the issue in the future.

Please let us know if you need further assistance.