How to disable IP blocking for many requests

[NeedHelp4Indy]

Hello all,

I have the following problem: My delphi application blocks an entire IP address range for a certain number of requests. After a manual restart of the process everything works as expected. Is there a way to stop the blocking of IP ranges? Or alternatively to increase the limit? If yes, would be great if someone can share his knowledge.

Kind regards and best thanks!

[rlebeau]

I don't understand what you are describing. Which Indy component(s) are you using? What exactly is being blocked? Is the blockage in your app, or in other apps? Can you please clarify the issue with more detail? Thanks.

[NeedHelp4Indy]

The blockage is in our app. If there are too many HTTP requests in a short time window (from another server), it seems that our app activate a security mechanism. There must be some kind of blacklist in which requests from the subnets are blocked directly (from which the high load originated). If you try to reach the app from another network, everything works as it should. The blacklist can currently be refreshed by restarting the application. However, this is only a workaround in my eyes and I am looking to see if there are potentially other solutions.

Which Indy components are used exactly, I don't know myself at the moment. If absolutely necessary, I would try to list them.

Thanks in advance & Best regards!

[rlebeau]

The blockage is in our app. If there are too many HTTP requests in a short time window (from another server), it seems that our app activate a security mechanism.

What EXACTLY are you seeing? Can you provide a log or screenshot? How short of a window, and how many requests in that window?

There must be some kind of blacklist in which requests from the subnets are blocked directly (from which the high load originated).

If there is, that is not in Indy itself, that would have to be in either an OS firewall or on the network itself.

The blacklist can currently be refreshed by restarting the application. However, this is only a workaround in my eyes and I am looking to see if there are potentially other solutions.

Not without first understanding what is actually happening to begin with.

[NeedHelp4Indy]

At a certain point, HTTP requests are no longer accepted. From the client's point of view, it seems that the server on which the application is running is unreachable.

However, requests from any other client are processed correctly.

I can possibly provide the following information about the components: A CustomHTTPServer is used. I suspect that the error is related to the "SessionList" in some way.

I have just tried different things, so far without success. Do you have any further experience here, is it because of the SessionList or are there other error sources?

[NeedHelp4Indy]

Additional Information: it seems that the problem is only with HTTPS communication, not with HTTP.

[rlebeau]

At a certain point, HTTP requests are no longer accepted. From the client's point of view, it seems that the server on which the application is running is unreachable.

That is likely, if the network (not Indy) is blocking the connections. But again, I would need to see EXACTLY what you are seeing, I can't diagnose your issue with the information you have provided so far. What is the EXACT error you are seeing?

A CustomHTTPServer is used. I suspect that the error is related to the "SessionList" in some way.

I don't see how it could be. The SessionList is merely a cache of TIdHTTPSession objects in memory so they can be shared from one HTTP request to another. It has nothing to do with the server's ability to accept client connections or HTTP requests. When a new session is created, a cookie is sent back to the client, which can then be sent back to the server on subsequent requests. If a request is received without the cookie, or if the specified session has expired, then a new session is created.

Do you have any further experience here, is it because of the SessionList or are there other error sources?

It would have to be caused by an external source, it is not Indy itself blocking the connections.

Additional Information: it seems that the problem is only with HTTPS communication, not with HTTP.

Have you tried using a packet sniffer like Wireshark to look at the TLS handshakes on each connection? Are they being responded to? Are they being rejected?

[NeedHelp4Indy]

If requests are sent from the client, the behavior looks like this:

The behavior is due to the TLS handshake responses being reset immediately. The handshakes are answered, but reset instantly. I was able to detect this with Wireshark.#

There are no error messages on the server and in our applications.

Now if requests are sent from another client, the application responds. From the client, which is misinterpreted in some way as a "DOS attack", all requests are blocked until OUR application (not server!) is restarted. This is the reason I am having such a hard time describing the behavior I am seeing.

Maybe it is also blocked on OS level. Unfortunately, I still could not fix the error. Alternatively, we can try to find a joint appointment, then I could show them the problem live.

[rlebeau]

The behavior is due to the TLS handshake responses being reset immediately. The handshakes are answered, but reset instantly. I was able to detect this with Wireshark.#

I see nothing in that screenshot that implies TLS is involved. The TCP handshake is being rejected immediately, before the TLS handshake has a chance to begin. Most likely, a firewall/router sitting between your client and the server has blacklisted your client IP from making new connections to the server. Again, nothing related to Indy itself.

There are no error messages on the server

Right, because the server is not even seeing the TCP connection to begin with. The network/OS is rejecting the connection, so the server's listening socket doesn't see it and can't accept it.

Now if requests are sent from another client, the application responds. From the client, which is misinterpreted in some way as a "DOS attack", all requests are blocked until OUR application (not server!) is restarted.

Is your client app binding to the same local IP/Port for each new outgoing connection it makes? Or, is it using a random Port each time (as it should be)? In the latter case, since the blocked connections are coming from the same process, that would imply that the blocker is the OS/firewall on the same local machine that the client app is running on, since the PID changes if the app is closed and restarted. A remote blocker would not be able to identify new processes.

Maybe it is also blocked on OS level. Unfortunately, I still could not fix the error. Alternatively, we can try to find a joint appointment, then I could show them the problem live.

You will likely have to either 1) make your client app make fewer requests in a short time, or 2) whitelist your client app in the OS/firewall settings so it stops blocking you in the first place.

[NeedHelp4Indy]

The application listens to one port, this port does not vary. And I think this is the problem. Only one port is closed, now the question is where this happens. Normally the app runs on port 1090, after high load generation it is simply closed (see screen).

Do you definitely close the application and one should configure that on operating system level? I'll analyze if we have any influence on this programmatically.

[rlebeau]

The application listens to one port, this port does not vary.

So, the application you are having trouble with is the server, not a client?

And I think this is the problem.

A server listening on one port is not a problem at all (that is how most servers work).

Do you definitely close the application and one should configure that on operating system level? I'll analyze if we have any influence on this programmatically.

I don't understand what you are asking. But it is absolutely clear that your problem is happening outside of Indy, so I'm inclined to close this ticket.

[NeedHelp4Indy]

Now I have (hopefully) finally identified the problem.

Our application runs into a deadlock. In the Delphi code I must have realized that my predecessors implemented a logic that should normally prevent deadlocks.

Currently I have doubts to adapt the logic so that no deadlocks can occur anymore. For this I probably lack some background knowledge.

Alternatively, do you know of a way to programmatically check for deadlocks and remove them if necessary? Or do you have to take the approach of correcting the logic to prevent deadlocks in the first place?

[NeedHelp4Indy]

The CriticalSection is another potential cause of errors. I am additionally trying to get some help internally, somehow the deadlocks must be able to be prevented.

[rlebeau]

Alternatively, do you know of a way to programmatically check for deadlocks and remove them if necessary? Or do you have to take the approach of correcting the logic to prevent deadlocks in the first place?

You can't simply remove a deadlock once it has occurred. You have to prevent it from happening in the first place.