Robot crawlers locking up imageflow_server

[Nordhbane]

We've been running imageflow_server for over a year now to deliver our product images, but recently the server keep locking up and need to be manually restarted.

It started when a colleague started doing audits on our websites to improve our content delivery, but even when he's not doing audits it looks like all of a sudden we're getting a lot of robots trying to crawl our imageflow-server.

We are getting a few of these: ERROR:iron::iron: Error handling: Request { url: Url { generic_url: "http://cdn.example.com/robots.txt" } method: Get remote_addr: V4(10.13.37.3:53822) local_addr: V4(0.0.0.0:8080) }

Which are then often followed by a lot of these: Error was: NoMatch ERROR:iron::iron: Error handling: Request { url: Url { generic_url: "http://cdn.example.com/favicon.ico" } method: Get remote_addr: V4(10.13.37.3:49696) local_addr: V4(0.0.0.0:8080) }

After a bit these error starts showing up more and more frequently. At first only once every few rows, and then at the end several next to each other: ERROR:iron::response: Error writing response: An established connection was aborted by the software in your host machine. (os error 10053)

I also notice a few calls to images that have been removed or moved, which again seems to suggest there are robots involved using cached image URLs.

Is there any way to prevent robots trying to crawl into the image server and causing this problem? I'll happily send a log with more details in private.

[lilith]

Which reverse proxy are you using in front? IIS/Apache/NGINX?

The current iteration of imageflow_server is not suitable for directly serving content and must be proxied.

On Fri, Mar 19, 2021, 9:02 AM Nordhbane @.***> wrote:

We've been running imageflow_server for over a year now to deliver our product images, but recently the server keep locking up and need to be manually restarted.

It started when a colleague started doing audits on our websites to improve our content delivery, but even when he's not doing audits it looks like all of a sudden we're getting a lot of robots trying to crawl our imageflow-server.

We are getting a few of these: ERROR:iron::iron: Error handling: Request { url: Url { generic_url: "http://cdn.example.com/robots.txt" } method: Get remote_addr: V4(10.13.37.3:53822) local_addr: V4(0.0.0.0:8080) }

Which are then often followed by a lot of these: Error was: NoMatch ERROR:iron::iron: Error handling: Request { url: Url { generic_url: "http://cdn.example.com/favicon.ico" } method: Get remote_addr: V4(10.13.37.3:49696) local_addr: V4(0.0.0.0:8080) }

After a bit these error starts showing up more and more frequently. At first only once every few rows, and then at the end several next to each other: ERROR:iron::response: Error writing response: An established connection was aborted by the software in your host machine. (os error 10053)

I also notice a few calls to images that have been removed or moved, which again seems to suggest there are robots involved using cached image URLs.

Is there any way to prevent robots trying to crawl into the image server and causing this problem? I'll happily send a log with more details in private.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/imazen/imageflow/issues/581, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2LHZFSIO7SSX2S3ZBTATTENRRFANCNFSM4ZO6XFCA .

[Nordhbane]

Cheers for getting back to me so swiftly.

We're using IIS10, on a 2 node server system for load balancing purposes. So we're actually running an instance of the imageflow_server on each node and they go down independently of each other. And as said, it's worked for well over a year with serving content with no real issues.

[lilith]

So we could either add a robots.txt file to serve or you could send the logs to @.*** for root cause analysis.

On Fri, Mar 19, 2021, 4:30 PM Nordhbane @.***> wrote:

Cheers for getting back to me so swiftly.

We're using IIS10, on a 2 node server system for load balancing purposes. So we're actually running an instance of the imageflow_server on each node and they go down independently of each other. And as said, it's worked for well over a year with serving content with no real issues.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/imazen/imageflow/issues/581#issuecomment-803170431, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2LH6KWTZMDNMFHJD2XWDTEPGBDANCNFSM4ZO6XFCA .

[Nordhbane]

I've sent you a mail and written some more info about our server environment that I'd prefer to keep out of the public. Thanks again for helping.

[Nordhbane]

Greetings @lilith , Just checking if I sent the mail to the correct address. Your email in your comment was censored, so I sent it to the one in your profile with imazen.io. Hope it reached you!

[lilith]

Yes it did! Thanks. It may take a bit for me to analyze.

On Thu, Mar 25, 2021, 1:41 AM Nordhbane @.***> wrote:

Greetings @lilith https://github.com/lilith , Just checking if I sent the mail to the correct address. Your email in your comment was censored, so I sent it to the one in your profile with imazen.io. Hope it reached you!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/imazen/imageflow/issues/581#issuecomment-806436249, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2LH64NWZH5I2MRGJ2DETTFLSJFANCNFSM4ZO6XFCA .

[lilith]

Sorry for the delay in looking into this. A few notes:

1. This is an extremely rare error code. It may be connected to data trickling, i.e., very slow clients. This is odd, however, because ARR is supposed to proxy and prevent data trickling.

2. There's not enough information in the logs to dig deeper. You'd need Fiddler running to figure out exactly what is going on with those problematic requests.

3. I am guessing that the error code is connected to the server going down because it's running out of some kind of resource. Looking into the Windows Application logs might help.

4. imageflow_server is currently built on Iron, which has not been maintained very well in the last few years. It's possible it's not being used enough for someone else to report this bug. The plan is to rewrite imageflow_server on top of Actix-Web once Imageflow .NET Server reaches 1.0 and we know which features and configurations are most popular. (Imageflow.NET Server is configured with C#, which allows for a lot more customization and integration. We'd have to drop most of that power to make configuration TOML-based, so we need a solid understanding of what people use the most.

5. Imageflow.NET Server is a much more solid choice than imageflow_server in reliability, speed, and power. It's much easier to maintain than imageflow_server and I can ship patches faster. It's also running a much larger number of production servers at the moment. It's fully async and should be able to handle a higher number of concurrent connections than imageflow_server due to that and other reasons.

6. If you want to stick with imageflow_server, you could try https://docs.microsoft.com/en-us/iis/extensions/iis-search-engine-optimization-toolkit/managing-robotstxt-and-sitemap-files (hopefully compatible with ARR) to serve a robots.txt file, and hope that that fixes the problem.

[Nordhbane]

Thank you so much for an exhaustive reply with lots of good information to pick apart and go through to try out on our end. We are probably going to try getting the Imageflow.NET up and running as a test and see how it goes and hopefully we can move over to that one. If we discover anything useful to report that may help you squish or circumvent the bug we're experiencing I will let you know!

[lilith]

imageflow_server is obsolete: https://github.com/imazen/imageflow/issues/598