search

weserv / images

Source code of wsrv.nl (formerly images.weserv.nl), to be used on your own server(s).
https://wsrv.nl/
BSD 3-Clause "New" or "Revised" License
2.03k stars 200 forks source link

Performance issues - Cloudflare is throttling weserv.nl #360

Closed ebaynaud closed 1 year ago

ebaynaud commented 2 years ago

Hi guys, don't know how to check that but is there any ongoing issue? It seems terribly long to load an image when using the proxy today.

@andrieslouw @kleisauke

andrieslouw commented 2 years ago

Investigating, we also saw some strange issues.

andrieslouw commented 2 years ago

We are seeing increased response times, even leading to timeouts sometimes, looking into this.

bolda500 commented 2 years ago

I confirm that the performance for servers located in the EU is also very low. It could also be a problem on the cloudflare side - https://www.cloudflarestatus.com/

andrieslouw commented 2 years ago

Yes, thanks, we can confirm this, Cloudflare is really slow at this time. We just tried with a new server, and putting Cloudflare in front of a simple image, slows down image loading significantly. It almost looks like it's throttling to a few kb/s. Ping times are perfect, just the throughput is really slow at this moment. This causes numerous slowdowns, as a lot of origins are also behind Cloudflare.

andrieslouw commented 2 years ago

For future reference, Cloudflare finally confirms issues:

Screenshot_2022-11-02-19-34-24-94_40deb401b9ffe8e1df2f1cc5ba480b12~2

raramuridesign commented 2 years ago

@andrieslouw I guess there is not much we can do but wait till the routing is resolved. Thanks for investigating.

andrieslouw commented 2 years ago

We have asked Cloudflare for comments on this issue, and will keep you updated. At this moment we are still seeing slowdowns in throughput.

andrieslouw commented 2 years ago

We're 99% sure that Cloudflare is throttling the domain images.weserv.nl and even other subdomains of weserv.nl. The domain wsrv.nl seems to be unaffected. Will keep you updated.

jesperordrup commented 2 years ago

Just confirming that we see the same problem (from denmark / finland)

ebaynaud commented 2 years ago

Can we safely replace images.weserv.nl by wsrv.nl?

andrieslouw commented 2 years ago

Can we safely replace images.weserv.nl by wsrv.nl?

Yes, it's the same infrastructure. It was build to be the future address of the service, and is in use for a while now. But we'll try our best to get the original images.weserv.nl up to speed again, and solve this issue. wsrv.nl will stay, and is unaffected from what we can see.

raRaRa commented 2 years ago

Based on what I read on Cloudflare Discord, the throttle is put on websites that are on the free plan and are using "excessive" bandwidth. Would it be of any help to upgrade from the free plan or are Cloudflare willing to help out?

andrieslouw commented 2 years ago

We have upgraded, but it is to no avail yet. Response from Cloudflare is taking time unfortunately.

raRaRa commented 2 years ago

Is it just me, or is it loading much faster now?

andrieslouw commented 2 years ago

Our monitoring shows requests are still taking 1.05 seconds on average (with 95% in 0.52 - 2.90). Normally this is around 0.52 seconds (with 95% in 0.15 - 0.75).

I'm afraid it is still slow, we're evaluating on how to proceed. The slowness is deliberately caused by Cloudflare, and so far they're only pointing at section 2.8 of their TOS. Without assistance from Cloudflare, I don't see any alternatives at this time.

Apologies for the inconvenience, it is hard to see this happening to the project, I would love to have better news or answers for you all.

kleisauke commented 2 years ago

For reference, this is section 2.8 of Cloudflare's terms of services: https://www.cloudflare.com/terms/#28-limitation-on-serving-non-html-content

raRaRa commented 2 years ago

The ToS is a bit weird since Cloudflare has introduced R2 and Cloudflare Images. Does it go against their ToS if the images are stored on R2 or delivered through Cloudflare Images?

Would it make any sense to look into other solutions such as AWS CloudFront and maybe start selling API keys for users who want to use weserv to pay for the egress bill?

andrieslouw commented 2 years ago

From what I understand, the R2 and Images are exempt by the supplemental terms: https://www.cloudflare.com/supplemental-terms/

So anything non-HTML would require the R2 or Images/Stream according to the latest TOS. Which is probably commercially motivated.

It's not the intention of this service to accept payments, or go commercial. We have never done so. That's not the spirit of an internet service. We don't even sell ads, and the only logo's and names we mention on our project are from the ones that help us. Names like nginx, libvips, GitHub, Cloudflare, Hetzner, OVH, Scaleway, and many others. If applicable, we privately fund small costs where needed (servers, licenses, open source development support to others), to help in creating a better internet.

Be reminded that the free service processes 6 million requests/images per hour on average, with ~400TB ingress (our servers) and ~250TB egress (Cloudflare) per month. I think it would be cheaper to just rent out servers in different locations global, and build our own CDN based on DNS geolocation.

Commercial services are also not 1-on-1 compatible in offering, they would require a good rewrite of almost everything. Apart from that, let's look at the money: Cloudflare Images you're looking at $2/million requests (assuming 90% cache ratio), Cloudflare R2 is around ~$4.5/million, AWS Cloudfront is ~$4.3/million. So you're looking somewhere around $9k up to $20k/month.

Any cloud-related offering is always crazy expensive, when compared with 1 Gb/s unmetered/1 CPU/32GB RAM for ~$50/month, which will almost handle the same amount of traffic, including the all the image operations (but just from 1 location).

I think a cheap 10 Gb/s unmetered server that just does what Cloudflare does (nginx-proxying) is around $200/month all-in now, and this will handle 3000 TB/month in and out (again, single location).

With scale, around $30/month should be the cost of the mentioned 6 million reqs/hr / 400TB/month CDN offering, so the paid cloud services designed for this operate at a 300x - 700x markup, ballpark figure.

You can always decide to use the source code, and do the roll-out you mention on any cloudservice, we'd welcome this, and appreciate all feedback. It's how we learn, how we improve the code.

With cloud offerings you sure pay for more than just some hardware, fibre and electricity. However, those are the only 3 things our project actually needs to work. The project itself is just a piece of love, which is free, as it should.

ebaynaud commented 2 years ago

Respect 🙌🏼

tompodsiadlo commented 2 years ago

Hey guys, since last week we can see performance problems with resizer : image we are using https://images.weserv.nl/?url=

Out app could be sending multiple request (but nothing excessive like more than 1k in 10s ), but we didn't have problems so far. Could it be connected to throttling? Should we try running our own instance of resizer or is it temporary?

raramuridesign commented 2 years ago

@andrieslouw I am just curious if you had a more positive response from Cloudflare on this issue?

andrieslouw commented 2 years ago

@tompodsiadlo: Please try using https://wsrv.nl/?url=, it should perform better.

@raramuridesign: No, still not yet.

raramuridesign commented 2 years ago

@andrieslouw I have successfully switched to the other domain mentioned.

Questions

  1. will this not have the same issue as the previous domain?
  2. can cloudflare not be switched off on images.weserv.nl?
  3. or will this cause too much stress on the current server?

M.

andrieslouw commented 2 years ago

@raramuridesign:

  1. We don't know, it is hard to get answers from Cloudflare.
  2. No, unfortunately, this will also impact performance enormous now. This is however something we're evaluating, as said in the comment about building out our own CDN.
  3. It will cause long response times for everyone outside Europe, and the bandwidth on peaks is not yet there.
raramuridesign commented 2 years ago

@andrieslouw I am on the discord channel from Cloudflare, if you chat with them there - you might get to the right person quicker. Just a thought.

andrieslouw commented 2 years ago

@raramuridesign: Ah, you probably saw the messages from @kleisauke in that Discord already, on 2 nov 2022 around 17:37 UTC. There wasn't a lot of response, we even shared the ticket numbers.

raRaRa commented 2 years ago

I was just curious to know if there has been any update. If you were to build your own CDN, would you physically host your own server, or utilize AWS or any other cloud provider? Would love to hear some rough ideas on your plans going forward :)

BitesizedLion commented 1 year ago

I was just curious to know if there has been any update. If you were to build your own CDN, would you physically host your own server, or utilize AWS or any other cloud provider? Would love to hear some rough ideas on your plans going forward :)

My guess is any cloud provider (especially AWS) would become too expensive.

kleisauke commented 1 year ago

Good news, Cloudflare has decided to lift the restrictions on the weserv.nl domain (and subdomains) after a manual review.

raramuridesign commented 1 year ago

@kleisauke This is great news... well done for persisting!

maltesa commented 1 year ago

Does this also apply to wsrv.nl?

andrieslouw commented 1 year ago

wsrv.nl is and was unaffected by all of this. But everything is working as it should again.

betweenbrain commented 1 year ago

Hello, It seems that this issue may be occurring again. Is anyone else experiencing slow loading?

kleisauke commented 1 year ago

I just filled a support ticket at Cloudflare. It seems to be occurring again (with the same symptoms, wsrv.nl is not affected).

kleisauke commented 1 year ago

Cloudflare's engineering team also removed this secondary throttle from the weserv.nl zone.

I'll close, please open a new issue if this occurs again.