🪐 NFT.Storage Gateway, the IPFS gateway for NFT.Storage is not "another gateway", but a caching layer for NFTs that sits on top of existing IPFS public gateways. ***Notice: Uploads have been decommissioned.**** Learn more and find a new hot storage provider for uploading new assets: nft.storage/nft-storage-classic
While we should definitely handle these errors for other Durable Objects, they are not so critical given they are only used after a response is provided to the user (but we won't collect metrics). However, if the Rate Limit Prevention DO is overwhelmed, users are affected.
There are a few ways we can move around this:
Do not perform Rate Limit Prevention
We have a solution that works within Durable Object limits and is good. Its main motivation was when we were being aggressively rate limited by Public Gateways. Since then, we have solutions in place to be rate limited by user IP instead of worker IP for ipfs.io and Cloudflare gateways.
Given dedicated Pinata gateways do not have rate limits mechanisms in place, we could consider just not performing rate limit prevention at all.
The main disadvantage of this option is that we might overwhelm Pinata and force them to add Rate Limit (and of course hit their performance to handle requests), as well as adding other gateways that also have rate limits will become a problem.
Rely on CNAME DNS Record and CF Firewall Rate Limits
This solution would build on top of "Do not perform Rate Limit Prevention"
We can also remove Rate Limit Prevention from worker code, and push that responsibility within Cloudflare Firewall. For this, we would simply need to setup a DNS CNAME record for each gateway we want to support like pinata.nftstorage.link pointing to the real dedicated gateway. We can then setup Firewall to only allow requests from our worker to it and setup the rate limits we want to use there.
The main disadvantage for this solution is that we need to do this configuration for each new gateway we add. However, this might not be so bad as we might want different rate limits per gateway.
Rely on a new worker for generic rate limiting by query parameter
This solution would build on top of "Do not perform Rate Limit Prevention"
We can have a worker that just proxies requests to public gateways and rate limit using a query parameter. This would allow us to use a DNS record like gw.nftstorage.link?gw='pinata'. This solution would help us not needing to configure new Firewall/Rate limit rules by just updating an ENV var in the worker.
The main disadvantages are that this solution would likely be more pricey, would need code and if we would like different rate limit strategies we would need to hardcode them in the code.
Note: I think we would need to do a redirect here and not proxy traffic or we would likely need to pay for incoming traffic? So, we would need an extra hop.
Conclusions
I think solution "Rely on CNAME DNS Record and CF Firewall Rate Limits" provides us the most flexibility and is easy to implement. We can easily just setup the CF rules as needed as they will mostly be copy paste of already created ones. They have limits of rules we can have, but it is 100 rules, so we are fine on that.
Can we just put a CF Firewall rate limit on nftstorage.link directly? If we're racing gateways, we can tollerate some degree of being rate limited by others.
Can we just put a CF Firewall rate limit on nftstorage.link directly? If we're racing gateways, we can tollerate some degree of being rate limited by others.
The core problem here in terms of rate limiting is that typically gateways will rate limit by IP and therefore they will rate limit nftstorage.link rather than the client IP and the blockage can be super long. This way, we have been doing some work to work around this as you can read history on https://github.com/nftstorage/nft.storage/issues/1195
We started by creating the concept of Rate Limit Prevention Durable Object, which kept state of timestamp from requests within the timeframe of their rate limit properties. This allowed us to prevent requests that would rate limit our worker from performing requests. For you to have a better understanding of the impact here, CF public gateway would block us completely for 60 minutes if we got rate limited.
From that initial point, we moved with ipfs.io and CF teams with better solutions to work around this. Namely, ipfs.io now supports X-Forwarded-For header to understand we are proxying and rate limiting the client. I had some talks with CF and they kindly provided us a beta feature with a dedicated gateway manually setup via a DNS record from our own CF zone. This means that we now have cf.nftstorage.link pointing to this dedicated gateway and we can tweak its configuration of rate limiting instead of being under the strict rate limits in the public gateway. For better optimization of resources, we have a firewall in place there that does not allow any request to cf.nftstorage.link outside the nftstorage.link CF zone.
The proposals 2 and 3 here, are to do something similar to what we do with CF dedicated gateway in CF Firewall/rate limit setup, rather than relying on tracking state in the Durable Object
@vasco-santos commented on Thu Mar 17 2022
Currently Rate Limit Prevention Durable Object is a bottleneck that might cause problems as gateway scales with errors like https://developers.cloudflare.com/workers/learning/using-durable-objects/#error-durable-object-is-overloaded
While we should definitely handle these errors for other Durable Objects, they are not so critical given they are only used after a response is provided to the user (but we won't collect metrics). However, if the Rate Limit Prevention DO is overwhelmed, users are affected.
There are a few ways we can move around this:
Do not perform Rate Limit Prevention
We have a solution that works within Durable Object limits and is good. Its main motivation was when we were being aggressively rate limited by Public Gateways. Since then, we have solutions in place to be rate limited by user IP instead of worker IP for ipfs.io and Cloudflare gateways. Given dedicated Pinata gateways do not have rate limits mechanisms in place, we could consider just not performing rate limit prevention at all. The main disadvantage of this option is that we might overwhelm Pinata and force them to add Rate Limit (and of course hit their performance to handle requests), as well as adding other gateways that also have rate limits will become a problem.
Rely on CNAME DNS Record and CF Firewall Rate Limits
We can also remove Rate Limit Prevention from worker code, and push that responsibility within Cloudflare Firewall. For this, we would simply need to setup a DNS CNAME record for each gateway we want to support like
pinata.nftstorage.linkpointing to the real dedicated gateway. We can then setup Firewall to only allow requests from our worker to it and setup the rate limits we want to use there. The main disadvantage for this solution is that we need to do this configuration for each new gateway we add. However, this might not be so bad as we might want different rate limits per gateway.Rely on a new worker for generic rate limiting by query parameter
We can have a worker that just proxies requests to public gateways and rate limit using a query parameter. This would allow us to use a DNS record like
gw.nftstorage.link?gw='pinata'. This solution would help us not needing to configure new Firewall/Rate limit rules by just updating an ENV var in the worker.The main disadvantages are that this solution would likely be more pricey, would need code and if we would like different rate limit strategies we would need to hardcode them in the code.
Note: I think we would need to do a redirect here and not proxy traffic or we would likely need to pay for incoming traffic? So, we would need an extra hop.
Conclusions
I think solution "Rely on CNAME DNS Record and CF Firewall Rate Limits" provides us the most flexibility and is easy to implement. We can easily just setup the CF rules as needed as they will mostly be copy paste of already created ones. They have limits of rules we can have, but it is 100 rules, so we are fine on that.
@olizilla commented on Thu Mar 17 2022
Can we just put a CF Firewall rate limit on nftstorage.link directly? If we're racing gateways, we can tollerate some degree of being rate limited by others.
@vasco-santos commented on Thu Mar 17 2022
@olizilla We have it already https://github.com/nftstorage/nft.storage/tree/main/packages/gateway#rate-limiting
The core problem here in terms of rate limiting is that typically gateways will rate limit by IP and therefore they will rate limit nftstorage.link rather than the client IP and the blockage can be super long. This way, we have been doing some work to work around this as you can read history on https://github.com/nftstorage/nft.storage/issues/1195
We started by creating the concept of Rate Limit Prevention Durable Object, which kept state of timestamp from requests within the timeframe of their rate limit properties. This allowed us to prevent requests that would rate limit our worker from performing requests. For you to have a better understanding of the impact here, CF public gateway would block us completely for 60 minutes if we got rate limited.
From that initial point, we moved with ipfs.io and CF teams with better solutions to work around this. Namely, ipfs.io now supports
X-Forwarded-Forheader to understand we are proxying and rate limiting the client. I had some talks with CF and they kindly provided us a beta feature with a dedicated gateway manually setup via a DNS record from our own CF zone. This means that we now havecf.nftstorage.linkpointing to this dedicated gateway and we can tweak its configuration of rate limiting instead of being under the strict rate limits in the public gateway. For better optimization of resources, we have a firewall in place there that does not allow any request tocf.nftstorage.linkoutside thenftstorage.linkCF zone.The proposals 2 and 3 here, are to do something similar to what we do with CF dedicated gateway in CF Firewall/rate limit setup, rather than relying on tracking state in the Durable Object
@olizilla commented on Thu Mar 17 2022
Thanks for the context! I prefer