is it possible to change the limit of the gateway NFT.Storage?

[nhutleFlydinotech]

I looked at the docs and found that the gateway NFT.Storage has a limit of 200 requests, is there a way to increase this limit?

[vasco-santos]

Hey @nhutleFlydinotech

nftstorage.link is a public good shared by lot's of parties. This way we need to guarantee it works well for everyone. Allowing increased rate limits would degrade overall performance. We will be working on improving the gateway over time and probably we will be able to increase it a bit in the future.

Anyway, let me know what is your use case and I can help out with alternatives for you to achieve what you are looking for.

[nhutleFlydinotech]

In my case i need to load hundreds of images on ipfs at the same time and i think that action will make a lot of requests. My requests count may exceed limit of 200 requests of NFT.Storage providing.

[vasco-santos]

Is this in a website? Can't you lazy load them as needed instead?

[danizord]

@vasco-santos In my case I'd like to fetch all token metadata in order to calculate rarity scores. Is there any way to fetch the whole collection in a single request?

[vasco-santos]

@danizord any reason for not being able to throttle your requests? It looks like a one time job, it will of course take longer than just being able to request unlimited number of CIDs, but then you likely will need time to process them. IPFS Public Gateways are usually a public good, this way they are rate limited to avoid abused and guarantee a nice experience for everyone.

If you really need a way bigger rate, the best option is to spin up your own IPFS Node. This would mean only you would be using it and can perform unlimited requests. Of course this is more time consuming and needs Infra. An alternative for slightly increasing rates is to rely on multiple public gateways at the same time, performing requests to them in a round robin fashion or similar.

Is there any way to fetch the whole collection in a single request?

If files for a given collection are all within same folder, you can definitely just download the folder and iterate through its content.

[danizord]

@vasco-santos appreciate your help!

@danizord any reason for not being able to throttle your requests? It looks like a one time job

My application is time-sensitive. I'm trying to download fresh metadata immediately after reveal to calculate rarity and snipe rare tokens while they are at lower prices.

If you really need a way bigger rate, the best option is to spin up your own IPFS Node.

I tried this option, but seems like it is waay slower since it would take some time for my node to find the route to the newly published metadata, while those public gateways are usually very fast as it seems they have direct peering with the most popular pinning services.

If files for a given collection are all within same folder, you can definitely just download the folder and iterate through its content.

Is it possible to download a folder using nftstorage.link?

[vasco-santos]

Getting the folder basically gives you HTML file with links for each file.

By iterating, I mean that you don't need to download every single file in the directory given you want to calculate rarity. If you see https://bafybeiedv7sowwxamly4oicivudp45rsfvbklnf3fvbvonxrwoxqylhtwq.ipfs.nftstorage.link/ the html contains the list of files in that directory and the file own CID, which will be unique per file. So, I would say you just need to keep track of occurrences of those CIDs per collection folder?

[danizord]

@vasco-santos

So, I would say you just need to keep track of occurrences of those CIDs per collection folder?

That's indeed an interesting approach! But it won't work for some collections that include the token ID as part of the JSON metadata :/

Seems like a recent version of go-ipfs gateway is able to provide the directory in CAR format, but not sure if major gateways are going to adopt 🤔

[vasco-santos]

That's indeed an interesting approach! But it won't work for some collections that include the token ID as part of the JSON metadata :/

Gotcha

Seems like a recent version of go-ipfs gateway is able to provide the directory in CAR format, but not sure if major gateways are going to adopt 🤔

Yes, that is a good way to go! Please follow advancements at https://github.com/nftstorage/nftstorage.link/issues/147

I have been talking with gateway operators we use (ipfs.io, cf-ipfs and Pinata) to get that in. Once we have support in one of the gateways, we will special case query param format car as just relying on one. And update to race all of them as they update.

[vasco-santos]

Closing this @danizord

Feel free to re-open if you would like to discuss anything else