nft.storage + w3up uses more than 128MB memory sometimes in a way that can cause an out of memory exception when running in cloudflare workers runtime (e.g. staging.nft.storage)
Idea:
1) Use client.capability.store.add() and then upload.add() because uploadCAR is doing sharding when there is no need because the CAR is guaranteed to be less than 100MB.
via @alanshaw and @travis
right now the integration is using w3up-client uploadCAR method to upload CARs of file shards sent via HTTP to the server. We think in practice the nft.storage client is already shareding and making sure these CARs are smaller than the shard size.
uploadCAR has sharding functionality that we don't need here, and that could be source of memory leak
we don't need/want the sharding that uploadCAR does
Proposal:
instead of using uploadCAR, let's use lower level functionality to manually build the store/add and upload/add invocation
gobengo
commented
7 months ago
Context
Idea:
uploadCARmethod to upload CARs of file shards sent via HTTP to the server. We think in practice the nft.storage client is already shareding and making sure these CARs are smaller than the shard size.uploadCARdoesProposal:
store/addandupload/addinvocation