Error 400 (Payload too large) when deploying

[ekollof]

$ ipd
ℹ 🤔 No path argument specified. Looking for common ones…
✔ 📂 Found local public directory. Deploying that.
✔ 🚚 public weighs 133.3 MiB.
✖ 💔 Uploading to Infura didn't work.

  ┌─────────────────────────────────────────────────────────────────────────────────────────┐
  │                                                                                         │
  │ {                                                                                       │
  │    "name": "Error",                                                                     │
  │    "className": "Error",                                                                │
  │    "message": "payload too large\n",                                                    │
  │    "superclasses": ["Object"],                                                          │
  │    "enumerableFields": { "statusCode": 400 },                                           │
  │    "stack": [                                                                           │
  │      "Error: payload too large",                                                        │
  │      "at parseError                                                                     │
  │ (/usr/local/share/.config/yarn/global/node_modules/ipfs-http-client/src/utils/send-requ │
  │ est.js:22:17)",                                                                         │
  │      "at ClientRequest.<anonymous>                                                      │
  │ (/usr/local/share/.config/yarn/global/node_modules/ipfs-http-client/src/utils/send-requ │
  │ est.js:64:14)",                                                                         │
  │      "at Object.onceWrapper (events.js:288:20)",                                        │
  │      "at ClientRequest.emit (events.js:200:13)",                                        │
  │      "at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:565:23)",   │
  │      "at HTTPParser.parserOnHeadersComplete (_http_common.js:116:17)",                  │
  │      "at TLSSocket.socketOnData (_http_client.js:452:22)",                              │
  │      "at TLSSocket.emit (events.js:200:13)",                                            │
  │      "at addChunk (_stream_readable.js:294:12)",                                        │
  │      "at readableAddChunk (_stream_readable.js:275:11)"                                 │
  │    ]                                                                                    │
  │ }                                                                                       │
  │                                                                                         │
  │                                                                                         │
  └─────────────────────────────────────────────────────────────────────────────────────────┘

  ┌─────────────────────────────────────────────────────────────────────────────────────────┐
  │                                                                                         │
  │ Failed to deploy.                                                                       │
  │                                                                                         │
  └─────────────────────────────────────────────────────────────────────────────────────────┘

Also happens with pinata.

[agentofuser]

@obo20 is there an official size limit for Pinata uploads? Couldn't find it in the docs. Thanks!

[agentofuser]

@ekollof there isn't anything I can do about their size limits, but if you know what they are exactly I can add a check after calculating the directory size so that it doesn't attempt upload if it's past the limit. Thanks for reporting!

[obo20]

@agentofuser our upload limit is 1GB per file, so we shouldn't be running into any issues with a 133MB file / or directory

[agentofuser]

@ekollof can you retry with pinata and post the error please?

[agentofuser]

@obo20 so when building the form data for uploading a whole directory, the limit is not 1GB on the whole form data, but on each file within it? There's no limit on the total size of the upload?

[obo20]

@agentofuser no, the directory uploads have a combined max size

[agentofuser]

@obo20 thank you! So 1GB max total per request, no matter how many files, correct?

@ekollof according to Infura staff, their max file size is 100MB:

https://community.infura.io/t/is-there-a-file-size-limit-for-files-attached-to-ipfs-infura-io/61/2.

I'll close this and add a separate issue for ipfs-deploy to check if the directory size exceeds the any of the limits of the chosen pinning services.

Feel free to open a new issue with the error messages if you somehow bump into a size limit on Pinata with the 133MB file. Thanks!

[obo20]

@agentofuser That's correct. 1GB max total per request, no matter how many files. This will likely go up in the future, as we've upgraded our infrastructure fairly significantly since that initial limit was put in place.

[ekollof]

Could the upload be split up or chunked in multiple requests maybe? Also, the whole public/ dir is 133 MB, the files included are way smaller :)

[agentofuser]

@ekollof duh, yeah, I hadn't considered chunking 😅

If no file is bigger than the limit, ipfs-deploy could build up the tree with successive uploads.

If any one file is bigger than the limit though, i don't know if it would be possible to split it up, upload the chunks, then get the hash for the concatenated file so that it can be referred to in the parent directory. Do you know if that would be possible, @obo20?

@ekollof is this something you would be interested in working on?

[ekollof]

@agentofuser I would be if I could code in javascript. I'd have to teach myself that first. :)

[obo20]

@agentofuser We don't have plans to do any sort of chunking concatenation. That would be very complicated / error-prone and it would be simpler for us to simply raise our upload limits.

[LanYunDev]

我也遇到类似的问题,我使用Pinata进行上传,经过很长时间的排查,发现不是上传大小的问题,是上传文件数量的问题.