[Bug]: Writing 10 indices in candy machine in one instruction overruns transaction buffer with long links/names

polluterofminds commented 2 years ago

Which package is this bug report for?

candy machine cli

Issue description

While working on a PR for the Pinata integration, I discovered that the metadata JSON and the media link lengths can be too large causing an encoding overruns Buffer error.

When using nft.storage or Infura, the IPFS URI for both metadata and media is limited to https://ipfs.io. However, with Pinata, customers can have dedicated IPFS gateways. If they want to provide their gateway as the URI to read the IPFS content from, it's impossible to know up front what their gateway name might be.

When testing, I found that IPFS URIs prefixed with https://opengateway.mypinata.cloud would fail during the verify upload step.

However, IPFS URIs prefixed with https://open.mypinata.cloud would work.

If it's not possible to enable longer IPFS URIs for metadata json and media links, then is it at least possible to understand what the length limits are?

Command

ts-node ~/metaplex/js/packages/cli/src/candy-machine-v2-cli.ts verify_upload \
    -e devnet \
    -k ~/.config/solana/devnet.json \
    -c example

Relevant log output

saving config line 90-99 failed RangeError: encoding overruns Buffer
    at Blob.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:2325:13)
    at Structure.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1263:26)
    at WrappedLayout.encode (/Users/polluterofminds/metaplex/js/node_modules/@project-serum/borsh/src/index.ts:104:24)
    at WrappedLayout.encode (/Users/polluterofminds/metaplex/js/node_modules/@project-serum/borsh/src/index.ts:104:24)
    at Structure.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1263:26)
    at /Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1113:25
    at Array.reduce (<anonymous>)
    at Sequence.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1112:22)
    at Structure.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1263:26)
    at WrappedLayout.encode (/Users/polluterofminds/metaplex/js/node_modules/@project-serum/borsh/src/index.ts:104:24)

Operating system

macOS 12.0.1

Priority this issue should have

Low (slightly annoying)

Check the Docs First

[X] I have checked the docs and it didn't solve my issue

stegaBOB commented 2 years ago

Can you share an example link that is too long? Thanks

polluterofminds commented 2 years ago

It's not any one link that's too long. It seems to be an aggregate limit. The upload works. Then when writing the indices, the problem arises. Here's the truncated output from the CLI showing one of the links followed by the error.

uploaded manifest:  https://opengateway.mypinata.cloud/ipfs/QmXsrjL8Xz9sipxeqZ7QtsKnDJYMXr8XAWFNEtZa2wGoUD
Writing indices 0-9
saving config line 0-9 failed RangeError: encoding overruns Buffer
    at Blob.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:2325:13)
    at Structure.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1263:26)
    at WrappedLayout.encode (/Users/polluterofminds/metaplex/js/node_modules/@project-serum/borsh/src/index.ts:104:24)
    at WrappedLayout.encode (/Users/polluterofminds/metaplex/js/node_modules/@project-serum/borsh/src/index.ts:104:24)
    at Structure.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1263:26)
    at /Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1113:25
    at Array.reduce (<anonymous>)
    at Sequence.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1112:22)
    at Structure.encode (/Users/polluterofminds/metaplex/js/node_modules/buffer-layout/lib/Layout.js:1263:26)
    at WrappedLayout.encode (/Users/polluterofminds/metaplex/js/node_modules/@project-serum/borsh/src/index.ts:104:24)

It's also worth noting that this actually happens during the upload script. I just didn't see it until running the verify upload command.

stegaBOB commented 2 years ago

Ah gotcha. I think I know whats happening. It is trying to write 10 lines at a time and that is too long for the current txn limit in certain cases when the link is too long. A fix would be to dynamically upload less than 10 at a time if the links/names are too long.

stegaBOB commented 2 years ago

The limit per txn is ~1200 bytes or something along those lines. some of that is taken up by accounts and some other stuff. This is definitely something we need to fix

metaplex-foundation / metaplex