Easily upload large files and directories

[dchoi27]

More and more users are trying to upload large files and directories and are having trouble. For NFTs, being able to do so means simplifying the minting flow for users, especially being able to do large drops (e.g., 10K NFT drops).

This involves a combination of fundamental improvements, as well as a good overall UX to satisfy various user types. The two main user archetypes are:

• Individual creators / minters (end-users) who want to upload a ton of data at once (e.g., 10K NFT drops in a single, flat directory), generally not that technically savvy and are generally in hardware-constrained environments. There's two main sub-cases here:
  • End-users who are hand-rolling their own metadata to mint (either directly or through a minting service / marketplace), and could use a no-code application to help them upload their data, and get the data structure they need to then put each file's IPFS URL in an NFT (note that this would probably end up being used for smaller uploads as well)
  • End-user who are using an SDK or marketplace that is integrated with a storage solution to mint, and uploads to the storage solution directly (e.g., Metaplex); the pipes probably look the same, but there needs to be a programmatic way / API for the integration to happen (with the end-user not exposed to the complexity)
• Marketplaces who run a centralized server for their service, and want to increase the flexibility of what they can provide users (generally larger data limits); in this case, they need a way to integrate via client library / API, with a solution that ideally doesn't require heavy hardware (but can if it needs to in most cases)

To focus things a bit, we should make sure we have solutions for the following cases:

• [ ] (P1) End users uploading flat directory with a lot of small files (10K) that add up to 2GiB+ (#981)
• [ ] (P1) End users uploading individual file that is 2GiB (#981, potentially with #980)
• [ ] (P1) Marketplaces uploading individual file that is 8GiB+ from their server (#980)
• [ ] (P2) End users uploading individual file that is 8GiB+ (#981 + #980)
• [ ] (P2) Marketplaces uploading directory with a few larger files (<20) that add up to 8GiB+ from their server (#980) ^ potentially something around file type here, will update as we get more clarity (can default to JPEG and MP4 in the meantime)

Additionally, we should make sure to add docs / warnings to discourage uploading in ways that are likely to fail (e.g., shouldn't use browser to upload big files). (#982)

[atarev-crypto]

May I suggest including 8GiB+ CAR files as well as a specific test?

[dchoi27]

@atarev-crypto the only way 8GiB+ files can be uploaded is via CAR (on the client side, the file has to be converted to a CAR and chunked) so it will be covered.

[mikeal]

(P1) End users uploading flat directory with a lot of small files (10K) that add up to 2GiB+ (P1) End users uploading individual file that is 2GiB+

@dchoi27 and I were speaking today and he noted that the folks asking for this the most are not developers.

the best solution for these users is probably an Electron app to help them upload large datasets. we could write it in such a way that it avoids the extra ondisc copy into the blockstore the current ipfs-car streaming uses and offers more intermediate guarantees.

(P1) Marketplaces uploading individual file that is 8GiB+ (P2) Marketplaces uploading directory with a few larger files (<20) that add up to 8GiB+ ^ probably something around file type here, will update as we get more clarity (can default to JPEG and MP4 in the meantime)

are these pinning requests or CAR uploads?

also, when we say “marketplace” do we mean “from the marketplace’s infrastructure” or simply “from the marketplace SDK?”

[dchoi27]

for this issue, let's assume all CAR uploads. the pinning request flow is largely from marketplaces already deeply having integrated with someone like Pinata, but they're getting additional redundancy via NFT.Storage. The next wave of users will be more likely to just integrate with us directly. (The pinning service flow is less problematic overall, anyway)

also, when we say “marketplace” do we mean “from the marketplace’s infrastructure” or simply “from the marketplace SDK?”

That's a good distinction. I mean from the marketplace's infra. From the marketplace SDK I think is covered in the end user flows represented above, but I'll edit the OP to note the distinction (i.e., the pipes should work the same, but the UI will be different between an Electron app type thing and an in-code integration with an SDK).

[jchris]

Seems like the killer feature would be outputting a file listing that is ready to go for ingest in tools like Candy Machine, etc.

[dchoi27]

Yeah! I think we need to do a bit more digging on what the right interface is there. Specifically for Candy Machine, I think the from the end-user perspective, they upload files using a Candy Machine command, and the SDK itself handles putting the metadata URLs into the minting transactions (correct me if I'm wrong @yusefnapora), but there are definitely other (maybe sometimes varied) cases that we should figure out what the flow looks like to integrate with different ecosystems/minting tools (copy/paste? output a file they can iterate over? etc.)

[atarev-crypto]

Not sure if this should be a separate issue, or if it should be part of the conversation here, but I was able to perform a direct (un-chunked) CAR upload of a 4GiB file (using modified storecar.js), and while all the data appears to be there, the list of files at https://nft.storage/files/ shows a 9.78MB file size? Not sure if this is a display issue or something else going on.

[dchoi27]

@atarev-crypto just confirming - you didn't get an error message at all? There's a known issue when the upload terminates (see: #896 - basically the server doesn't know that there was an error and displays the CAR file in the uploads list as if the whole thing got uploaded) but if the upload completed with no error, then it could be something else.

[atarev-crypto]

Correct, no error message. The data all appears to be present as well, though there are 10K files so I have not tested each one individually. I'd be happy to share the hash privately if there's a way to do so?

[dchoi27]

@atarev-crypto the CID would be super useful! if you're on discord, it'd be great if you don't mind DMing me

[atarev-crypto]

I am, in fact I was looking for a discord server for this project, as I figured that would be a good way to give feedback and get help. I have sent you a friend request.

[dchoi27]

This is mostly solved with NFT Up and E-IPFS in production. Console v2 should make this further better, but let's close this one.