Upload directory / wrapper

[douglasmsi]

Hi, everyone!

I'm studying about IPFS and find the IPFS-Store. It's great project and now I have a question. How can I upload a structure of directories and subdirectories with IPFS-STORE?

Has someone an example or how I can do this?

Tks

[gjeanmart]

Hi, This feature isn't available at the moment but it could be an interesting one. What is your use case ?

IPFS-Store is a HTTP API on top of IPFS, uploading folders over HTTP isn't a common thing. Usually, the client sends each file one by one alongside metadata (like parent folder) and the backend stores files and metadata in a database or a file system.

First of all, IPFS stores directories and files using a graph format called DAG (Directed acyclic graph), while each file stored on IPFS is represented as a tree (chunk of 256kb) . Directories actually use the same principle building a larger tree. So IPFS clients recursively uploads each UnixFS file on IPFS, generate a tree of these files (links) and upload the tree on IPFS and start over on the parent folder.

Building this kind of behavior over an API like IPFS-Store wouldn't be easy as the client would have to upload the directory file by file (no parallelism possible) and IPFS-Store to build manually the DAG before storing it on IPFS and returning the final root hash to the client.
That's just for the IPFS part.

For the ElasticSearch, files would have to be indexed afterward with the root hash and the path.

It's an interesting feature, before starting implementing this, I would suggest to make a prototype to upload a directory from a web client to an IPFS node.

[douglasmsi]

Hi, gjeanmart.

The prototype is ready. I did 10 days ago... It's necessary realize some adjust, but the upload multiple files works for one leaf. Maybe could be better, but it's working. The main controller is File Controller, and I tested with ipfs cluster too, just change the port. It's working too.

I used the api Java-ipfs and the method upload Multiple files contains the upload of directories. Now, are you think it's possible implements this feature in the IPFS-Store?

My use case is uploading structures to ipfs cluster /daemon and after be possible distribute in some workstations.

https://github.com/DMSec/ipfs-directory-uploader/blob/master/src/main/java/br/com/dmsec/idup/controller/FileController.java

Tks

[gjeanmart]

Hi Douglas, That's awesome! Yeah I think it could be possible, I have a few comments:

• It seems the program uploads multiple files that represent a directory packageName using HTTP MultiPart. But MultiPart have a flat structure, you can't keep the directory/subdirectory structure easily imo. Is it a valid usecase for you to lose the structure ?

• Using the MultiPart solution, I'm also not sure it would scale very much. Imagine a MultiPart request that contains hundred of files over one HTTP call.

• That's why modern api usually makes upload file by file alongside metadata (structure) to control the bandwidth

In any case, the files and folders need to be written on the filesystem before pushing to IPFS.

I can try to explore a new controller which would basically store a directory. Just wondering how to index this in ElasticSearch (one document per root directory or one document per file).

Thanks!