ComLake / comlake.core

USTH data lake core
https://ComLake.github.io/comlake.core
GNU Affero General Public License v3.0
0 stars 0 forks source link

Support for hierarchical data #4

Closed McSinyx closed 3 years ago

McSinyx commented 3 years ago

Or folders in plain English. AFAICT the following tasks needs to be supported through core's HTTP API:

/cc @larycoder and @npthao1312 for further discussion

npthao1312 commented 3 years ago

Should both file and folder include source and topics? It feels like a bit of duplication.

McSinyx commented 3 years ago

Sorry for the late reply, and yes, that's a good point. I looked a bit deeper into this and apparently core API should provide a way to add files and directories without/with minimal metadata. I imagine there are two ways to do this:

  1. Having a more versatile POST /add which can handle both file and folder (of multiple files and folders) upload. Implementing this as a multipart file upload is quite simple since I'll only need to proxy the request (with some headers modified) to IPFS, so it's up to you to decide if you want it (I've no idea how hard it is to implement on the caller side). Then we'll only need a POST /insert (or PUT /insert in case of mutable datasets) to cover all user cases.
  2. Drop current POST /add in favor of lower-level calls, namely POST /save, POST /mkdir, POST /cp (files and folders into a folder) and a new POST /add which takes the metadata and the CID (instead of the data).
npthao1312 commented 3 years ago

Either way is fine for me. But I would prefer 2.

npthao1312 commented 3 years ago

Remember to update the API doc

McSinyx commented 3 years ago

@npthao1312, I've just published a new release with support for this, please refer to the updated documentation (which is technically still the same URI, relinked for convenience purposes).