ipld / js-unixfs

UnixFS Directed Acyclic Graph for IPLD
Other
6 stars 4 forks source link

feat: implement flat dir support #27

Closed Gozala closed 1 year ago

Gozala commented 2 years ago

Implements preliminary directory support (no sharding or blocksize checks yet). Also makes few API changes to support this:

  1. File.createImporter API is gone is removed. Instead one should do following:

    import * as UnixFS from "@ipld/unixfs"
    
    // Create filesystem writer and a blocks stream from which created blocks
    // can be read.
    const fs = UnixFS.create()
    
    // Create file writer that can be used to encode file and write
    // content into it.
    const file = UnixFS.createFileWriter(fs)
    file.write(new TextEncoder().encode('hello world'))
    const { cid } = await file.close()
    // eventually
    fs.close()
  2. Implementation had be updated to be based on WritableStream interface which seems to be more widely available now. This means you can simply create UnixFS writer from WritableStream directly:

    import * as UnixFS from "@ipld/unixfs"
    const { writable, readable } = new TransformStream()
    const fs = UnixFS.createWriter({ writable })
    const file = UnixFS.createFileWriter(fs)
    file.write(new TextEncoder().encode('hello world'))
    const { cid } = await file.close()
    // eventually
    fs.close()

Directory writer interface is intentionally very limited. You can only add entries that were created by writing a file or a directory in prior. This design choice guarantees that you can't have directories with partially written files which you'd need to somehow ensure get closed before you close the directory or have internals that would await for closures. I feel this is a better design as it makes certain bugs impossible (e.g. never resolving promise because some file was not closed).

In addition I have also added experimental .fork() method to directories which allows you to "fork" directory into a multiple divergent copies. This could be utilized for example to encode filesystem patches, but it's pretty rough.

⚠️ I had no time to write nearly enough tests here to make sure everything works as expected ⚠️