Q4 Dependency needs for IPLD

[mikeal]

I’d like to surface our current work load and priorities for Q4 and make sure this is in alignment with the dependencies for which this work is being done.

The following IPLD efforts are top level priorities in Q4.

• For Filecoin
  • Codegen in go-ipld-prime
• For IPFS
  • UnixFSv1 Improvements
  • UnixFSv2
• libp2p
  • None that we are aware of

Q4 Expectations

This is our current plan, based on what we’ve heard so far from dependent projects. If these priorities or timeline do not align with your needs please let us know so that we can adjust.

Codegen

@warpfork hit a large milestone recently in codegen for go-ipld-prime. This work will continue through Q4 and we expect to see other milestones along the way. This is the highest priority work item for Go in IPLD through the end of the year, and most likely in early 2020 as well.

Given that @whyrusleeping has written a smaller library for his immediate use we’d like to know what will still be blocked, and on what timeline, until codegen in go-ipld-prime is complete.

UnixFS

The schema for UnixFSv1 improvements will land soon (within the week). After that the changes in Go and JS IPFS will need to be resourced. Our team has much more limited experience with the UnixFSv1 code base than IPFS engineers do, so we would prefer for these changed to be resourced by those engineering teams. If not, we can adjust our workload to take it on with the understanding that the timeline for everything else mentioned here (codegen, unixfsv2) will be pushed out.

A usable implementation of UnixFSv2 will be completed in JS by the end of Q4, with milestone releases we can use to gather feedback as early as late Q3. Integration into js-ipfs (with support from js-ipfs engineering) and any remaining feature requirements is slated for Q1 2020.

The Go implementation of UnixFSv2 is not yet resourced. Waiting until 2020 will allow the schema to mature through the JS implementation and should make the Go implementation much easier, but doing this serially rather than in parallel with the JS implementation will push out the final completion date. It’s our understanding that, at this time, codegen is a higher priority for Go dependencies than UnixFSv2.

Feedback

@momack2 @warpfork @rvagg @vmx @jbenet @daviddias @Stebalien @alanshaw @mgoelzer @whyrusleeping

Please read this over and let us know what you think about this timeline and this set of priorities. If any of these are out of alignment with the needs of your project we still have plenty of time to adjust our planning to compensate.

[rvagg]

The schema for UnixFSv1 improvements

Can you expand on what this is about, or is there an issue describing this somewhere? I was only aware of active development on UnixFSv2.

[mikeal]

@rvagg https://github.com/ipfs/specs/pull/220 , I’ll put a link in the Q4 OKR’s as well.

[warpfork]

This is just details in support of the summary above, but might be worth noting -- there are lots of reasons we're focusing hard on codegen at this point.

Some set of choices that is {fast}&&{supports-schemas}&&{pleasantly-programmable-against} is (IMO) a prereq to doing unixfsv2 in Go (or any of the other sizable hobby-projects I have that I'd like to move over to IPLD :)) -- and that meant either reflective-bind-nodes impl plus runtime/wrapper schema nodes; or, codegen. At this point we've chosen codegen. Since we now have significant progress on that front, resultingly, carrying codegen to full fruition is a pretty unwavering goal.

[momack2]

Thanks for this issue and asking proactively about dependencies! Very useful.

Needs from IPLD I know of: [P0] Update to unixfsv1 to support modification time / executable bit [P1] (experimental?) Integration of go-graphsync into go-ipfs (performance) [P1] Unixfsv2 [P2] Integration of go-ipld-prime into go-ipfs (performance)

@Stebalien @hsanjuan @achingbrain - anything to add or re-rank here?

[mikeal]

[P1] (experimental?) Integration of go-graphsync into go-ipfs (performance)

I’d love to dig into this a little bit.

What would the Q4 P1 target for this be? There’s a big gap in the effort required to make this work for something like replicating Filecoin data into IPFS and making it work for UnixFSv1/Protobuf data since that code pre-dates IPLD in IPFS.

[momack2]

Yep - not trying to add things to the Q4 list you stated above (agree integrating go-graphsync is a non-trivial amount of work), however want to you be aware of where I think our IPFS priorities related to IPLD lie. It might be that Unixfsv2 is actually above go-graphsync - but given the focus on updating Unixfsv1 to unblock the current work, I think our next priorities focus on performance (especially related to e2e data transfer) and the timeline for unixfsv2 gets more breathing room.

[achingbrain]

Looks good to me.

We should probably clear up the prioritisation around UnixFSv2 though, if our focus in the next quarter is performance and not features it should probably be marked as P2 and perf-related tasks as P1. Otherwise it should be P1, the perf tasks P2 and we should all pull together to get it over the line (after the UnixFSv1 improvements which should be relatively straightforward once the discussion in https://github.com/ipfs/specs/pull/220 dies down).

What do you think?

[mikeal]

Keep in mind that we need some lead time in order to feed into your schedule and priorities.

In JS, if we get UnixFSv2 working in Q4 that means the integration would mostly be happening in Q1 at the earliest, so even if Q4 is focused on perf, if Q1 is going to open up to features again this could actually align nicely with your schedule. If JS is leading Go on this, that means Go integration isn’t happening until Q2 at the earliest with some implementation work happening in Q1 at the earliest.

Also, keep in mind that there is a limit to the perf improvements that we can make to UnixFSv1 before we hit a ceiling that UnixFSv2 is, in part, designed to break through. That said, since perf is such a high priority, I’ll integrate performance benchmarking into the Q4 UnixFSv2 work so that we can track and evaluate that difference so that, once it’s performant, that difference can be considered when prioritizing the integration work.

[rvagg]

closing for archival