Sprint Dec 14

[RichardLitt]

Sprint Dec 14

• Completed IPFS API Spec (core + transports) by Jan 1

  Sprint Goals

  Sprint Discussions

Schedule

Please take notes in a separate pad, if you can, and link it here.

Endeavor	Lead	Time (PDT - UTC/Z - CET)	Pad
Apps on IPFS	Juan Benet	10:30PDT 18:30Z 19:30CET	https://public.etherpad-mozilla.org/p/ipfs-dec-14-apps-on-ipfs
infrastructure	Lars Gierth	11:00PDT 19:00Z 20:00CET	https://public.etherpad-mozilla.org/p/ipfs-dec-14-infrastructure
libp2p	David Dias	11:30PDT 19:30Z 20:30CET	https://public.etherpad-mozilla.org/p/ipfs-dec-14-libp2p
js-ipfs	David Dias	12:00PDT 20:00Z 21:00CET	https://public.etherpad-mozilla.org/p/ipfs-dec-14-js-ipfs
go-ipfs	Jeromy Johnson	12:30PDT 20:30Z 21:30CET	https://public.etherpad-mozilla.org/p/ipfs-dec-14-go-ipfs

Please add the Agenda to the Pad before the endeavor sprint starts.

Sprint Deliverables

• Add your tasks below in a comment, that way we only have people listed who are really in the sprint
• Add links to issues down here. Only add things you can finish this sprint.

[ghost]

infrastructure

Lead: @lgierth

Agenda

• dev040 deployment
  • update strategy, also for the future
  • v04x.ipfs.io / v03x.ipfs.io
• storage
  • biham and nihal
  • castor and sirius
• deploying site updates automatically in a consistent way via IPFS or IPNS (project repos, website, maybe blog?)
  • see also https://github.com/ipfs/blog/issues/9#issuecomment-164192556

Participants

• @diasdavid
• @harlantwood
• @jbenet
• @lgierth

Notes

• dev040
  • open issue to come up with upgrade plan
  • include certain ipfs users who are embedding ipfs in their software and might have trouble updating
  • one option: have the multistream handshake be backwards-compatible
  • another option: break compat, but do so with a plan of how to do it better on both our end and their end
• storage
  • nihal for internal use
  • biham "free for all"
  • copying castor's and sirius' data to biham, then shutting them down
  • daviddias can resume NPM on biham once the copying is finished
• storage at jbenet's place
  • the host formerly known as earth will be resurrected in the coming days
  • will serve as OSX build machine
  • hands-on storage
• deploying site updates automatically
  • spin up server which receives github hooks, builds sites, publishes to IPNS
  • start with project-repos (aka ci-status)

[daviddias]

Apps on IPFS

Lead: @dignifiedquire + @jbenet

Agenda

• registry-mirror
• have a node with full npm
• make experience of getting it ready really nice
• release 0.4.0 on go builder with latest bell and whistles
• migration https://github.com/ipfs/infrastructure/issues/134
• ipfs/distributions
• IPNS + CoreNetwork

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

registry-mirror

• try the new 0.4.0 with Rabin again in Jeromy Machine (check with Jeromy)
• use one of the 17TB nodes to be the main one for npm on IPFS
• try with tar importer
• improve installation process of 0.4.0 (check with Jeromy)

ipfs/distributions Update

• design draft approved last week
• in the middle of implmentation
• Goal: Get it ready before 0.4.0

IPNS + Core Network

• Core Network is using IPNS to publish/consume social network style streams
• IPNS speed concerns
• PubSub may fix this, either by sending updates directly, or pushing IPNS updates over PubSub

[daviddias]

libp2p

Lead: @diasdavid

Agenda

• factoring go-libp2p
• swarm Interop
  • secio in JavaScript
• Are we going TLS?
• python libp2p
• rust libp2p
• update roadmap if needed https://github.com/ipfs/pm/pull/72

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

• crypto channels
  • multistream enables us to have several protocols already
  • implement TLS 1.3 (check if there are implementations that work in JS) as a upgrade transport
  • https://github.com/digitalbazaar/forge
  • look into curveCP
  • skip secio in JS land for now
  • list all of the interfaces in libp2p for Juan to bikeshedding

[daviddias]

js-ipfs

Lead: @diasdavid

Agenda

• js-ipfs-repo
• arch https://github.com/ipfs/js-ipfs-repo/issues/9#issuecomment-163835869
• nail down https://github.com/ipfs/js-ipfs-repo/issues/13
• IPLD Data Importing
  • https://github.com/ipfs/js-ipfs/issues/41#issuecomment-164294266
• BitSwap
  • https://github.com/ipfs/js-ipfs/issues/17
• Update Roadmap (update next action items)

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

• js-ipfs-repo
  • datastore
  • dir
  • levelDB -> /datastore
  • flatfs -> /blocks
  • might move to arenaStore in the long term
  • itemzise what is on leveldb-datastore (@jbenet)
  • DHT (provider records)
• jsipfs
  • what does ipfs init do (@jbenet)
• IPLD Data Importing
  • write up as specs https://github.com/ipfs/js-ipfs/issues/41#issuecomment-164370167 and rip out go-ipfs pieces (@jbenet)
• Bitswap
  • dag path notation - https://github.com/ipfs/notes/issues/12
  • spec interface (upgrade with dag path notation) (ion is working this)
  • right now JS implementation can use the same 'dumb' approach

[jbenet]

go-ipfs

Lead: @whyrusleeping (@jbenet subbing)

Agenda

• ship distributions
• ipfs-update testing
• fs-repo-migration testing
• 0.4.0 things
• upgrade path and compatibility

Participants

• @jbenet
• @whyrusleeping
• @lgierth
• @dignifiedquire

Notes

Questions:

• what's the 0.4.0 upgrade path? we'll ask users who care.
• what's in leveldb? DHT Records mostly.
• we could implement a kv-store (like pin set) on top of ipfs objects (basically ipfs-blob-store) and transition everything in one go.
• DHT Records will be ipfs objects eventually anyway, under iprs.

TODOS:

• ship new distributions (post to ipfs-update issues) (@jbenet)
• do fs-repo-migration (@whyrusleeping)
• migration path issue (@lgierth, @jbenet)
• ipfs-update tests (@chriscool)
• investigate lo mem repo (@whyrusleeping, @diasdavid)
• shared key transport (@jbenet)

[harlantwood]

@harlantwood hopes to:

• [ ] https://github.com/ipfs/infrastructure/issues/137 - Deploy site x to IPNS when site x source is updated in github

[daviddias]

David Dias TODO

• npm on IPFS
  • [x] feature: registry-mirror npm publish https://github.com/diasdavid/registry-mirror/pull/22
  • [x] feature: registry-mirror npm update https://github.com/diasdavid/registry-mirror/pull/23
  • [x] patch: make it work with large dirs (until we have dir sharding) https://github.com/diasdavid/registry-mirror/pull/26
  • [x] Restart registry-mirror on biham Node (one Castor IPFS repo moves there). Also make sure it is using the latest 0.4.0
  • [ ] Try ipfs add -r with Rabin again with latest 0.4.0 on Jeromy Machine
  • [ ] Check with Jeromy when ipfs tar -r will be ready
  • [ ] Improve installation process, with ipfs update (check with Jeromy) https://github.com/diasdavid/registry-mirror/issues/24
  • [x] Improve demo video Narration based on Juan’s feedback https://github.com/ipfs/blog/pull/12#issuecomment-164564363
  • [x] Make IPFS be 1st class on registry-mirror https://github.com/diasdavid/registry-mirror/pull/20
• js-ipfs-api
  • [x] Make js-ipfs-api compatible with 0.3.10
  • https://github.com/ipfs/js-ipfsd-ctl/pull/42
  • https://github.com/ipfs/js-ipfs-api/pull/169 (^5 for @Dignifiedquire which ended up doing more of the work)
  • [ ] Update feat/files-api https://github.com/ipfs/js-ipfs-api/pull/74#discussion_r48093673
• libp2p
  • [ ] Get TLS 1.3 started/implemented (https://github.com/digitalbazaar/forge)
  • [ ] Look into curveCP (check what is already available)
  • [x] list all the interfaces of libp2p for Juan to ‘bikeshed at will’
• js-ipfs
  • js-ipfs-repo
  • [x] Update documentation, architecture and API interface https://github.com/ipfs/js-ipfs-repo/pull/14
  • [x] deprecate/remove old code, structure stuff https://github.com/ipfs/js-ipfs-repo/pull/15
  • [x] feat/config + tests https://github.com/ipfs/js-ipfs-repo/pull/16
  • [x] feat/keys + tests https://github.com/ipfs/js-ipfs-repo/pull/17
  • [x] feat/datastore https://github.com/ipfs/js-ipfs-repo/pull/20
  • [ ] datastore-legacy (levelDB stuff)
    • [x] leveldb-blob-store https://github.com/diasdavid/level-blob-store
  • [ ] js-ipfs init
  • [ ] js-ipfs daemon
  • IPLD Data Importing - This depends of the spec being written
  • [ ] BitSwap
  • Work on the dumb JS implementation (sending want lists
• ipfs levelDOWN (hope)
  • https://github.com/ipfs/notes/issues/72
• extra
  • [x] Berlin logistics (for me and Jeromy)

---

@jbenet there are some things you can do that will help me a ton if you manage to go through them quickly, these are:

• Itemise the things available on the leveldb - /datastore on an IPFS Repo
• Steps that happen on a IPFS init (this could be part of IPFS Spec)
• Elaborate on your comment https://github.com/ipfs/js-ipfs/issues/41#issuecomment-164370167, making it a spec
• make a call on https://github.com/ipfs/specs/issues/32, so that we can have full npm working on IPFS

[RichardLitt]

This Week

• [x] Sprint Management
• API
  • [x] Stub out the entire IPFS api at ipfs/api. PR api/pull/13
  • [ ] Clean up and get the API working on Apiary.

[dignifiedquire]

Todo

• [ ] getting IPFS distributions ready for shipping
• [ ] Work on rust-multiaddr
• [ ] CR api doc PR from @RichardLitt
• [x] Move apiary source to ipfs/api

[whyrusleeping]

TODO

• [ ] fs-repo-migration shipped
• [ ] dist.ipfs.io 'build things ourselves' shipped
• [ ] new ipfs add make fewer objects
• [ ] discuss directory sharding
• [ ] look at modtimes in mfs

[ghost]

@lgierth TODO

• dev0.4.0
  • [ ] discuss migration strategy
  • [x] utp for dev0.4.0 bootstrappers ipfs/infrastructure#133
  • [x] test fs-repo-migrations
  • [ ] more dev0.4.0 bootstrappers
  • [~] empty and scrub sirius and castor
  • [x] update metrics images ipfs/infrastructure#141
• go-ipfs
  • [ ] harden gateway path prefix
  • [ ] fix gateway IPNS resolution recursion
• infrastructure
  • [x] order domains
  • [x] castor data loss

[chriscool]

@chriscool TODO

• ipfs-update
  • [x] ⭐️ write more tests
  • [x] started fixing ipfs-update
• go-ipfs
  • [ ] get docker image test merged (PR https://github.com/ipfs/go-ipfs/pull/2064)
  • [ ] write more tests

[jbenet]

@jbenet TODO

• [x] new root for distributions: https://github.com/ipfs/ipfs-update/issues/7#issuecomment-165025573
• [x] human interfacing
  • [x] talked with btrask about files/blocks
  • [x] talked with parkan
  • [ ] update https://github.com/ipfs/notes/issues/85
  • [ ] spec out ipfs-chain aggregation tool
  • [ ] spec out timestamping to btc tool
• [ ] itemize what's in go-ipfs/$repo/leveldb
• [ ] upgrade repo spec with "state only in ipfs objects"
• [ ] write up transition out of leveldb plan for go-ipfs
• [ ] finalize go-ipfs 0.4.0 upgrade path
• [ ] resurrect osx earth node
• [ ] spec doc for IPLD importing https://github.com/ipfs/js-ipfs/issues/41#issuecomment-164370167
• [ ] begin kv-store on IPFS discussion
• [ ] add papers to read to pub/sub https://github.com/ipfs/notes/issues/64#issuecomment-164683760
• [x] ipfs.io/docs/install use releases branch https://github.com/ipfs/website/issues/82

if i told anyone i'd do something and it's not here, please let me know.

[daviddias]

@jbenet what I had for you was:

• Itemise the things available on the leveldb - /datastore on an IPFS Repo https://github.com/ipfs/js-ipfs-repo/issues/21
• Steps that happen on a IPFS init (this could be part of IPFS Spec) https://github.com/ipfs/js-ipfs/issues/42
• Elaborate on your comment ipfs/js-ipfs#41 (comment), making it a spec
• make a call on ipfs/specs#32, so that we can have full npm working on IPFS
• bikeshed libp2p interfaces:
  • https://github.com/diasdavid/js-libp2p-swarm#usage
  • https://github.com/diasdavid/interface-transport
  • https://github.com/diasdavid/interface-connection
  • https://github.com/diasdavid/interface-stream-muxer
  • https://github.com/diasdavid/interface-record-store (less urgent)
  • https://github.com/diasdavid/interface-peer-routing (less urgent)

[rht]

what's the 0.4.0 upgrade path? we'll ask users who care.

• Is there a way to sample/snapshot the distribution of the versions the nodes are running?
• What measure should be used for the network consensus? e.g. 1. the node itself, so that if 51% uses 0.3.8 then that's considered the 'majority', everything else are 'forks', 2. the number/size of hashes of each nodes
• How are hotfixes broadcasted? Perhaps a semver header check + 101 status response can be used here. If a 0.4.0 node talks to a 0.3.5 node it could notify the latter to upgrade due to such and such list of vulnerabilities.

[RichardLitt]

This Week

• [x] Sprint Management
• API
  • [x] :star: Stub out the entire IPFS api at ipfs/api. PR api/pull/13
  • [x] Clean up and get the API working on Apiary.

Done

• [x] Helped plan IPFS Weekly. Comment pm/issues/73
• [x] Added shields to ipfs-specs. PR specs/pull/52
• [x] Updated project-dir with new data for project-repos. PR /ipfs/pull/137, PR ipfs/pull/138
• [x] Wrote a doc on how to manually downgrade. PR fs-repo-migrations/pull/10
• [x] Created and filled out ipfs/roundups
• [x] CR + M awesome-ipfs/pull/20
• [x] Work on awesome-ipfs. Closed issues, opened PR awesome-ipfs/pull/29, awesome-ipfs/pull/30, awesome-ipfs/pull/31
• [x] Added Travis to awesome-ipfs. awesome-ipfs/issues/27

[whyrusleeping]

TODO

• [x] fs-repo-migration shipped :star:
• [ ] dist.ipfs.io 'build things ourselves' shipped
  • closer now, builds dist.json file
• [ ] new ipfs add make fewer objects
• [ ] discuss directory sharding
• [ ] look at modtimes in mfs
• [x] work on mfs locking issues and performance

[dignifiedquire]

@dignifiedquire sync update

Planned

• [~] getting IPFS distributions ready for shipping
• [x] ⭐️ Work on rust-multiaddr
  • [x] enque/dequeue
  • [x] auto generated, published docs to github pages
  • [ ] support for IPFS and onion addresses
• [x] CR api doc PR from @RichardLitt
• [x] Move apiary source to ipfs/api

Extra

• [x] Document hangouts on air
• [x] Upgrade js-ipfs-api to 0.3.10
• [x] Upgrade node dependency for js-ipfs-api
• [x] Upgrade dependencies on station and investigate various issues
• [x] CR on ipfs/awesome-ipfs

Special

• [x] Start benchmarking different request modules in node dignifiedquire/request-bench this is part of a more major undertaking that I might talk about if someone is interested as it is only partialy related to IPFS

[daviddias]

SPRINT CHECKIN

• npm on IPFS 🌟
  • [x] feature: registry-mirror npm publish https://github.com/diasdavid/registry-mirror/pull/22
  • [x] feature: registry-mirror npm update https://github.com/diasdavid/registry-mirror/pull/23
  • [x] patch: make it work with large dirs (until we have dir sharding) https://github.com/diasdavid/registry-mirror/pull/26
  • [x] Restart registry-mirror on biham Node (one Castor IPFS repo moves there). Also make sure it is using the latest 0.4.0 https://github.com/diasdavid/registry-mirror/issues/24
  • [x] Improve demo video Narration based on Juan’s feedback https://github.com/ipfs/blog/pull/12#issuecomment-164564363
  • [x] Make IPFS be 1st class on registry-mirror https://github.com/diasdavid/registry-mirror/pull/20
• js-ipfs-api
  • [x] Make js-ipfs-api compatible with 0.3.10
  • https://github.com/ipfs/js-ipfsd-ctl/pull/42
  • https://github.com/ipfs/js-ipfs-api/pull/169 (^5 for @Dignifiedquire which ended up doing more of the work)
  • [~] Update feat/files-api https://github.com/ipfs/js-ipfs-api/pull/74#discussion_r48093673
• libp2p
  • TLS
  • Started playing with forge, it is really nice, but also complex, wrapping it in a simpler interface that we can use for libp2p is not direct, need a bit more time. Documentation of the effort here: https://github.com/diasdavid/js-libp2p/issues/9
  • curveCP
  • Researched curveCP, learned a lot of nice things (it is really exciting to work on this stuff!). So far, to the extent of my knowledge, there isn't that much activity around it though and it is hard to find implementations even, more info documented here: https://github.com/diasdavid/js-libp2p/issues/10
  • doc effort udt - https://github.com/diasdavid/js-libp2p/issues/8
  • doc effort utp - https://github.com/diasdavid/js-libp2p/issues/7
  • [x] list all the interfaces of libp2p for Juan to ‘bikeshed at will’ https://github.com/ipfs/pm/issues/74#issuecomment-165056950
• js-ipfs
  • js-ipfs-repo 🌟
  • [x] Update documentation, architecture and API interface https://github.com/ipfs/js-ipfs-repo/pull/14
  • [x] deprecate/remove old code, structure stuff https://github.com/ipfs/js-ipfs-repo/pull/15
  • [x] feat/config + tests https://github.com/ipfs/js-ipfs-repo/pull/16
  • [x] feat/keys + tests https://github.com/ipfs/js-ipfs-repo/pull/17
  • [x] feat/datastore https://github.com/ipfs/js-ipfs-repo/pull/20
  • [~] datastore-legacy (levelDB stuff)
    • [x] leveldb-blob-store https://github.com/diasdavid/level-blob-store
    • To be able to finish, I need to know what is actually there https://github.com/ipfs/js-ipfs-repo/issues/21
  • Bitswap
  • Worked on in, specially wrapping my head around it and figuring out how to describe it in a way that can be spec'ed out, so that we can make sure it works the same way in Go and JS. I need to sync up with @whyrusleeping on somethings to make sure we are walking towards the same direction when it comes of passing protobufs + IPLD objects in the wire.
  • https://github.com/ipfs/specs/pull/53
  • https://github.com/diasdavid/js-ipfs-bitswap
• extra
  • [x] Book flights to Hamburg
  • [x] Got a nice shack for me an @whyrusleeping to spend NYE :P
• didn't get to work on (#sadpanda) :
  • IPLD importing tools (🎄 all I want for christmas is a speeeec https://github.com/ipfs/js-ipfs/issues/41)
  • js-ipfs init -> I need something like a spec (or description at least) https://github.com/ipfs/js-ipfs/issues/42
  • Try ipfs add -r with Rabin again with latest 0.4.0 on Jeromy Machine
  • Check with Jeromy when ipfs tar -r will be ready
  • Improve installation process, with ipfs update (check with Jeromy)

[ghost]

@lgierth sprint update

This sprint I managed to lose all of Castor's storage -- yay! :S And sorry again @davidar.

Apart from that dev040 is on the plate. We haven't gotten to discuss the migration strategy with users, I'll get to it right away.

Not really any * from me this week :)

• dev0.4.0
  • [ ] discuss migration strategy
  • [x] utp for dev0.4.0 bootstrappers ipfs/infrastructure#133
  • [x] test fs-repo-migrations
  • [~] more dev0.4.0 bootstrappers
  • [~] empty and scrub sirius and castor
  • [x] update metrics images ipfs/infrastructure#141
• go-ipfs
  • [ ] harden gateway path prefix
  • [ ] fix gateway IPNS resolution recursion
• infrastructure
  • [x] order domains
  • [x] castor data loss