Sprint: January 16-23

[RichardLitt]

Note: We skipped a week -- we did not have a sprint call on Monday, January 9th.

Note: Unlike the past sprints, this sprint we are going to change it up a bit. Instead of standing times for talking about js-ipfs, libp2p, go-ipfs, and so on, we will have the All Hands call at the normal time, and then a slot for an hour-long sprint call discussion to discuss the next week's work on a given sprint (based on scrum methodology). After this, there will be free time to allocate - as needed - to other issues raised in the All Hands call. This means that, as a community member, if you'd like to raise any topics to talk about, be sure to mention them in the All Hands call and put them on the agenda first. Thanks!

---

For information about these calls and how they work, read the ipfs/pm README.

Calendar

Zoom and Stream links will be updated directly before the calls. Links to them will also be dropped in IRC, at the start of each call. If the link below seems to link to this issue, the link has not been updated yet.

Video Conference link

Streaming link

Monday

Endeavour	Lead	Time (PST - UTC - CET)	Pad
All Hands Call	@lgierth	9:00 17:00 18:00	agenda and notes
Sync Betwen Sprints	@flyingzumwalt	10:00 18:00 19:00	agenda and notes

Tuesday

Endeavour	Lead	Time (PST - UTC - CET)	Pad
Sprint Planning: data.gov (aka 300 TB Challenge)	@flyingzumwalt	10:00 18:00 19:00	agenda and notes

Not Yet Scheduled

Endeavour	Lead	Time (PST - UTC - CET)	Pad
Sprint Planning: Browser accesses go-ipfs content	@diasdavid		agenda and notes

Please add any agenda items you have to the pad before the project-specific sprint call starts.

Note: Because the Notetaker and Moderator were not added to the All Hands notes, I have added @lgierth and @diasdavid as Lead and Notetaker for this week's meeting.

[flyingzumwalt]

FYI: We are thinking of changing the format & schedule for these weekly sprint calls. @RichardLitt will provide more info when the schedule is clear. In the meantime, the essential info: we will have an hour long All Hands meeting as usual, but the rest of the schedule is likely to get rearranged or scrapped.

cc @kumavis @gavinmcdermott @SidHarder @doctorcrunch @keks @kevina

[RichardLitt]

I've changed the sprint to reflect this: currently, we have one All Hands call planned for monday, with a sprint call afterwards, followed by free time which may be allocated to anything which comes up during the All Hands call which warrants further discussion. Let me know if you've got any questions!

[flyingzumwalt]

We need to push the 300TB Challenge Sprint Call back to Tuesday. @whyrusleeping will be traveling on Monday. Let's do a brief check-in on Monday to figure out what @lgierth & @Kubuxu can work on before Tuesday's "launch" call.

cc @jbenet

[jbenet]

Some things to do that would help, before tue:

• review existing filestore stuff
• review ipfs-pack draft proposal -- https://github.com/ipfs/notes/issues/205
• make a short list of the "big bugs" and "big optimizations" relevant for this sprint. (i can think of a couple -- file attrs, bitswap supporting paths (kills so many RTTs) -- , but we'll want a good list to have in mind)
• refine the concrete use case and UX we're shooting for
• create the test workloads + scripts we'll work against:
  • for each order of magnitude of total data, from 1M -> 100TB
  • with giant files
  • with lots of little files
  • can just add tests to https://github.com/ipfs/fs-stress-test
• review ipfs-s3 datastore options -- https://github.com/ipfs/notes/issues/214

[flyingzumwalt]

Thanks @jbenet. I've added that info to the data.gov sprint issue. I'm already working on some of the things you've listed, and created issues for a couple of them in the ipfs/archives repo.

[flyingzumwalt]

@diasdavid are you going to have a call for the "Browser accesses go-ipfs content" sprint tomorrow? It wasn't on the schedule so I added it, but now I'm wondering if you left if off the schedule intentionally.

[kevina]

Note, in case I am needed: I probably won't attend the all hands meeting but will be available that day on IRC. I will not be able to attend any meeting on Tuesday unless its after 22:00 UTC. I will be available Wednesday.

[daviddias]

@flyingzumwalt I see I miscommunicated my proposal. What I intended is to have 1 hour to sync between sprints, since we have at max 3 at the same time, it means each group has 20 minutes to update where they are and sync any missing detail.

I don't think we need a space for intense discussion about the sprint, as we will have that later in the week and we will be constantly checking in. The benefit of having a whole team check in, to me, is that we see how others are collaborating (now that we are in the adopting phase) and we promote productivity and proactivity by getting learning how everyone is going.

[ghost]

I'd rather not lead the all hands call since I'm getting sick and my throat hurts. Can someone else cover?

[flyingzumwalt]

@diasdavid there were multiple proposals. I've had trouble keeping track. For today's call, I'm fine with spending the hour (or less) after the all-hands doing a "sync between sprints". We can also use that time to settle on a good format for the monday calls.

I was running on the idea that after the all hands the sprints would, in parallel, do something like a Sprint Planning Meeting. When done right, those take less than an hour but it would be hard to fit one in 20 minutes.

[RichardLitt]

@lgierth I got it. :)

[kevina]

Tuesdays are bad for me. But, unless there is a major misunderstanding, the data.gov sprint involves me. So I will see what I can do to attend. For the record (and not aimed at anyone in particular) I do not appreciate the suddenness of this Sprint or the fact that I was never directly notified.

[RichardLitt]

@kevina Understood. Thank you for being honest.

[flyingzumwalt]

Moved the sprint planning call for data.gov to 10:00 PST/18:00 UTC because @jbenet and @whyrusleeping will be sleeping until then. That's as far back as I can push it. @kevina after the call I will have a better sense of how this sprint impacts you.

[kevina]

@flyingzumwalt I completely understand. Unfortunately at that time it will be even more difficult for me to attend. If at all possible please make sure to record it. I might be able to follow up via GitHub or IRC later today.

[RichardLitt]

@flyingzumwalt Do you think we ought to have video calls next Monday?

[flyingzumwalt]

@RichardLitt we should at least have the All Hands as usual, possibly with some demos. I think the team working on data.gov will only need a standup call (15-30 minutes) after that. @diasdavid will your team need a longer video check-in on Monday?

[RichardLitt]

Sounds good. Will create a sprint issue, then.

[jbenet]

Hey @kevina -- there's a lot moving very suddenly, so i'll try to summarize the missing context behind the sudden changes. Sorry, this all came out suddenly. Please excuse the very lengthy and wordy outpouring here; if I had more time, i would be more succinct and clear. Hopefully it provides you the context you need. Please know you are very welcome to be involved in the sprint, and that I did not assign you because (a) I hadn't gotten a chance to speak with you about it and ask for your involvement, (b) details are still being figured out, and (c) you and I have to have a discussion about the changes I require.

Background before Dec

• Filestore is one of the most requested features
• You've been pushing for it strongly, with several iterations
• Your implementation has been stalled due to my lack of time & focus on it
  • I needed to review this very carefully, see if interfaces make sense long term, or what needs to change
  • I couldn't prioritize it higher than other things last 2 quarters.
  • Sorry :( -- it's unfortunate that so much depends on my review atm, but I would much rather make sure we have a fantastic go-ipfs than potentially cause damage.
  • also quality of the 0.4.x series rose and fell here and there, so we're more wary of merging very large things without in-depth review, particularly mine for major things.

Background during Dec

• The results of the US election sparked an important effort to back up huge amounts of climate (and other) data. There were strong indications and precedents that new administrations have cut funding and scrubbed data for initiatives they are against. This is a very sudden, very dangerous position for many critically important datasets. Preventing scientific data destruction and abuse is one of the core reasons the IPFS project started, and a mission many of us care deeply about.
  • There are many articles about this. two recent ones:
  • https://insideepa.com/daily-news/trump-transition-preparing-scrub-some-climate-data-epa-website
  • http://www.ppehlab.org/blogposts/2017/1/15/datarescue-philly-builds-datarefuge
  • The bulk of the work on this is being done by other institutions. We're helping a bit by cleaning up and upgrading our tooling for these use cases, but the heavy lifting is done by other people. Our goal here is to make ipfs help as much as possible.
• As you may know, Filestore is one of the key features missing for such archival and data distribution efforts. Therefore, it rose in our (and my) priority list dramatically, and suddenly.
• At a Protocol Labs gathering, we discussed certain things evolving in the IPFS landscape, in particular this effort and our ability to help.
  • You'll be happy to know that Filestore came up repeatedly, and that we figured you would be happy to see this get significant work and closer to merging to master.
  • We are still discussing whether and what we're going to do, though. We're not yet sure what and how much is highest priority. Lots is needed.

This sprint

• Due to lack of time, we have to act fast. We had to pull this up into first thing this quarter, as inaguration day is imminent and these datasets face could suddenly disappear. The institutions doing the hard work are doing it now, and if we're going to help, it has to be now.
• In a few short in-person discussions (sorry, we all know very well that in person discussions are bad for everyone not present, but in this case the urgency called for it), we decided:
  • that I would review your existing implementation, other proposals, and come up with a plan.
  • i've done this, and can give you review, and thoughts about other proposals. i'm writing some up, but as said above, call will be best.
  • that we would create this work sprint, with a few people committed to it, and open to absorbing more.
  • You and others are welcome to join, though note that this requires a large time commitment, and requires sticking to the plan.
  • that we would create a draft 2 designdoc, submit it for everyone's review and discussion (in particular review from you, @whyrusleeping, @lgierth, @jbenet, @flyingzumwalt, @kubuxu, and maybe others who have show interest in this feature set), with the understanding that the clock is ticking and we rather get a solid draft 2, and prep for a draft 3 to replace or extend it as needed, than to spend a lot of time discussing and aim for draft 3 now.
  • i'm still working on this o/

On the current filestore interfaces and implementation:

• I did my review of your implementation and other possible implementations during a flight, so i did not have access to take notes directly on github as I reviewed.
• There is more, but the gist is that I found serious UX and systems problems with both the interfaces and implementation of the current draft.
  • I'm happy to go over those with you, but I'd like to do that in a call given my time constraints, the time sensitivity of this effort, and the nature of the issues I found. I don't have the bandwidth or luxury to discuss all that in depth over async github comments at this time. I will try to get at least notes written up before a call.
  • I will remind here that this feature is one of the most pernicious filesystem things we face, and I have long avoided working on it (before you started your implementation), because of how horribly difficult it actually is to get right. There are a ton of UX and systems challenges, that many popular and seemingly good solutions get wrong. This is very hard.
  • I think we require a re-design of core parts. Particularly when it comes to how this interacts with ipfs-pack, ipfs-cluster, and other tooling.
• Other suggested alternatives and solutions did not show as much promise though, so at this time I still think using some of your draft as a base is the right way to go -- so see the effort i'm calling for as a draft 2. It will get a lot of design and implementation changes. You are very welcome to take part, but note that we are not going to be beholden to existing decisions or code, and we are going to be very focused in achieving the right UX. that is the least we can do for our users.
  • I don't think what we'll get to after this week and next is final, but it will be an improvement and at least give us a cleaner path to merging. I am interested in your and others' participation, both in improving the design and implementation, in testing everything, and in feedback going fwd. (what should Draft 3 be? how close are we to merging? are all UX problems covered? what failure cases are there, and how do we handle them?)
• I hoped to publish a draft of the changes I see necessary before the start of this week, but I haven't been able to. I have too much going on. I will hopefully get to it by tomorrow or the next day. We should discuss together afterward
• Get involved in this sprint if you would like to -- it's up to you to self select into it; we cannot assign people without first talking to them, and talking to them involves (a) posting all this material (which you've seen and responded to), (b) and having the conversations mentioned above. The posts about this are the first you've seen, the effort is very new, and we're still figuring things out. Sorry the times for the calls didnt work for you, but there's a lot of people, and a lot of timing considerations. Hopefully we can do a call that works for you this week. (Friday is my only option unfort.).

[kevina]

@jbenet anytime this Friday would be fine for me.

[ghost]

@diasdavid will your team need a longer video check-in on Monday?

Yes we'll do the same kind of check-in as three days ago -- we probably won't exhaust the full 60 minutes though.

[kevina]

We should probably take this off this issue, but for a lack of a better place:

@jbenet

but the gist is that I found serious UX and systems problems with both the interfaces and implementation of the current draft.

My coding style can sometimes be a bit unconventional. I have a felling that these "serious" problems you found may be due to some misunderstanding that I would like to clear up ASAP. But before I can do that I need to know some examples of what they are. Also it would be helpful to know what version you reviewed. There is the basic implementation, https://github.com/ipfs/go-ipfs/pull/3368, and the complete implementation at https://github.com/ipfs/go-ipfs/pull/2634.

[jbenet]

@kevina yeah maybe email me :smile: 👍