Sprint Planning Meeting: Wednesday, May 11th

[github-actions[bot]]

2i2c Sprint Planning meeting

This is a 60 minute recurring meeting every other Wednesday at 7:30AM Pacific time (here's a timezone converter, ignore the date!).

Our goal is to synchronize the team on the most important things to work on, and to divide work between one another so we know what to work on next.

• Team backlog
  • list of items with people assigned to them
  • list of items without any people assigned
• Meeting facilitator:

Meeting agenda

Review operations and support items (20 min)

For each operations or support item, we should:

• Assign a team member to items that don't yet have one
• Discuss any quick blockers or major questions that need to be answered
• Discuss the next steps to resolve this item

Review team projects. (30 min)

For each project, its champion does the following:

• Share accomplishments since the last sprint, demos, etc
• Discuss any quick blockers or major questions to answer
• Discuss the work plan for the next sprint for this project

Context share from admin and sustainability (5 min, if time)

A representative that has been working on these parts of 2i2c shares any significant updates or upcoming items.

[damianavila]

@colliand, this is the place to drop your notes when you have time!

[colliand]

Notes on biweekly Team Work Sync

2022-05-11

Participants: DA, GD, SG, YP, JC, CH

Agenda:

1. review of completed work
2. discussion of in-progress work; prioritization

• Investigate using CIlogon vs. Auth0
  • GDC: nothing new here on recent cycle; what remains? Move hubs over; merge PR with OAuthenticator
• Allow configurable retention policies
  • YP: scratch buckets; MEOM scratches don't do that; added docs on how to set up; recent work addresses a request from Ryan A.
• Security backdoor
  • GD: turns out this one was easily resolved because of prior work in JupyterHub

Operation Support Section

• request to install Tensorflow and Keras on same hub
  • DA: replied to U. Toronto; pointed to article on the subject written by DA. Considered complete for now
• Deploy pd-balance change to cloudbank
  • YP: We don't want uniform GPU availability across all hubs
  • cloudbank hubs are used heavily and frequently; upside for Yuvi to be in India is that his daytime overlaps with low usage times
  • GCP Terraform is newly clean and should allow for easier deploys in future
• update daskhub gateway
  • hubs have been updated using Erik's recent version
• incident: m2lines hub unavailable
  • YP: Ryan was giving a demo and the hub flopped; cause of trouble was attributed to Google doing a point update or similar leading to unavailabilty for a few minutes; solution was to switch over to "high availablilty" masters to avoid future blips; CH inquired if "high availabilty" incurs greater cloud cost; chat suggested that there would be higher cost but this was not quantified
  • redeployed m2lines twice and it just worked! That's a good sign.
• Sidecars issue?
  • YP: solved by boosting storage size to 2GB. Other discussion with questions in corresponding issue to be addressed later.
• Pangeo scratch?
  • YP: The resolution of the underlying problem by GD allowed Yuvi to quickly address this issue
• Blog post on CI/CD infrastructure
  • CH reports that the bulk of the post was written by SG. This post provides a summary/motivation for improvements to deployment

Projects section

• Cloud usage monitoring?
  • DA: There is a dedicated board with more details; keeping this issue here to track the overall project as part of the sprint cadence
  • GD: Not much work here during this cycle; PR for upstream Grafana Dashboard; show all the data source at once
• Deploy/operate Binderhub for Pangeo
  • SG: Which cluster will host binderhub for Pangeo? Columbia U. rules limit engineering team access and we want to avoid that; 2i2c controls cloud billing seems to require renegotation or a new subaward; business obstacle!
  • upstreaming persistent hub work with GESIS overlaps here
  • target is an improved Binderhub Helm chart. Some discussion about whether this will even exist in the future. (I didn't grok much of this...)
  • CH: transition plan?
  • JC: Binderhub federation page identifies two extreme ways that an organization can join the federation. Can we run Binderhub with an allow list? This appears to be a product category that some of our customers find interesting. SG: Yes, we can do that. I helped set that up for Turing. Perhaps best discussed in another forum. JC: retreat from this for now....thanks!
• short release plan for pydata theme
  • CH: Another developer is generating PRs here. Goal is to make sure those PRs check out and get them merged. We want to reward this other person by allowing them to keep making PRs and improving things here.
• Define process for helping with image changes
  • DA: this is a discussion issue; work has not happened in this cycle; DA will contribute to the thread to move discussion forward
• Move to use OAuthenticator from upstream
  • GD: switching from pointing to Yuvi's fork. Yuvi has already commented; issue will close soon
• support choosing images in profile list for Pangeo hubs
  • YP: merged next week when Erik returns from vacation
  • may require z2jh upgrades before merging into upstream
• make role-change events more visible/explicit
  • SG: GH issues/actions needed work; used idea from Turing to use Geekbot to improve automated messaging against a time-specified cadence. SG built a Geekbot that pulls a list of tech team members from Slack and then generates role assignments and announcements based on a json file. SG contacted Geekbot team with feature requests (e.g. support biweekly cadence); overcome using some hacks in current version

Engineering operations and support

• move openscapes to Github Auth using teams
  • SG: just swapped this over to YP; it was not touched during last cycle.
  • CH: How toily is this?
  • SG: Work is not hard but requires context that SG could develop with a plunge into the details. YP had that context so can be resolved faster. Removing a member of a Github team member revokes access to the corresponding hub but there is an issue! The team member's access to the hub will persist until the team member logs out. That's not ideal as current approach to resolving that gap in auth control requires a hub administrator to take another action using the hub admin dashboard. There is a "refresh user" traitlet to fix this gap in auth control that needs to built in upstream JupyterHub.
• metrics/reporting for LEAP Hub
  • YP: Ryan A and LEAP team worried about exploding cloud costs
  • A "budget alert" was built that notifies 2i2c freshdesk and Ryan via email if forecast usage will generate a monthly cloud cost exceeding $2K.
  • This alert addresses anxiety over cloud cost.
  • JC: Is the alert agnostic to cloud provider? YP: No, GCP-specific....some discussion
  • YP: Budget insights will be connected to cloud vendor specific tooling since that's where the budget is defined; usage insights can be built to be agnostic using Grafana; workflow will be that budget alert triggers a careful analysis of what's going on via Grafana.
• commit to team meetings on specific day?
  • seems there is consensus pending confirmation from YP
• Dask workers limit
  • YP: quota issue with Google; LEAP hub needs an increase but number of IPs allocated by Google is limited leading to total allocation of up to 69? nodes; m2lines has much higher limits by default
  • YP contacted Google with a request to raise the limit. Request was denied with a footnote to contact sales; YP wishes to explore relationship with Google via the CS&S connection
  • JC (in chat): we should perhaps form an agenda internally and then arrange meetings with the cloud vendors. Contacts with Google and Azure in Canada have indicated interest in supporting 2i2c sales; a hotline relationship with cloud provider tech support seems like something we can develop
• Hiring?
  • CH: down to 3 candidates; plan and appointments for next week's interviews.
  • Check calendar requests!

Org Admin/tasks

• Berkeley and Toronto?
  • JC: emerging definition of "partnership tier" as an abstraction of the project pilot terms for Berkeley
  • CH: Toronto relationship has been more valuable than the revenue it has so far generated.
  • JC: For meeting next week, I suggest we listen to the request from Nathan on grading support and softly pivot by saying "Yeah, we can do that but need to find a way to fairly cover costs" and propose another exchange to redefine the 2i2c + U. Toronto business relationship using the "partnership" stuff.
• Organiztion - leads
  • DA: need to focus some effort soon on the UW Hackweeks hub. Target is June 1 for a GPU-enriched hub.
  • JC: should organize a meeting to define improved information/coordination between sales/partnerships and engineernig. A lead says "I will buy" and we need a process to convert that into a hub launch with onboarding. There is overlap here with the soon-to-be-hired Product/Community lead.
  • JC: need docs to better specify 2i2c's ontology for support. Terms Level 1 and Level 2 are loosely understood but need to be specified, perhaps in the Support Level Objectives page.

[damianavila]

Lovely notes, @colliand! Thank YOU!