[Umbrella] 1.17 Release Retrospective Action Items

[justaugustus]

(Starting this off as a dump of the retro notes. @lachie83 and I will clean this up and create issues later.)

(+) What went well in 1.17

• There was a lot of communication between sig release and sig scalability. Scalability made sure to always have an update by Monday’s meeting [alejandrox1] [+1 guineveresaenger]
• Seconding sig-scalability communication - we (CI signal) ended up doing asynchronous syncs and promptly reach out to sig-scalability with every issue, they were also keeping an eye on the pipelines and overall the cooperation was a very good experience [alenkacz]
• Maria attended SIG-scalability’s meeting, and we gathered a doc with relevant topics to bring up. Meeting them on their turf was successful. [guineveresaenger]
• Mostly very very good reaction time from other sigs with regards to the new issues in CI pipelines. Reaching out directly in slack worked well. It helped us keep the blocking dashboards green for a long time [alenkacz]
  • In general most people are much more responsive on slack
• For CI signal, picking people to cover multiple timezones was a great move, we had a great coverage in the meetings even during US public holidays [alenkacz]
• I think Enhancements had a good number of shadows to make the work load distributed well [jeremyrickard]
  • 4 shadows + 1 lead
  • Multiple shadows
  • Active shadows sharing the workload
  • All shadows got to practice the role thoroughly
  • Role includes a lot of cat herding for updates, so distributed workload is more successful.
• Iterative improvements in the Enhancements tracking sheet cut down on missing items that should be tracked and more evenly spread the efforts across Enhancement team members. [mrbobbytables]
  • [stephen] make sure to talk to other leads (who also use spreadsheets) to work together on ways to improve tools/automation.
• We had very distributed time zone coverage [guineveresaenger]
• SIG-Docs and SIG-Testing, among others, were very responsive throughout the release [guineveresaenger]
  • [stephen] also kudos to Release Engineering team for cleaning tests and test dashboards
• Starting the release before 4pm US Pacific was overall a good decision, as it left many of us awake and alert to touch up wrinkles. [guineveresaenger]
  • Need to insure the process is being worked through while people involved can reasonably be around
  • Need to have time to fix up surprises without driving people into their dinner/family/life time.
  • Release notes final draft needs merged before the final release build, role handbooks docs don’t clearly reflect this.
  • [stephen] proposes no releases on Mondays. Any preparatory work that can be done opportunistically then happens on the weekend, and may give folks a sense of obligation to work on their weekend. Targeting a Tuesday/Wednesday can mean initiating final prep is starting Monday and being worked on a normal work schedule.

(+) What could have gone better in 1.17

• [x] Original Doodle had a majority voting to meet Monday morning in Pacific timezone. This was very hard for some people. We need to do another Doodle. Can we return to have weekly release team meetings on Tuesdays [guineveresaenger]
  • It is useful to front load discussions at the beginning of a week.
  • (reminder this also means since the retro is usually two-part, part 2 will be Monday 8am US Pacific Time Dec. 16)
• SIG-Docs would like to see the Docs placeholder PRs put in earlier so they can review and work through the list sooner. Potentially add a message as Enhancements reaches out to people? [jeefy via SIG-docs]
  • [x] Discussion thread: https://github.com/kubernetes/release/issues/956
  • Also discussed in Dec. 9 Release Engineering recording and meeting minutes.
  • Docs workflow may be considering a “rebase workflow” for their branches.
  • Another option is to do the same as release branches with more regular branch fast-fowards.
  • [stephen] shout out to the Release Engineering folks working to shift the bash release tools to more generic, testable, golang which are repo/branch specific instead of k/k:master or k/k:release-N specific.
• Run scalability tests on non-latest versions kubernetes/kubernetes#86134 [alejandrox1]
  • Assess version markers on release branches: https://github.com/kubernetes/sig-release/issues/850
  • Need more flexibility to understand if a performance issue is in the release branch or just master
    • This also came up in the 1.15 and 1.16 retrospectives
    • Cost is a concern and is undefined today given the Google hosting. As this shifts to CNCF hosted and WG K8s Infra progresses, we may better understand and find ways to fund this.
    • [stephen] if we cut the release branches sooner, could this help?
• [stephen] project has historical implicit assumption of linux is the node. Community-wide people need to understand that windows is an option too and not a secondary thing. There are also skills gaps as the average contributor may not have the right experience or knowledge to implement Windows coverage for their thing. As a community we need to figure out how to address that gap, beyond the release team.
  • Sig-windows test was failing even during the 1.16 release iteration, CI signal spent whole 1.17 trying to push this fix via various channels and we still ended with that test still be constantly red https://github.com/kubernetes/kubernetes/issues/80534
    • (probably need a better channel with sig-windows?) [alenkacz]
  • [michaelmichael] was anybody from sig-windows active on the issue? Yes, including a SIG chair.
  • [alejandrox1] in a way this is an example of a good problem to have as it showed areas we were/are lacking coverage for Windows node testing. There were a lot of issues stacked up and were being iteratively addressed. It spans SIG Windows, Node, Testing, WG K8s Infra, and others.
• I would like to question the need to write CI signal reports. We did not really do it this iteration (and it seems nobody missed them that much?). They are pretty labor intensive to make the worth reading, the same report is written 3-5 times a week into the release team meeting notes and we preferred actively reaching out to people in case of problems instead of writing report and hoping someone will read it and react. Definitely worth discussion what to do with the reports in the future releases [alenkacz]
  • [stephen] there used to be a “go signal” report, similar to what’s in the release team weekly meeting minutes. If the report is not going somewhere useful, hitting the right eyes, getting responsive action, then it’s not useful. But if we shift to more asynchronous collaboration in the release team (versus synchronous meetings), writing consistent reports will be important.
  • [alejandrox1] persistent artifacts and reports are important as team becomes more distributed, more asynchronous, and also for the historical record. On prior releases people did report getting positive responses to the report. And generating it was a useful practice to walk through for personal understanding and demonstrating/teaching/learning along with shadows in the role.
  • Report generation should be more push-button automated.
  • [tpepper] can feel redundant building the context of status, but the steady state practice is what prepares you for how to handle the emergency situation when things are breaking. The practice is valuable.
  • [stephen] Nevertheless please please do question the value of actions you see in the role handbooks (and everywhere else in the project docs also beyond release team).
  • [alejandrox1] Alena wrote a cool tool that helps generate the issues list. (this documented, but is also living outside the k-org, can we bring it in?)
  • [alejandrox1] Writing a small, short summary of the current status of issues is really useful for folks who aren’t in a given meeting where it is discussed. And provides historical record.
• I think it would have been helpful for each team to have an initial call to introduce everyone, their collective tasks, and what the upcoming timeline was going to look like. As an example, because we didn’t do this, I wasn’t aware of the time zone issues my team was dealing with until halfway through the release cycle. [karenhchu]
  • Important for the team to know who’s who and who’s where so they can more efficiently get up to speed meeting and sharing load.
  • [stephen] this is an expectation of all role leads and should be clearly documented and delivered too
  • [guineveresaenger] bring issues like this up ASAP with the emeritus lead. If anybody sees an unexpected gap, bring it up with the emeritus lead. They’re a safe space to question and propose.
  • [stephen] Lachie, Claire & Stephen met recently to discuss enhancing the emeritus role.
  • [claire] it might be beneficial to have a doc describing the general obligations/expectations for all role leads in common
  • Is this side-stepping around your role leader to talk directly to the emeritus lead? Some concern for this. But also expression that the emeritus lead represents a safe space if somebody feels they need that. The team should feel flat enough that leads speak to each other regularly, including with the emeritus lead, and individual contributors should feel free to talk with anybody in the team.
• I think this retro is too bike-shed-y. Expecting two parts is something that happened because people were talking a lot about ideas that should be captured in issues. [guineveresaenger]
  • Conflicting views on whether the retro must be two part and how much time needs to be dedicated to discussion versus punting clarification to an issue.
  • The emeritus lead does have the responsibility of collating this doc into new issues.
  • Do issues need to be opened ahead of the retro?
  • Dual meetings at different times can give ability for other timezones
  • We do have a meeting moderator who can declare when we’ve discussed enough to punt an issue and move forward. The document indicates we timebox five minutes max per topic.

Part Two started here (Monday 8am US Pacific Time Dec. 16)

• Clarity around timing of merging release notes and the actual release. This release was out of sync and required additional work to fix. This can be resolved by updating the docs to better detail the process [macintoshprime][+1 guineveresaenger]
  • [x] https://github.com/kubernetes/sig-release/pull/916
  • Release team lead handbook does have a notation on this, but it wasn’t as clear, AND it was not connected to the Rel Notes Handbook
  • “The following final actions must occur in order, with successful completion of each being the entry criteria to the next."
  • Cross linking these handbook points might be beneficial
• There was some confusion between docs and enhancements on who should respond regarding issues (e.g. being removed for not having docs placeholder PRs) [mrbobbytables]
  • [stephen] After enhancements freeze, anybody can handle these
  • [tpepper] Do we need a “somebody must handle these” instead of “anybody can handle these”?
  • [guin] Could opposing time zones between docs and enhancements leads have been a contributing factor?
  • [bob] there were asynchronous notes passed, but close to deadlines having more synchronous chatting to double check things would have helped
  • [JimAngel] especially at crunch time at end of release could be a good idea to have non-Pacific time zone sync meetings (this should be a bullet point further down)
• There were numerous merge conflicts with docs dev-1.17 branch and master at the end of the cycle. [mrbobbytables, Jim Angel]
  • Branch syncing across master and release branches had an issue as a significant translation effort was happening on the release branch for their dev work.
  • In the future this work will happen on a unique branch and merge independently.
  • Docs wants to leverage the release git automation in their repo/branches.
  • [x] Ref: https://github.com/kubernetes/release/issues/956
  • Might need a few tweaks to krel and/or their workflow, but looks like this combination would enable docs to focus on their actual reviews and not these mechanical git tasks
  • Need playground for Release Engineering: https://github.com/kubernetes/org/issues/1474
• k8s PRs vs website PRs need to be in sync: web docs need to not merge if the k8s enhancement PR has not merged. If milestone label is changed on a k/k enhancement, today it is a manual process to check if the milestone for website/docs are in sync. [irvifa]
  • Does this require a new label?
  • Can a dashboard track changes? Automated report generation?
  • Easily track docs and tied to a release + easier signaling if docs are needed for a specific enhancement, If there is no docs PR added to the metadata by docs deadline, we remove them
    • Probably moving it to toolings and metadata
  • How can this be happening earlier, instead of being discovered later? How to tell if incoming PR is user-facing change that requires docs (and hold the k/k PR until the docs one is ready?)?
  • Need to brainstorm enhancing this part of the enhancements
  • Example KEP for linking k/k and unit test can be found in https://github.com/kubernetes/enhancements/issues/617. Probably should figure out how current metadata for KEP, PRs for k/k and k/docs.
  • Most (80-90%) of tracked enhancements do need docs. When an enhancement is targeted to a milestone there should be an automated way indicating the docs changes requirement(YES/NO). So that it helps docs release team to track the enhancements better rather waiting for the enhancement owner to reply back.
  • Anything that changes the contributor experience for everybody in the community (eg: multiple new additional /needs-foo labels) needs consideration. If it’s primarily about 20-40 enhancements and impacts all 2000-3000 PRs in a cycle, that may be onerous for the broader community.
• There were a few items that ended up missing code freeze and needed exceptions and last minute scrambles to remove/include things in the milestone. [jeremyrickard]
  • This happens every release.
  • [guin] The team is already a bit fuzzy between intended time of freeze and actual code enabling freeze, in order to be pragmatic around conversations with folks and the depth of the tide queue.
  • [stephen] Would bringing back code slush help?
  • [guin] Is there data that during slush we had less exceptions?
  • [tpepper] Few understood the slush concept
  • Does it need understood? Is pragmatic gap between declared start of freeze and code enforcing of freeze already this?
  • Why was it removed? It was not understood, it was not enforced, it had no explicit criteria, it had no mechanical backing. It was only a weak ask of all reviewers/approvers to start dialing back their merges ahead of freeze, instead of using the impending freeze as a signal to get anything and everything merged asap and fix it later after freeze.
  • What are the exception granting criteria? Is the process onerous today? It is lightweight and pragmatic and usually the applicants are obvious, eg:
  • Didn’t know there was a schedule, saw it’s possible to get an exception, weren’t ready...declined.
  • Knew there was a schedule, were within a day or few, had track record of working with the release process...accepted.
  • [guin] Feels like process is actually working sufficiently today.
• Time zone representation was good, but meeting times did not reflect this. I should have gathered preferences via a poll rather than by informally asking “is this time ok for everyone”[guineveresaenger]
  • Do more explicitly asking for preferences and use multiple forums including checking in with individuals.
  • Individual hesitation in a group setting may not be clearly expressed or seen
  • Do a Doodle
  • Moving from 10am to 9am helped some, but wasn’t enough.
  • [tpepper] Release 1.12 (and a few later?) cycle had the primary weekly meeting but also the next day a very different time to pull in more of the others. Somebody (multiple?) need to be in both meetings to get cross pollination of discussions. Can be hard if required of a specific role to be up very early or very late to run the alternate meeting.
  • [guin] Leads/shadows need to more explicitly keep each other up to date when not at meetings.
• Sharing a zoom account was a challenge when the SIG meeting was adjacent to the release team meeting. [guineveresaenger]
  • With the SIG Release and Release Team meetings adjacent, Zoom was confusing.
  • We had a broken link or the Release Team Meeting
  • We had Monday morning fog. This led the Doodle, but was notably hard on half the people too.
  • We had a shared host key (which couldn’t be used twice in parallel?). The host key is per account, not per meeting.
  • [bob] CNCF has had similar issue and has created multiple accounts so a SIG can have multiple meetings at the same time.
  • [tpepper] SIG Cluster Lifecycle has a loooot of meetings. How do they handle it?
  • [jorge] They’re super diligent on ending right on time.
  • [stephen] They have their meetings non-overlapping.
• The calendar and google group permission setup was very much not self-serve, nor obvious. Could be fixed with better docs [guineveresaenger]
  • [guin] new onboarding template for release leads and branch managers may help
  • This is in the release lead handbook, but semi-ambiguous and assumes certain prerequisites. Does require some SIG chair permissions and depends on SIG ContribEx docs (now out of date) around chair management of calendars.
  • Our docs need to be explicit on how incoming Release Lead interacts just with SIG Release chairs to get the needed things.
• Several SIGs were not responsive to the point that enhancements slipped due to this, and I am hoping to trigger a better process for holding SIGs accountable https://github.com/kubernetes/community/issues/4289[guineveresaenger]
  • Attitude of ‘we can just cherry pick it in later’ was disconcerting :| [mrbobbytables]
  • Attitude of “this isn’t tied to a release milestone anyway” was disconcerting and baffling as well [guineveresaenger]
  • Does this need escalated to steering? At a minimum it is a Contributor Experience issue, as it is a culture issue in our community.
  • Part of the answer is being more clear on what release team expects for work in the milestone. Part is recognizing and being okay with some things not being tracked as enhancements.
  • [stephen] Would always requiring /milestone have helped? Unlikely as people->process->automation. If people don’t understand today, they wont understand the change either, and simply apply the label without doing the other cultural interactions desired.
  • [guin] Have received apologies from multiple SIGs.
  • [guin] Have received queries from SIG chairs about how to get on the SIG chairs mailing list to hear about what they’re supposed to be doing(?!).
• Several times, I woke up on Monday morning to release engineering and release process decisions that were started on Saturday, and finalized on Monday in an earlier time zone. While it was nice to see work done, not only did this keep me out of the loop, but it also is a bad example for overall team mental health (don’t work on weekends)[guineveresaenger]
  • The release team lead handbook indicates one should expect to work on the weekend. This is a poor requirement of our volunteers. Sets bad example.
  • Relates to discussion of not having release team meeting on Monday.
  • Relates to discussion of not making release on Monday.
  • Volunteers may volunteer to work on the weekend, which is nice.
  • PRs come in, are planned for review, and before they are read, they’ve been merged.
  • How do we balance merging efficiently and iterating, versus having more people involved with reviewing and discussing and aware of the current state of what is merged?
  • Pre-staging of release was great. But it happened on the weekend. Can’t make that as an expectation.
• Test infra code freeze to aid in stability for the kubernetes release kubernetes/kubernetes#86135 [alejandrox1]
  • This has been discussed quite a bit and didn’t get great traction in SIG Testing
  • [stephen] Need to tighten the proposal. Test infra needs to be able to move. The portions that notably impact release need more control.
  • [jorge] How do we get quality signal and feedback around changes, or even know a test infra change was made to watch for changed signal?
  • https://github.com/kubernetes/kubernetes/issues/86182
  • [stephen] Even simple changes to the system can be devastating.
  • [tpepper] Need more culture of understanding your stakeholders and being interlocked.

(🔺) What will we do differently in 1.18 (and beyond)

• Work towards adding real aka non-synthetic cluster e2e tests https://github.com/kubernetes/sig-release/issues/829 [guineveresaenger]
  • Jorge has visibility on this issue
• Improve meeting times for non-US timezones [guineveresaenger]
  • Jorge/Alena discussing this
• CI signal weekly reports: should we keep doing them? [alejandrox1]
• No need to add a Scalability liaison role to the release team [stephen]
• Need more discussion between lead roles who use spreadsheets to work together on ways to improve tools/automation [stephen]
• Tune the role handbooks and process synchronization points to shift more work earlier. Release 1.18.0 day’s release cutting work should not start at 4pm Pacific, but more finish then [guineveresaenger]
• Fresh Doodle for time for the weekly Release Team meeting. [stephen]
• Determine what communications (eg: CI signal report comment from [alenkacz] above) can be informal/asynchronous and which need to be more formal reports, and what frequency. [alejandrox1]**
• Aim for a more efficient retro meeting, timebox discussion, pragmatically allow it to continue or agree to move it to an issue. Retro documentation needs to be updated to reflect this. [chrisshort]
  • After last week’s retro discussion on the community call, multiple retrospective retrospectives happened last week.
  • [stephen] This document is a key tool for the emeritus advisor and needs to be as complete as possible. Worries that insufficient time is given to discussion towards making the discussion actionable. Need to balance experienced leads discussing things and also looping in shadows. Doesn’t want a separate discussion focused on shadows. Retro at community meeting is done to be as inclusive as possible and need to keep that focus. More frequent touch points before the retro should be the path to better enabling shadows.
  • [guin] This is similar to needing to have different meeting times. Different people communicate in different ways, which does require different venues for discussion.
  • [guin] Aiming for higher efficiency is a matter of degree. Not trying to shut down conversation, but be able to agree that we’ve agreed enough for now or it’s a large enough conversation to take a lot of time in another forum (meeting, issue, …).
  • [stephen] If the emeritus advisor is the retro facilitator, they have sufficient seniority/experience to see that conversation could move elsewhere.
  • Individuals can also off to the side just open issues, and this has happened during today and last week retro meetings.
  • [jorge] Advertise the retrospective more ahead of it. Explicitly reach out to shadows. Ask them to make a note, or make notes on their behalf, in the retrospective document.
  • [jorge] Can the retrospective document be made to read more like a story? Go back to 1.14 and you’ll see “code slush” mentioned once briefly as being removed. “Why?” was not tracked from the meeting discussion. You can find it if you go through the video and multiple issue/PR artifacts. Should have the retrospective information summarized completely in one place.
  • [jorge] Leads need to be actively recombining information from multiple meetings/discussions.
  • [stephen] Does initial community meeting need to be longer?
  • [stephen] Can we separate out topics which are global/broad (community meeting retro) versus team local (SIG Release meeting, issues)?
  • [jorge] If we separate too much, other teams don’t get visibility into the release team internals (eg: docs branchff discussions?)
  • [stephen] Could team-local pre-retrospective happen first in SIG Release, then bring a set of global topics for broader discussion to the community meeting retrospective?
• Implement release style release branch automation and branch fast forwarding in the docs repo [SIG Release subproject release engineering, Jim Angel]

ACTION ITEMS

• Open an issue to continue discussing K8s PRs and Website PRs need to be in sync (Vineeth Pothulapati, Irvi Aini)

Full 1.17 Retrospective notes: https://docs.google.com/document/d/1AtZ_81F3E4y_04Gx31mnG5w6AG3E0AXDOuLU7RcUIso/edit?usp=sharing Previous release AIs: https://github.com/kubernetes/sig-release/issues/806 /assign @lachie83 @alejandrox1 /sig release /area release-team

[alejandrox1]

/assign @mrbobbytables @onlydole @alejandrox1 /unassign @justaugustus

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/926#issuecomment-663257880): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[alejandrox1]

I'm sorry fejta, will actually get to it within the next couple weeks :smile: /reopen /unassign @mrbobbytables @lachie83

[k8s-ci-robot]

@alejandrox1: Reopened this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/926#issuecomment-663296510): >I'm sorry fejta, will actually get to it within the next couple weeks :smile: >/reopen >/unassign @mrbobbytables @lachie83 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[alejandrox1]

/remove-lifecycle rotten

[lasomethingsomething]

I've identified what seemed like action items and collected them into a separate doc that will be turned into an issue so we can review all past retro items in one place. We can then determine if the items expressed in this particular issue were resolved or not, or if they should be reframed or combined with more recent ideas.