Formally agree on release team for Kubeflow 1.5

[thesuperzapper]

I see that @annajung, @jbottum, @kimwnasptd, @shannonbradshaw are taking actions under the banner of a “Kubeflow 1.5 release team”. I don't see how this is possible, as no release team has been endorsed by the Working Groups or @kubeflow/project-steering-group.

I am concerned that having 75% members of the "Kubeflow 1.5 release team" be from one company (@kubeflow/arrikto) is not in the best interest of the Kubeflow project.

• @annajung (VMWare)
• @jbottum (Arrikto)
• @kimwnasptd (Arrikto)
• @shannonbradshaw (Arrikto)

Therefore, as one of the @kubeflow/wg-notebooks-leads I would like to formally object to the proposed release team of: @annajung, @jbottum, @kimwnasptd and @shannonbradshaw.

---

I propose we follow the same process as we did for Kubeflow 1.4's release team:

1. Publicly request for nominations on kubeflow-discuss (see 1.4's thread)
2. Raise an issue with the proposed release team, and request that the @kubeflow/project-steering-group and Working Groups, endorse orobject` to the group (see 1.4's issue)

---

cc @kubeflow/project-steering-group cc @kubeflow/wg-automl-leads cc @kubeflow/wg-deployment-leads cc @kubeflow/wg-manifests-leads cc @kubeflow/wg-notebooks-leads cc @kubeflow/wg-pipeline-leads cc @kubeflow/wg-training-leads

[thesuperzapper]

I have raised the same concerns on kubeflow-discuss in this thread:

• https://groups.google.com/g/kubeflow-discuss/c/ZD3pqGCiTMU

[annajung]

I am not against publicly requesting nominations for the release team. I just want to clarify a few things that may have been missed.

• There was a nomination started for 1.5 which didn’t get a lot of volunteers or endorsement.
• There was a discussion during retro/release team meetings on the best way to move forward with choosing the next release team
  • Decisions made during those meetings were:
    • Instead of community nomination, the previous leads (release manager and release docs lead) should decide the next successor
      • Ideally, the person they choose should come from the release team members without the lead role. This would ensure that the person nominated for the release manager role or docs lead role would be successful in the role as they have participated in the release team before.
    • Release member and shadow roles are not clear at the moment especially with how the 1.4 release team turned out. But, what we want to encourage is that release meetings are open to the community and should allow anyone interested in joining the release team to join
      • Open release team would allow a bigger pool of people to be nominated for the leads roles like release manager and release docs lead
• The proposal for the 4 people you mention wouldn't be the complete list of members for 1.5. Since making the PR, there were 3 more members from the community who were interested in being part of the release team as just members.
  • I think this would also mean that maybe only the leads be added to the github kubeflow/release-team that has write access permission to make sure that permissions aren't shared with too many people
  • I also might have acted too soon on the PR as a way to quickly startup 1.5 with initial members. So, we can definitely close out the PR. that's not a problem :)

@kimonas and I spoke today about how we could have better communicated the retro outcomes to the community and definitely something we could have done better. With the release team first forming during 1.4 and now continuing on with 1.5 and on, I think there is a lot of processes that need to be decided as this was not something that was clearly defined anywhere.

This was also discussed during the community meeting today and while there were no objections from the community during the meeting, we were going to send out an email regardless to gather feedback from the people who weren't able to attend. But, looks like you beat us to it.

Happy to hear feedback on the release team process and will definitely work with the community to find the best way to move forward without impacting the release too much. Thanks!

[aronchick]

Hi @thesuperzapper - thank you so much for raising your concerns. I'm not as plugged in as I used to be, but I do remember the call for nominations a few weeks ago that @annajung called out. I think your #1 issue was addressed via this concern (it's been 2 weeks).

Unfortunately, everything here is still volunteer only and, while it's never ideal to have concentration around a single company (not because they're not good, but because that can lead to a limited set of needs represented). Would you want to be part of the release team? Or have someone else that you know be part of it?

I want to thank all the folks who have started this formal process of release teams - everyone on 1.4 and of course, the folks who stood up for 1.5 - @annajung @jbottum @kimwnasptd @shannonbradshaw.

@thesuperzapper what would you recommend from this point? Without slowing us down too much, and we're not set in stone (nothing prevents us from changing the make up of the release committee at this point) - what would be the alternative? Just ask the leads of each WG to sign off? Or something else?

We're all learning - I don't think there's any malicious intent!

[thesuperzapper]

@aronchick I note that the thread from @kimwnasptd only has one nomination (a self-nomination of @shannonbradshaw), I also note that https://github.com/kubeflow/internal-acls/pull/511 is proposing multiple people who were never nominated publicly.

---

I think the best steps forwards are:

1. Formally agree that the current "Kubeflow 1.5 release team" has not been established yet
2. Put out a new request for nominations:
   • (We can limit the time period to 1 week to limit the wasted time)
   • (People can self-nominate if they like, or nominate others)
   • (I think a sensible requirement is that a non-affiliated person seconds the nomination)
3. The group of ALL seconded nominations becomes the "proposed release team":
   • (The specific roles, like "Release Manager" and "Docs Lead" can be decided by consensus among the team)
4. The "proposed release team" will be presented (as a group) in a GitHub issue for "endorsement" or "objection":
   • (If more than 60% of the Working Groups "endorse" the release team, it is formalized)
   • (The Project Steering Group gets a hard veto)

[aronchick]

I think this is very fair - I would like to hear from other folks. If we don’t hear any objections by midnight on Monday (Pacific Time), I propose the below goes into effect, with the closing of the release team occurring one week from when this goes into effect.

Any thoughts?

[lalithvaka]

It's a fair observation and proposal from @thesuperzapper . I concur @aronchick .

[james-jwu]

The proposal from @thesuperzapper looks reasonable to me.

[ca-scribner]

There's both good practical and ideological points in here.

Ideologically, I'm with @thesuperzapper that an open project that can be driven by a single entity has at a minimum a perception problem, but could also have a real takeover problem (I am not suggestion that's happened here, just saying it could happen). Release management picking future management has the same issue - someone with bad intent could effectively take over the project. But practically, there's also been a void of people stepping up for the jobs, so I can understand the 1.4 release team feeling the need to do something. fwiw, there was a call for more people to join the release team during the community meeting yesterday and it seems like there'll be a few more. @thesuperzapper's proposal of asking again for more people sounds good to me and I expect there will be more interest this time

I don't recall if this was discussed before, but if part of the concern here is ensuring leadership comes from a broad consensus maybe any decisions by the release team should be done on a one-organisation, one-vote basis? Eg: if multiple people on the team are from the same org, they collectively have one vote.

[DomFleischmann]

@DnPlas and I, would like to join the Release team as members representing Canonical. For now, we have added the Release Team meeting to our calendars and will join the next one. If there are any further steps required please let us know.

[kimwnasptd]

First off some clarifications: https://github.com/kubeflow/community/issues/540#issuecomment-977915428

Release management picking future management has the same issue - someone with bad intent could effectively take over the project.

The release team does not have any authority over any asset of the project. The team leads are only there to coordinate the effort and ensure the manifests and docs of different components will be ready within a timeline agreed to by the WG leads.

The release team does not control the roadmap of the components. The WGs are fully responsible to follow a different release schedule, which is the case for a lot of the Kubeflow components.

https://github.com/kubeflow/community/issues/540#issue-1061767603

I see that @annajung , @jbottum , @kimwnasptd, @shannonbradshaw are taking actions under the banner of a "Kubeflow 1.5 release team"

From the initial release team that was selected via nominations in 1.4 only @jbottum , @annajung and myself remained until the end to actually produce a release, and @shannonbradshaw joined in the mid of the release to help with the docs heavy lifting, since there was no one else to do the actual work.

After the successful release of KF 1.4, we decided to stick around and help bootstrap the 1.5 release. We also wanted to welcome more members and that's why we sent a nomination email to ask for people interested 3 weeks ago. Again, no one nominated themselves and the email got zero interest from the community.

After that we continued doing the release team meetings as we did for 1.4, and no other member of the community aside the above list of people, showed up for weeks.

[kimwnasptd]

Regarding nominations, @annajung described in the release team meetings a handful of valid reasons why we shouldn't be doing just nominations, but have the previous team select the next release team from people who have nominated themselves, which is the exact same things that the K8s community does. And I agreed when I heard her arguments.

But even before discussing process and nominations, @thesuperzapper @aronchick @lalithvaka @james-jwu, I'd like to get your feedback on @annajung's points about the alternatives she provided.

The release team has discussed and agreed about being completely open for new members to join the release team as well as having the leads select their successors, which is also what K8s is doing. This was not decided neither overnight nor in closed doors. We discussed this in a release team meeting right after we sent the nominations email, in which no one replied, and this was also a distinct item in the release retro [1] [2] and discussed over two release team meetings, which were open for everyone to join, but almost no one showed up.

So my question is why do we suddenly want to break the consensus we concluded upon during the last month and after all those meetings?

Note that I'm totally fine waiting for another week to see if someone else wants to join the release team, but my personal opinion is that we are just discussing process right now, when we have a lot of work to do to make the new release, and very few people to do the actual work. I still don't see a member of the community who was ready to put a lot of work into the release, and was left out.

[james-jwu]

My read of proposal from @thesuperzapper is that the current process can be improved when there is a significant concentration of one company in release team. I am super appreciative of the work done by the 1.4 release team, and the people who stepped forward to volunteer for 1.5 release.

I can see that the invitation for 1.5 release team was publicly announced. The title can probably be improved (i.e. "Call for 1.5 release team nomination"). Some additional improvement to attract more participation could include:

1. Publish at lest 2 invitations during the 2 week period. A bit more communication should be good - it looks like we added 2 more volunteers as a result of this discussion.

2. Publish a PR at the end of the 2 week period and cc the WGs.

3. When there is >X% of members coming from one company, or there are insufficient membership (e.g. < Y members)

• An additional invitation is sent to invite more diverse participation

• The WG leads are notified to add additional nomination or approve current nominations.

I don't feel very strongly about non-affiliated person seconding the nomination. I think release team volunteers are in short supply, and I would find ways to encourage more participation.

[kimwnasptd]

We discussed about this extensively during today's Release team meeting and all current members @kimwnasptd @annajung @DomFleischmann @DnPlas @js-ts @jbottum @shannonbradshaw unanimously agreed to the following process, after taking into consideration all the feedback:

We will keep being open to anyone that wants to join the release team. @james-jwu shared some very effective ideas above that we could adopt to have a more efficient outreach process.

At the same time the leads [Release Manager, Docs Lead, Product Manager roles] will be selecting their successors. This incentivizes people to have a shadow role, and the release team can ensure the selected people can successfully handle a lead role. This comes hand in hand with the current leads always striving to:

1. Make the expectations and requirements for each role more clear
2. Mentor their successor, to ensure they can be ready for a lead role

Regarding a final approval of WG/PSG we would actually like to involve more stakeholders, like end users as well as distributions. Their feedback is very valuable, since they are part of the release process.

The process will look like this:

1. For a couple of weeks we have the nomination process
   • Through this period we publish at lest 2 invitations in the mailing list
   • Publish a PR at the end of the 2 week period, to commit members to the ACL, and cc the WGs
2. After this period if no concerns are raised then we move forward with the proposed team
3. The established team defines its roles members and is ready to go

This is only a high level overview. I'm going to open a PR to also nail down the small details in the handbook and further discuss there as well.

[thesuperzapper]

@kimwnasptd that's an interesting suggestion for a process, however, I believe the process for deciding the release team should be established by the community more generally, rather than by the members of a past or present release team.

The consensus on this issue, and on mailing list thread seems to be that my proposal is a good first step for the 1.5 release team.

I have copied it below:

---

1. Formally agree that the current "Kubeflow 1.5 release team" has not been established yet
2. Put out a new request for nominations:
   • (We can limit the time period to 1 week to limit the wasted time)
   • (People can self-nominate if they like, or nominate others)
   • (I think a sensible requirement is that a non-affiliated person seconds the nomination)
3. The group of ALL seconded nominations becomes the "proposed release team":
   • (The specific roles, like "Release Manager" and "Docs Lead" can be decided by consensus among the team)
4. The "proposed release team" will be presented (as a group) in a GitHub issue for "endorsement" or "objection":
   • (If more than 60% of the Working Groups "endorse" the release team, it is formalized)
   • (The Project Steering Group gets a hard veto)

[thesuperzapper]

I see that @kimwnasptd has created a new nomination thread for the Kubeflow 1.5 release team.

Please raise your nominations and seconds on the thread, before Monday 2021-12-06.

[kimwnasptd]

@thesuperzapper we believe the same way the WGs are deciding on their direction, by the people that put in the effort, should be the case with the release team.

The above plan I posted is something we've been going on iteratively with everyone in the release team meetings for over a month, and also updated it to include feedback from this discussion.

So unless the @kubeflow/project-steering-group has any further objections we are moving forward with that plan.

[thesuperzapper]

After a relatively lengthy discussion in today's Kubeflow Community Meeting (2021-11-30), the general consensus was:

1. In the interest of releasing Kubeflow 1.5 in a timely way, we should allow the current "Release Team" to manage the release:
   • (This is in recognition that more diverse representation has already been added to the proposed "Kubeflow 1.5 Release Team")
   • (However, we still encourage any other members to join, and further increase diversity).
2. In parallel, we should formalize a "Release Working Group":
   • (This WG would be established using the same process as all the other Working Groups)
   • (This WG would have a formal charter - like all other WGs - and be stored in the kubeflow/community repo)

I am happy to work with the current "Release Team", and facilitate the creation of a formal charter for this future "Release Working Group".

[james-jwu]

@thesuperzapper On point #2 regarding formation of "Release Working Group", I can't agree that there was general consensus during KF community meeting on 11/29. Besides you and @jbottum, the people who offered opinions were myself and @andreyvelich (sorry if I missed someone). I was satisfied with one of the main contention points being addressed, which was more diversity and participation from the community in the release team. I just don't want to invest a lot of energy creating new processes/structures when we have a low cost way to improve current process.

I did not remember @andreyvelich endorsing the release WG idea, but please correct me if I am wrong.

[andreyvelich]

I think it was @thesuperzapper who proposed the Release WG. For me it is not clear why do we need Release WG.

What will be the main difference between Release WG and Manifests WG ? How we will define part of Kubeflow project that Release WG maintains ?

cc @kubeflow/wg-manifests-leads

[thesuperzapper]

@andreyvelich @james-jwu

The agreed charter of Kubeflow Manifests WG limits them to providing the kubeflow/manifests repository, there are many other aspects to creating a release that are out of their scope.

A great reference point for Kubeflow Release WG will be the Kubernetes SIG Release whose charter should help clarify the distinction.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.