search

jupyter / governance

The governance process and model for Project Jupyter
https://jupyter.org/governance/index.html
Creative Commons Zero v1.0 Universal
82 stars 71 forks source link

Require using the Jupyter Enterprise Org #221

Closed jasongrout closed 5 months ago

jasongrout commented 6 months ago

Questions to answer ❓

Background or context to help others understand the change.

Currently Project Jupyter development on GitHub is spread across a number of GitHub orgs. This provides autonomy for subprojects to maintain their own contributor lists, roles, and processes. However, as the number of GitHub orgs in Jupyter increases, it also causes confusion about the scope of official Jupyter activities and increases friction for Jupyter-wide policies.

Recently, GitHub offered Jupyter (and other open-source projects) a free upgrade to their GitHub Enterprise Cloud plan. The basic idea is that Project Jupyter would have an Enterprise org, and inside that enterprise org we have our existing GitHub orgs. The enterprise org provides a place to centrally manage policies and roles across all of the Jupyter GitHub orgs, while still enabling individual subprojects to maintain their individual GitHub orgs.

This upgrade helps solve pain points we've had with multiple GitHub orgs in Jupyter. For example:

Furthermore, using the enterprise org gives additional benifits, such as:

Based on this information, the Executive Council recently made the decision to create a Project Jupyter enterprise org, and we reached out to the SSC about having various Jupyter GitHub orgs join the enterprise org. At this point, many have joined.

There is further discussion of this at https://github.com/jupyter/governance/issues/219.

A brief summary of the change.

In this change, we require official Jupyter GitHub orgs to belong to the new Jupyter enterprise GitHub org. We also simplify other text in the subproject governance.

What is the reason for this change?

To make it easier to have consistency across Jupyter GitHub orgs, while still enabling autonomy for subprojects.

Alternatives to making this change and other considerations.

Voting

Vote is expected to close in 2 weeks, June 19.

EC members/voting checkboxes

SSC members/voting checkboxes

The process ❗

The process for changing the governance pages is as follows:

  • Open a pull request in draft state. This triggers a discussion and iteration phase for your proposed changes.
  • When you believe enough discussion has happened, move the pull request to an active state. This triggers a vote.
  • During the voting phase, no substantive changes may be made to the pull request.
  • The Executive Council and Software Steering Council will vote, and at the end of voting the pull request is merged or closed.

The discussion phase is meant to gather input and multiple perspectives from the community. Make sure that the community has had an opportunity to weigh in on the change before calling a vote. A good rule of thumb is to ask several Council members if they believe that it is time for a vote, and to let at least one person review the pull request for structural quality and typos.

fperez commented 6 months ago

Tiny edit in review. I also would like to repeat my previous suggestion:

My personal opinion is that now we have this option [the Enterprise feature], we should use MUST in that language. That would make it unambiguously clear whether something is or isn't "jupyter official".

I don't have a suggested edit yet, I think it would go in the final section of "responsibilities", indicating specifically how we ask subprojects to tag their org(s) as "Jupyter official" (I'm not even sure we know yet how that will be done, I know @jasongrout was asking some questions about the new enterprise setup).

Carreau commented 6 months ago

I can't see https://github.com/enterprises/jupyter/ it insist on redirecting me to accept the invite to the Jupyter Enterprise for JupyterHub.

So some question I can't answer as I can't visit the jupyter entreprise page.

This looks like a positive to me

fcollonval commented 6 months ago

Coming here from a discussion started on JupyterLab gitter, looking for advice on the best path for a specific setting now managed at the entreprise level:

@fcollonval (Frédéric Collonval)

Screenshot from 2024-05-14 09-24-45

I'm not able to publish a new version of the language packs due to the above actions setting being disabled (and not changeable in JupyterLab GitHub org). It may be a consequence of the move under the Jupyter entreprise umbrella - that setting is, according to stackoverflow, handled there now.

Not sure what is the best solution. Xref: https://stackoverflow.com/questions/72376229/github-actions-is-not-permitted-to-create-or-approve-pull-requests-createpullre

I don't know if it impacts the releaser

@jtpio (Jeremy Tuloup) this release of notebook went through yesterday, using the releaser: https://github.com/jupyter/notebook/releases/tag/v7.2.0rc1

jasongrout commented 6 months ago

Is there a way/how do you know all the official orgs that are part of Jupyter Enterprise ?

I've got a question into GitHub asking about this.

What does the global management interface looks like ? In particular can you manage things in one place and have all the org "inherit" the enterprise setting, or is it just a way to "impersonate" for every org and thus still have us check setting 13 times ?

It looks like there are typically three choices for settings: "No policy" -> deferred to org owners, like today; "Enabled"/"Disabled" -> overrides org setting. For example:

Screenshot 2024-05-14 at 09 01 23
jasongrout commented 6 months ago

@fcollonval - do you want to hop on a quick video conference to figure out the issue with actions you raised?

Edit: I just switched this setting on. Do things work now?

Screenshot 2024-05-14 at 09 09 49
rpwagner commented 6 months ago

Tiny edit in review. I also would like to repeat my previous suggestion:

My personal opinion is that now we have this option [the Enterprise feature], we should use MUST in that language. That would make it unambiguously clear whether something is or isn't "jupyter official".

I agree with the MUST language regarding GitHub as the service used to host Project Jupyter's source code and related work. There will be other services (e.g., Read the Docs, PyPI, conda-forge, etc.) used for different purposes, while GitHub is the service for the source code. As someone else commented, this helps to define the scope of which repositories may be in the scope of requirements including accessibility and security policies.

I don't have a suggested edit yet, I think it would go in the final section of "responsibilities", indicating specifically how we ask subprojects to tag their org(s) as "Jupyter official" (I'm not even sure we know yet how that will be done, I know @jasongrout was asking some questions about the new enterprise setup).

Once we have the policies and technical implications nailed down for adding orgs to Jupyter Enterprise org, the SSC may be the body to work with subprojects on being added.

jasongrout commented 6 months ago

I put the requirement to maintain source code on GitHub in the Project Jupyter enterprise org down in the list of subproject responsibilities. Is that better?

jasongrout commented 6 months ago

As a policy matter, who should have owner privileges on the enterprise org? @rpwagner, from the security standpoint, what do you think?

I think it makes sense for at least some members of the EC/SSC to be enterprise org owners, but I don't think it is necessary for all of them to have administrative permissions, and I think it makes sense that you could be a github admin without being a member of the EC/SSC. I'd rather we have a small group of active administrators particularly picked for this purpose, plus a few people for oversight/backup from EC/SSC.

rpwagner commented 6 months ago

As a policy matter, who should have owner privileges on the enterprise org? @rpwagner, from the security standpoint, what do you think?

I think it makes sense for at least some members of the EC/SSC to be enterprise org owners, but I don't think it is necessary for all of them to have administrative permissions, and I think it makes sense that you could be a github admin without being a member of the EC/SSC. I'd rather we have a small group of active administrators particularly picked for this purpose, plus a few people for oversight/backup from EC/SSC.

My suggestion is that the EC and SSC define a list of Jupyter communities members that are trusted to be enterprise owners, with at least one EC representative and at least one SSC representative. Equally as important, the list of enterprise owners should be reviewed by the EC and SSC on some regular interval, say quarterly. That review is as important as defining the initial list.

jasongrout commented 6 months ago

I think we should wait a couple more weeks to call a vote on this requirement to join the enterprise org to give us more time to surface and work out any other problems, and give subprojects that haven't joined (like JupyterHub) a chance to meet and discuss. In the meantime, I think we can proceed with figuring out the administration policy with the SSC.

Carreau commented 6 months ago

Can @jupyterhub owners refuse or accept the Jupyter enterprise invite, it's completely preventing me to access some of the links @jasongrout put in the description.

jasongrout commented 6 months ago

Can @jupyterhub owners refuse or accept the Jupyter enterprise invite, it's completely preventing me to access some of the links @jasongrout put in the description.

That's weird. I'll cancel the invite to JupyterHub for now. But JupyterHub folks, feel free to request an invite at any time! I think it's great to take your time here and discuss and see how it is working out.

CC @minrk as the SSC JupyterHub representative.

fcollonval commented 6 months ago

@fcollonval - do you want to hop on a quick video conference to figure out the issue with actions you raised?

Edit: I just switched this setting on. Do things work now?

Sorry for the intertwine discussion, we are now able to set that option at the JupyterLab org level. Thanks for the quick change.

minrk commented 6 months ago

We've no objections in https://github.com/jupyterhub/team-compass/issues/719, so I'll accept the invitation if you re-send it. I'm also okay waiting for this PR to land before joining.

Carreau commented 6 months ago

That's weird.

Yeah, my guess it's a GitHub bug/issue, not the first one I encountered recently.

Here is a few redacted screenshot of what it looks like. "Getting started" is 404 for me:

Screenshot 2024-05-15 at 14 12 47 Screenshot 2024-05-15 at 14 13 02 Screenshot 2024-05-15 at 14 13 17

I don't see a way to 'enforce' some setting on all orgs/repositories (like public security issue reporting), but it does give some overview to all the orgs/repos.

jasongrout commented 6 months ago

I don't see a way to 'enforce' some setting on all orgs/repositories (like public security issue reporting), but it does give some overview to all the orgs/repos.

You don't have the organization owner permission currently, so you won't see the settings menu option. We're figuring out who should have admin access to the enterprise org (see conversation above). Here's the main page for me, for example:

Screenshot 2024-05-15 at 09 51 33

Let's file a few bugs with github. Can you post a screenshot of what the Getting Started page looks like for you, Matthias? Also, after that, I'll resend the invite to JupyterHub (fyi, @minrk), and it would be great if you could screenshot that "approve/deny" page that prevented you from seeing the org.

Carreau commented 6 months ago

For getting to https://github.com/enterprises/jupyter , It would redirect me to https://github.com/enterprises/jupyter/invited which would show the (i guess), normal GitHub invitation accept page with a link to go to JupyterHub billing and plans

For getting started I get the usual 404 falling octocat, my guess is that's because i'm not an owner.

(for sec reason github often return a 404 instead of 403 so I'm guessing this is actually a 403 permission denied)

Screenshot 2024-05-16 at 07 10 27
jasongrout commented 6 months ago

Thanks. @minrk, I went ahead and invited JupyterHub to join the enterprise org again.

jasongrout commented 6 months ago

For the record, the Getting Started page seems to mostly be focused to org owners in various setup tasks.

Screenshot 2024-05-15 at 23 43 21

The links are:

GitHub Actions for enterprises Manage code security Monitor activity logs Enterprise policies Migrate your data to GitHub

I clicked the "Dismiss Onboarding" button at the bottom, and now it seems that page has disappeared.

Carreau commented 6 months ago

Here is where I'm redirected to

Screenshot 2024-05-16 at 07 55 57

"billing and plan" links to https://github.com/organizations/jupyterhub/settings/billing/summary and there no other actionable links in hamburger menu...

manics commented 6 months ago

@minrk has accepted the invite for JupyterHub, but I'm seeing the same as @Carreau in https://github.com/jupyter/governance/pull/221#issuecomment-2114092925

minrk commented 6 months ago

I accepted the invite, but I think the accepted invitation has to be confirmed by someone on the Enterprise, too, to finish it off.

rpwagner commented 6 months ago

I accepted the invite, but I think the accepted invitation has to be confirmed by someone on the Enterprise, too, to finish it off.

You are probably correct. I just accepted the transfer.

manics commented 6 months ago

https://github.com/enterprises/jupyter looks better now image

jasongrout commented 6 months ago

At this point, the following orgs have joined the enterprise org: IPython Project Jupyter Jupyter Governance jupyter-incubator Jupyter Server Jupyter Widgets Jupyter Xeus JupyterHub JupyterLab Voilà Dashboards Jupyter Attic

I think that is all of the official orgs of Jupyter. Are there any other orgs currently under Jupyter's purview?

manics commented 6 months ago

We've got a couple more Binder related orgs:

jasongrout commented 6 months ago

We've got a couple more Binder related orgs:

@manics - I invited both of those orgs

jtpio commented 6 months ago

For the Voila Dashboards sub-project, there is also the voila-gallery organization: https://github.com/voila-gallery

jasongrout commented 6 months ago

For the Voila Dashboards sub-project, there is also the voila-gallery organization: https://github.com/voila-gallery

I invited it. It looks like @rpwagner also invited a few other orgs like JupyterLite, jupyter-lsp, and Jupyter Standards.

@jtpio, is JupyterLite part of Jupyter at this point? I know we were all discussing it at some point, but I forget where that discussion ended up.

jtpio commented 6 months ago

I invited it

Thanks! I got the email, but the invitation does not show up in the voila-gallery organization settings to be able to accept it. Not sure why.

@jtpio, is JupyterLite part of Jupyter at this point? I know we were all discussing it at some point, but I forget where that discussion ended up.

Good question. I think this is still not clear yet, and iirc it is not listed as an official sub-project (or part of a sub-project) at the moment. Which is why I haven't accepted the invitation yet. Maybe it's currently at the same level as jupyter-incubator, as some parts of JupyterLite might end up being upstreamed in other existing Jupyter sub-projects in the long run.

But I'm happy to accept the invitation to join the Jupyter Enterprise Org, if it can help simplify processes for Jupyter related orgs.

jasongrout commented 6 months ago

Thanks! I got the email, but the invitation does not show up in the voila-gallery organization settings to be able to accept it. Not sure why.

Looks like you sorted it out, and I approved, so we're good on voila-gallery now.

But I'm happy to accept the invitation [for JupyterLite] to join the Jupyter Enterprise Org, if it can help simplify processes for Jupyter related orgs.

I suggest that orgs in the enterprise org be limited to those that are currently officially part of Jupyter, i.e., they consider themselves under Project Jupyter governance (including code of conduct, licenses, SSC and EC oversight, etc.), and they are accepted and acknowledged as under stewardship of a specific council (a subproject council, SSC, or EC). In other words, I see the enterprise org as being the place where we can draw a line around github activities and say "this is the official Project Jupyter".

If we're not sure (like with JupyterLite, it seems), I suggest we first clarify if JupyterLite is under the Project Jupyter jurisdiction. We have a process for creating a new subproject that I think several projects are currently pursuing, for example, or existing subproject councils (like the frontends subproject) can vote to adopt JupyterLite under its umbrella.

jasongrout commented 6 months ago

@krassowski noted here that he accepted the invitation for the jupyter-lsp org to join the jupyter enterprise org. In that same thread, there has been recent discussion of how jupyter-lsp could officially be adopted as under the Project Jupyter umbrella.

Like JupyterLite, can we clarify the status of jupyter-lsp first before approving its transfer to the jupyter enterprise org? For jupyter-lsp, it seemed in that thread that the current option under discussion is having a vote in the Jupyter Frontend subproject about adopting the jupyter-lsp org under the frontend subproject.

CC @rpwagner

rpwagner commented 6 months ago

If we're not sure (like with JupyterLite, it seems), I suggest we first clarify if JupyterLite is under the Project Jupyter jurisdiction. We have a process for creating a new subproject that I think several projects are currently pursuing, for example, or existing subproject councils (like the frontends subproject) can vote to adopt JupyterLite under its umbrella.

@jasongrout I think JupyterLite has made itself a de facto part of Project Jupyter, especially because JupyterLite is provided on the https://jupyter.org/try as the entry point to learning about Jupyter. If JupyterLite is not officially part of Project Jupyter, we should consider removing that. From a security or risk perspective, having some oversight of the JupyterLite development seems necessary if we're promoting its use this directly.

That was logic we went through on Tuesday in the Security Subproject on Tuesday as we reviewed the GitHub Enterprise features. Of course, it would be even better if JupyterLite were brought in to Project Jupyter in a more official capacity.

jtpio commented 6 months ago

Do we need to decide on the state of jupyter-lsp and jupyterlite now? Or could these orgs be left out of the Jupyter Enterprise org for now, and added later if they are accepted as Jupyter sub-project (or join an existing sub-project)?

To not block this PR, since it might take some time and discussions.

fperez commented 6 months ago

👍 to @jtpio's suggestion above. I think this is a good sign that we have a vibrant ecosystem of "adjacent" projects that have been incubated semi-informally, and we should work on processing that as cleanly as possible, but that shouldn't block this PR.

For example, @choldgraf just posted a a full JEP for JupyterBook, after a good amount of preliminary discussion and even meetings with the SSC. The jupyter-ai team similarly posted a vote for inclusion within the "umbrella" of jupyterlab, back last year. Those are examples of good practices in terms of allowing for discussion, consensus seeking and process we should keep.

I'm always open to figuring out where our processes become unnecessarily slow/cumbersome and to remove pointless red tape, but that's different from rushing things. Let's decouple the two things and get the (I think entirely non-controversial) Enterprise part done as a rubber band enclosing the things that are clearly, unambiguously, today official Jupyter projects. And we leave any new inclusions to proceed at their pace.

jasongrout commented 6 months ago

+1 on both Jeremy's and Fernando's comments. I'll cancel the invites to the enterprise org for jupyter-lsp and jupyterlite until we have more clarity on their status. CC @krassowski as well.

jasongrout commented 5 months ago

It's now been several weeks, and no other problems have come up. We already have two github approvals as well, so I'll move this to voting now. I've updated the description to have a ballot.

@jupyter/executive-council, @jupyter/software-steering-council - please vote in the PR description in the next 2 weeks.

jasongrout commented 5 months ago

@Zsailer - can you check the box in the description as your official vote?

jasongrout commented 5 months ago

A friendly ping to @Ruv7, @gabalafou, @SylvainCorlay for their thoughts/vote.

SylvainCorlay commented 5 months ago

Thanks for the reminder. I thought I had voted.

jasongrout commented 5 months ago

Circling back here, the vote has now closed. We have:

EC: 5 Yes, 1 Abstain SSC: 9 Yes, 1 non-vote

As such, this passes, so I will merge the PR. Thanks everyone!