search

jupyter-governance / ec-team-compass

A repository for Executive Council discussion, syncing, and meeting notes.
https://executive-council-team-compass.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
3 stars 5 forks source link

Request to Decrease the number of Jupyter Orgs. #12

Open Carreau opened 1 year ago

Carreau commented 1 year ago

Hi,

As of this morning I had to audit "all" the Jupyter organization for access and or update each organisation/team/repo accordingly if needs be. I'm going to keep the what/why private for good reason, but I'm happy to share why in private channel.

I was able to audit 9 organisations, though I believe we have more (my memory tells me 13).

It is extremely painful to make the audits, and to be sure I did the audit correctly.

I am once again going to ask for a consolidation of organisations. More than a couple of organization is not manageable from a security perspective.

Carreau commented 1 year ago

2 (3?) easy organisations to reduce.

1) Jupyter-attic: Github Now has the "archiving" feature. I suggest to move all repos from jupyter-attic back to Jupyter and use the archiving feature.

2) Jupyter-incubator has only 4 repository. I think the benefit of having an org for that is limited.

3) (?) https://github.com/jupyter-standard has a single repository. I don't see whay it can't be a team under the Jupyter org

I'm guessing already having a list of these orgs would help.

As for the other issue, I understand that it may take time to reply, but please at least acknowledge reception.

blink1073 commented 1 year ago

Hi @Carreau, I agree that there are too many orgs (speaking for myself not the EC). To give a concrete proposal to kick things off:

Note that the above would require a change in governance, and clear standards of onboarding and offboarding repos onto those orgs, as well as org ownership.

Carreau commented 1 year ago

Thanks @blink1073, One question I have is do we actually need to segregate things by orgs ? Arent's teams sufficient ? Why can't the kernels be under the Jupyter's Kernel Team for example ?

Note that the above would require a change in governance, and clear standards of onboarding and offboarding repos onto those orgs, as well as org ownership.

As long as a repo is under a given team it should not be a problem, for example, the rust-lang org has ~180 repo, and us projects (which are public and org-wide).

blink1073 commented 1 year ago

I personally don't think teams are sufficient, for two reasons:

Carreau commented 1 year ago

Try to restart this limited proposal, for a step by step what about just doing step 1 for now:

Jupyter-attic: Github Now has the "archiving" feature. I suggest to move all repos from jupyter-attic back to Jupyter and use the archiving feature.

No more, no less, that is already decreasing from 9 to 8, which a bit more than a 10% reduction in number of orgs.

Carreau commented 1 year ago

(Side addition, technically IPython also rely on https://github.com/pickleshare which we are the only maintainers as well, It's another discussion but I think we should fold that repo back into this at some point).

blink1073 commented 1 year ago

I agree we can get rid of jupyter-attic. For pickleshare, why not bring it up to the Jupyter Foundations and Standards council to move the repo to ipython?

Carreau commented 1 year ago

For pickleshare, why not bring it up to the Jupyter Foundations and Standards council to move the repo to ipython?

That is a good idea, I'll reach out to the rest of the Jupyter Foundation and Standards.

For Jupyter-attic, I'd love a few more +1 before moving it, unless you agree that this is a low traffic enough repository that we don't need to have EC approval.

blink1073 commented 1 year ago

jupyter-attic isn't called out in our governance docs, I view this as housekeeping.

Carreau commented 1 year ago

Ok. Then when I have some time I'll move all the repos (which are already archived) to jupyter. Unless there is someone that opposes to it in the meantime.

There are "only" 36 repos, so I'm likely to do that by hand instead of a script.

Carreau commented 1 year ago

I did not even realize that ec-team-compas is on it's own organisation called jupyter-governance.

Does this really have to been it's own org ?

Carreau commented 1 year ago

jupyter-attic isn't called out in our governance docs, I view this as housekeeping.

All repos migrated back to Jupyter, and marked as Archived, the Jupyter-attic org itself has been archived.

Carreau commented 10 months ago

25 is one more symptom.

I also discovered https://github.com/jupyter-native – which is empty.

krassowski commented 10 months ago

Oh, repos from attic are back in jupyter/ org? I was a huge fan of that solution and was trying to advocate it more. There are many novice e.g. googling "jupyterlab-debugger" which brings up https://github.com/jupyterlab/debugger and then they break their environment by following the severely outdated instructions in there (of course this is despite a clear banner saying it is archived); moving things like that to attic would have added another layer of "do not use me" disclaimer. Of course the problem that I am highlighting can (and should) be solved by other means (either us or GitHub improving messaging in the archived repositories, for example by modifiyng their READMEs)

Anyways, attic is dead, long live a single org to rule them all!

jtpio commented 10 months ago

I also discovered https://github.com/jupyter-native – which is empty.

This one can likely be deleted. If I remember correctly, it was an alternative to https://github.com/jupyter-xeus at the time the xeus proposal was created.

cc @SylvainCorlay @JohanMabille

Carreau commented 10 months ago

This one can likely be deleted. If I remember correctly, it was an alternative to @jupyter-xeus at the time the xeus proposal was created/

I'm happy if you want to keep it, maybe just remove the Jupyter Logo

Carreau commented 10 months ago

Oh, repos from attic are back in jupyter/ org? I was a huge fan of that solution and was trying to advocate it more. There are many novice e.g. googling "jupyterlab-debugger" which brings up jupyterlab/debugger and then they break their environment by following the severely outdated instructions in there (of course this is despite a clear banner saying it is archived); moving things like that to attic would have added another layer of "do not use me" disclaimer. Of course the problem that I am highlighting can (and should) be solved by other means (either us or GitHub improving messaging in the archived repositories, for example by modifiyng their READMEs)

Oh, we can still rename the repository and push a single commit that delete all files but readme on more repos, any particular repo we need to do that ?

krassowski commented 10 months ago

After some confusion, I interpret that @Carreau comment on the other issue (https://github.com/jupyter-governance/ec-team-compass/issues/25#issuecomment-1894186079) was encouraging me to move my comment from the other issue here, so here it is:

Just to spell it out, it looks like we are stuck in a bad place here (either inconveniencing maintainers or security team) because we are using a free produce whereas the platform also offers a paid version which does not have the limitation that the security team is facing:

One of the main differences between GitHub Enterprise Cloud and other plans for GitHub.com is access to an enterprise account. Enterprise accounts provide administrators with a single point of visibility and management across multiple organizations. For more information, see "About enterprise accounts."

Link: https://docs.github.com/en/enterprise-cloud@latest/admin/overview/about-github-enterprise-cloud

krassowski commented 6 months ago

So apparently Jupyter is now using Enterprise as per comment from @fperez https://github.com/jupyter/enhancement-proposals/issues/122#issuecomment-2099501888. This should resolve the issues which made the security team pursue reduction in the number of orgs :tada:

jasongrout commented 6 months ago

See https://github.com/jupyter/governance/issues/219 for more info about the enterprise org.

jasongrout commented 4 months ago

Now that we have adopted the policy of all Jupyter GitHub orgs being under the Jupyter Enterprise org, can we close this issue as resolved?

fperez commented 4 months ago

I think so - @Carreau any objections?

Carreau commented 4 months ago

Now that we have adopted the policy of all Jupyter GitHub orgs being under the Jupyter Enterprise org, can we close this issue as resolved?

I would prefer not to. The high number of org is still cumbersome, even with jupyter enterprise , you still have to do some steps 20 times and that's only when it's possible to do those.

Like we still have to synchronise 20 security teams, or modify most settings in 20 places.

Having the enterprise only help a bit for a few things because it allows to list and sometime get access to all the orgs and enforce some settings organisation wide (2FA). But it's really only providing the bare minimum.

For example I still can't see the teams of https://github.com/binderhub-ci-repos because I'm not an owner. Now because I am part of Jupyter enterprise I could make myself an owner.

Here is a video of me just trying to view open the settings page of the 20 orgs we have. See how I can't even reach even 1/2 of those and how long it took.

https://github.com/jupyter-governance/ec-team-compass/assets/335567/84e208e7-5b0d-409a-9307-fcafdc44fa3b

And this is only the level 1 page, imagine when you need to go to the 3rd or 4th level down, or verify that some teams are the same.

Does that feel reasonable to you ?

There is also a bunch of features that you can't do across orgs (migrate issues from one repo to another). And integration with third party for example code scanning, where you need to give access to 20 orgs one by one.

If I can make a metaphor, I'm saying that having 20 keys/locks to open my door is annoying, and you are giving me a keychain and telling me I only need to carry one keychain and not 20 keys.

jasongrout commented 4 months ago

Point taken, and quite an analogy. Yes, I also found the enterprise tools quite minimal. Sure, let's keep this issue open to still have a bit of pressure, especially for those teams that function across Jupyter like the security team.

fperez commented 4 months ago

Sounds good Matthias, good points!

krassowski commented 4 months ago

This announcement from GitHub just dropped yesterday: https://github.blog/changelog/2024-07-10-pre-defined-organization-roles-that-grant-access-to-all-repositories/ It does not mention enterprise use case specifically but is labelled as "enterprise". Might be worth checking if new options are now available in Enterprise panel and if not, checking with GitHub if they plan to add them on Enterprise level any time soon.

Carreau commented 4 months ago

I completely see the point of enterprise, and I think it makes a lot of sens, it give you a master key and audit tools in case something goes wrong. And it is great. For a company it makes sens to have HR get access for when employee leave, or have audit. But I don't think enterprise have a goal which is aligned with large scale code management as we want to do in open source

I now for security we don't need all teams access that often, but there is a bunch of things that we might not do because we have 20 org. For example I just remember that .github repos are a thing, and we'd need to sync them across 20 orgs. Or list all the repos that are forks and can be deleted

I believe that the the criteria for some of these orgs to exists could be determined solely on the number of repo they contain:

(highest number is jupyter with 146, followed by Hub 75, and lab 72 which represent more than half of the 507 repos across all orgs).

I think that the cost of having an org for so few repos is too high, and the main case where an org should be created is when a group of repos want to split because an already existing org is too large. Not the opposite of creating an org by default.

Even the IPython orgs with 30 repos is IMHO on the edge and in general https://github.com/orgs/$ORG/repositories lists 30 repos per page so even Hub and lab would "only" be 3 pages.

Carreau commented 2 months ago

In addition, I recently realized that having multiple orgs, means that we have to manage independently for each org:

Carreau commented 1 month ago

One more things, with 20 orgs, we need to join waiting lists for beta issues 20 times: For example: