search

executablebooks / team-compass

Organizational policy and internal documentation for the executablebooks community
https://compass.executablebooks.org
Creative Commons Zero v1.0 Universal
0 stars 0 forks source link

`ebp-bot` shared access and documentation for publishing to PyPI & NPM #10

Open choldgraf opened 3 years ago

choldgraf commented 3 years ago

In https://github.com/executablebooks/jupyterlab-myst/issues/4#issuecomment-868496530 @chrisjsewell noted that we're using a bot on PyPI called ebp-bot to handle ownership and publishing of PyPI packages in the EBP stack. We give the bot account permissions to publish PyPI packages, and use its API token as part of our CI/CD.

We should document this practice and share admin access with the team.

Accounts

To do

agoose77 commented 1 year ago

Added ebp-bot to jupyterlab-myst on PyPI, but not yet the CI

agoose77 commented 1 year ago

@rowanc1 are you happy to create the ebp-bot for MyST and share the creds with the steering council?

rowanc1 commented 1 year ago

Will do that now. :)

rowanc1 commented 1 year ago

I have created the bot, and shared most myst-related projects with it on npm. Shared the credentials. I hit a rate-limit, so I will aim to get the rest over soon. Getting mystjs over to using CI to deploy will force our hand on the rest of this.

choldgraf commented 1 year ago

We had a brief exchange about this in Slack today. @chrisjsewell said that he would share the PyPI credentials for the ebp-bot, and I believe @rowanc1 has the credentials on the NPM side. Once both of you have shared them with me and I've documented them, I'll check off the boxes above. The next step will be adding ebp-bot to all of our projects but that will likely take some time as a follow-up.

rowanc1 commented 1 year ago

NPM Repositories

I have shared the NPM credentials. The following are or should be shared with the ebp-bot account:

stevejpurves commented 1 year ago

@rowanc1 is there some action needed from me on the thebe repos?

rowanc1 commented 1 year ago

Jumped on with @stevejpurves and got all the thebe repos sorted. @chrisjsewell, just the @unified-myst team now for the npm side.

chrisjsewell commented 1 year ago

just the @unified-myst team now for the npm side.

yep I invited ebp-bot

agoose77 commented 1 year ago

Thanks @chrisjsewell! Did you manage to find the creds to send to the steering council? I think we're nearly done here!

rowanc1 commented 1 year ago

Thanks @chrisjsewell can you please change the access control for @unified-myst to owner for the ebp-bot rather than member?

I think one of the other things we should sort out is the VSCode extension:

@chrisjsewell do you know what the credentials are for the executable books Microsoft org?

agoose77 commented 1 year ago

PyPI Repositories

These should have ebp-bot added as a maintainer (owner). I haven't checked that it is the owner for these accounts, just that it's present.

agoose77 commented 1 year ago

conda-forge feedstocks

These should have conda-forge/jupyter-book added as a maintainer. I've issues bot commands for the unchecked repos.

chrisjsewell commented 1 year ago

can you please change the access control for @unified-myst to owner for the ebp-bot rather than member?

yep done (I didn't see the option when I initially added)

chrisjsewell commented 1 year ago

I hunted down the pypi ebp bot password and gave it to @choldgraf (for the second time 😜 )

agoose77 commented 1 year ago

@choldgraf, I'm thinking it might be worth converting this to a discussion? I anticipate it will involve lots of discussion, meanwhile we probably want the tasklist-aspect of this issue to remain fairly readable.

Pre-empt this with "I'm a member of the EB community", so this is an outsider-stakeholder's take on where things / are going.

Sharing credentials is fine, but I can't help but feel there is a bit of a trend recently towards lets say "corporate control"

My motivation for pushing this is that I use EB tools all over the place, and right now many of them are single-author maintained. Whilst I don't want to sound alarmist, this has been a problem in the past where maintainers have lost access to their accounts (2fa), been phished, or even just gone AWOL. In the case of EB, I doubt the latter is likely, but I would feel more comfortable if we knew that multiple people could administer these. Going forward, that's easiest done with scoped credentials in my opinion.

In other words, the particular issue of the steering council having admin access to every package is about widening access, not restricting it. Moving individual repos to scoped-access tokens only is more restrictive (and an objective that hasn't been debated yet), but makes an explicit trade-off for security (i.e. someone gets phished and every JB package is then vulnerable to malware injection attacks). MyST tools are now used in so many repos that I think we should be considering the security side of things fairly carefully at this point. Fundamentally, I think it's also the case that if we have a governance structure, it should at the minimum have access to the packages.

Now though, curvenote have added all this syntax into myst-tools.org/docs/mystjs/dropdowns-cards-and-tabs, (without really consulting me grimacing). That's fine on its own, I believe they should absolutely be allowed to do whatever they want in their repo. ... The problem comes now, its "defacto MyST", and I feel it will be very restrictive to work on sphinx-design, and I'm always to have to go through x levels of review.

When it comes to MyST, as I see it, MyST is well-defined as a standalone concept from JB around which there is a growing community. At the Jupyter Notebook workshop, people didn't name MyST tools inasmuch as "MyST" itself. The MEP effort feels like a good first-step in recognising the diversity and size of the user base; if tools outside of EB are going to work with MyST, we need to have good standards and a transparent process. I'm hopeful that we can continue that process; the work on link resolution looks well considered, and I've enjoyed seeing the range of stakeholders weighing in.

I'm not au-fait with the detailed overview of the MyST-JS work, but it's my understanding that that syntax predates the MEPs, and I would assume our MEP process will speak to both new and existing syntax. I can't see any mention yet of tabs in https://myst-tools.org/docs/spec/myst-schema, which makes me feel comfortable in that guess.

I think there is a necessary change in pace and approach in the same way that the JupyterLab team have well-defined processes and structures that lean towards collaborative discursive changes. It's something I've seen in my own work, where the project(s) that I've been working on have moved from "R&D" to "Really Dependable". In that change, we've come to observe a similar widening of responsibility and need for more consensus-based changes. For example, one of my repos is used by all JupyterLab extensions (built from cookie-cutter), and it's necessitated a more collaborative style on my part (I've actively noticed it). My estimation is that MyST will/needs that too if it's going to survive moving forward. Otherwise, I worry that we would be at-risk of presenting the same challenges to users & developers that the docutils collaboration does/did.

and lastly, we come to executablebooks/myst-vs-code. This was a pet project that I have essentially not touched for two years, nor has anyone else. But the moment I circled back to it (today), I feel that I was instantly being scrutinised on it and the changes.

I did some digging and found this PR, which I assume is the context here? I think this gets to the core of why anything would be under EB vs private repos; EB is a community-driven project and implies some expectations. Removing features and/or moving them to non-EB repos is not precedented. I can understand the rationale for wanting to reduce your scope, and in doing-so contribute only to parts that you feel you can. I think those should be separate concepts, though; at a technical level, it falls under increasing the number of third-party dependencies, but at a social level it also raises questions. I think there is precedence for this in the case that separate communities can co-exist, e.g. the distinction between MyST and Jupyter-Book, but at a personal level, it's more complex. What I'm saying here is that I think we need to discuss these concerns and ideas, such as the ones you mention above, so that we can figure out the best path forward. I know it's early in the week, and I don't know what the work/life schedule is like for the EB team, but I imagine it will take a few days to get everyone's viewpoints in here.

Together with the lack of admin access, this makes it too much of a burden to maintain ... Will I be given some leeway (like mystjs), to develop it as I see fit (within reason, and also I'm happy if others want to collaborate)?

I feel more qualified to speak to this than, say, the EB direction in general, whilst I'm not yet taking on a maintainer role in many repos besides jupyterlab-myst. Ultimately (post-grant), anyone working on these projects is primarily working in their free time. Part of figuring out the "path forward" will involve discussing what everyone feels they can contribute. I'll hold off from any more pontificating until there're more voices in the room, though.

chrisjsewell commented 1 year ago

I'm thinking it might be worth converting this to a discussion?

yeh thats fine, this comment can also be moved if necessary. I'm just trying to voice some concerns, in hopefully a constructive manner

and right now many of them are single-author maintained.

yeh exactly, I would emphasize that I was the one created the PyPI ebp-bot in the first place, so I'm certainly not against it 😅

I can't see any mention yet of tabs in myst-tools.org/docs/spec/myst-schema

It is not in the spec no, but.. it is very easy to get to for users (who probably will not be looking at the spec): https://myst-tools.org/ -> Guide -> Dropdowns, Tabs & Cards

slight side note A problem I feel, is that there is both myst-parser (python) and mystjs, but neither is a reference implementation. They don't do only what is in the spec and nothing else. Its understandable, because are made to be "products", but yeh it doesn't help. It is also perhaps debatable about what a reference implementation entails, when the roles/directives syntax is, by design, extensible. I would compare this with (my old favourite) djot that already at least 5 reference implementations already (https://github.com/jgm/djot.js, https://github.com/jgm/djot.lua, https://github.com/aarroyoc/djota, https://github.com/hellux/jotdown, https://github.com/matklad/djot-rs, ...) TLDR: Its a good start, but we can do better.

EB is a community-driven project and implies some expectations

and this kind of gets to the heart of the question, what are those "expectations"? I feel it is not so clear.

rowanc1 commented 1 year ago

I think there are lots of different conversations and ideas in the points raised @chrisjsewell. I am going to focus my response to this issue that @choldgraf asked us to complete.

As a reminder of context, MyST as a separate project was announced less than a month ago, and our first MEP is still not merged. Many of us are putting in a lot of effort into those initiatives to improve community decision making practices. Many of the uncertainty points you are raising will be addressed as we work together as a community. It will take time and discussion, and to get there, we will go through many motions like this issue, which improve the ExecutableBooks organization, and reinforce our governance.

To @agoose77's point this specific issue is about "widening access [to the Steering Council], not restricting it". Although many things are not yet fully defined in our team policies, the Steering Council's role in this respect is well defined: to steward all technical and IP assets of the organization (policy).

Let's help them by getting this issue tied up.

agoose77 commented 1 year ago

On the topic of sorting out package admin, @choldgraf can you add the ebp-bot to these packages?

chrisjsewell commented 1 year ago

reinforce our governance the Steering Council's role in this respect is well defined: to steward all technical and IP assets of the organization

As @agoose77 found out with https://github.com/conda-forge/jupyter-book-feedstock/pull/19#issuecomment-1456520108, essentially, repository/organisation owners of open-sourced licensed (particularly MIT) packages do not have any rights or IP over any distribution of that package. That's the trade-off you make when doing open-source programming So, although it is certainly a good idea to have shared ownership and happy to encourage / move in that direction, you might understand why it could be antagonistic to claim ownership/governance rights, especially with respect to retroactive enforcement of new policies

choldgraf commented 1 year ago

Hey all - sorry for slow responses, I've got a 10 day old infant at home and I've been horribly sick for the last 7 days of it.

I appreciate the conversation about top-down vs. bottom-up decision making and authority. I think it's an important topic and it is one that is near to my heart. There's a lot that we need to flesh out in our policy docs - what we have now is very basic and really just the first steps, I agree with @rowanc1 that the point is to iterate and have community discussion to improve it. I'm sorry @chrisjsewell that it is stressful to feel the responsibility of maintainership with policy that is incomplete and/or doesn't properly empower you.

Can we create a discussion and move these comments over there, and also invite conversations from others? Then we can use that to make some refinements to the org policy. I think it's important that we define values, norms, and policies that any core team member is expected to uphold and follow. We also want policies that properly empower our team to carry the responsibilities that they have. It will take time to define these and incorporate them into our practices.

On the topic of credential-sharing, I think this is an obviously good/important thing to do. We want to reduce bottlenecks of access and information as much as possible, and share accounts/assets with the organization rather than leave them w/ individuals. This was true when the project was largely one grant, and doubly-true now that it is formalizing/broadening participation. Let me know what I need to do to keep that process moving forward. (and @agoose77 ah yep - I will do so ASAP)

chrisjsewell commented 1 year ago

Sorry to hear 🤒

Yep thats fine by me

I think I've added PyPI ebp-bot to all the things I own now 👍

choldgraf commented 1 year ago

Just adding ebp-bot to the projects I'm bottlenecking on now - I assume I should add it as an owner to them? Can somebody provide guidance there?

EDIT: Thanks for the guidance, I have now invited ebp-bot as an owner in each of the PyPI projects that you shared above @agoose77 .

agoose77 commented 1 year ago

@choldgraf yes, I think the point is that we need the steering council (>1 person) to have ownership over all repos, so that if any single maintainer combusts, or we get new maintainers, it's not a chaotic problem!

agoose77 commented 1 year ago

As @agoose77 found out with https://github.com/conda-forge/jupyter-book-feedstock/pull/19#issuecomment-1456520108, essentially, repository/organisation owners of open-sourced licensed (particularly MIT) packages do not have any rights or IP over any distribution of that package.

As I see it, the problem with the "helpful PRs" that I created in these repos is not that licenses preclude EB from having rights over distribution on conda-forge. Whilst it's true that licenses often concern themselves with this part of the distribution problem, I haven't seen anyone raise that point, and that's not what I took from the maintainer's response.

As I outlined in that issue, the reason that (I believe) the maintainer was upset by my actions was that maintainers of recipes are expected to ... maintain. I acted too quickly without thinking about the human side of conda-forge! Making a fairly big change like "add a new maintainer to all these recipes, and BTW I am not visibly affiliated with them or their packages" is fairly unpalatable from the kind of social model that operates in conda-forge. I opened N>20 PRs with zero context, and I have no social standing in those repos.

The difference with providing the steering council with ownership to our existing packages is that conda-forge builds from our sdists. We are currently the upstream provider of those packages, and we need to take steps to ensure that we follow practices that align with the scale of the organisation. I know @chrisjsewell that we're on the same page here, so I'm stating this part for posterity in this wider discussion.

I maintain a few packages on conda-forge already, so I should have known better, but I made a mistake at the tail end of a long day. It was too easy to write a xonsh script. :facepalm:

choldgraf commented 1 year ago

OK I've added ebp-bot to each of the projects above, I also logged-in and accepted invitations to all projects that had a pending invitation.

I noted that there were a few projects where ebp-bot was a maintainer but not an owner. Can somebody please switch it to owner? (I think @chrisjsewell and @agoose77 are owners on one of each of them)