search

honojs / hono

Web framework built on Web Standards
https://hono.dev
MIT License
20.62k stars 602 forks source link

[Proposal] Middleware hosting and CI/CD automation #477

Open riywo opened 2 years ago

riywo commented 2 years ago

Hi, I've just briefly talked with @yusukebe about the way to host 3rd party middleware under honojs GitHub organization and publish them to npm and deno registries. I recently went through Jenkins plugin development processes and was really impressed how they manage such a huge ecosystem by using the similar concept. I believe we can get some insights from them and build a smart solution. Here is my rough proposal. Thank you for reading this proposal in advance!

Summary

We can create a dedicated GitHub and npmjs team for each 3rd party middleware and manage fine-grained access control for the hosted repository and the published package. We can start from whole manual workflow and gradually automate them by implementing several bots.

Source code hosting

We'd like to host all middleware as a GitHub repository under honojs GitHub organization to give more credibility to our users comparing with the ones hosted under arbitrary user/organization. The issue here is how to manage membership because the authors of 3rd party middleware shouldn't have write access to other honojs repository. Jenkins (jenkinsci organization) hosts all 3rd party plugins by following the procedure below:

  1. A contribute creates a public GitHub repository for the plugin under any GitHub user/organization.
  2. Once they finish the initial implementation, they cut an GitHub issue for "Hosting request" like https://github.com/jenkins-infra/repository-permissions-updater/issues/2708
  3. Some automated processes evaluate the request. Also, a few community members review it, then approve/reject.
  4. Then, the automated process generally does:
    1. Fork the original repository under jenkinsci organization. Ask the requester to delete their original repository.
    2. Creates a dedicated GitHub team like this and add the requested author(s). The list is managed by their infra repository like this
    3. Give Admin permission of the repository to the new GitHub team

After these steps, the author(s) have the full permission of the hosted repository like they had before beautifully. I believe we can mimic the way Jenkins does, manually first and automatically once we implement the bots.

Package publishing

We'd like to publish all middleware under @honojs scope for the same reason as the source code hosting. The issue here is how to give fine-grained permission to the authors of 3rd party middleware. Let's look into the Jenkins example again:

  1. In the "Hosting request", they also give release permissions to the requested members which allows them to manipulate Maven artifacts hosted by Jenkins team. This is done by giving the package release permission on the hosted repository website via bot.
  2. They also allow the authors to fully automate the release process by giving some credentials to the GitHub repository via GitHub Actions Secrets.

As you can see above, the concept is similar to the source code hosting and we can apply this idea to our package publishing as well. npmjs supports teams under organizations and we can assign selected packages and members to the team like GitHub's team. To automate the publishing, we should create a machine user per repository on npmjs, invite it to the npmjs's team and generate an automation token (allowed to bypass 2FA), and give the token to the repository via GitHub Actions Secrets.

Note: We also want to do the similar thing for Deno but https://deno.land/x doesn't support scope or namespace as far as I know. So, we might need to consider to use other hosting like https://esm.sh/. We don't discuss Deno now in this issue but will do later.

Other opinions

This is my preference but I would recommend you to name all middleware repositories prefixed or suffixed by middleware or something e.g. honojs/middleware-sentry or honojs/sentry-middleware. This enables us to identify which repositories are for middleware very easily. Jenkins requests all new plugins to name suffixed by -plugin. This also can avoid unnecessary future name conflicts.

References

Original discussion: https://github.com/honojs/hono/issues/350

Feel free to comment any feedback! I'm really interested in the implementation of these processes if you like.

yusukebe commented 2 years ago

Hi @riywo

Thanks for the very good suggestion! I am amazed that Jenkins has so many plugins and that they are well managed. I like most of your proposal. Please wait a moment while I reply in detail!

yusukebe commented 2 years ago

@riywo

We also have a similar process. But some details are different, and the processes are almost not automated. I'm not sure if a lot of middleware will be made for Hono. So, we want to automate from the minimum.

Source code hosting

The current process is below:

  1. The contributor creates the issue in this repository. e.g. #423
  2. Admin creates the repository like honojs/sentry. The important thing is to use the template project https://github.com/honojs/middleware-template. In this case, it is better to use the template because the setup is complicated.
  3. Add the privilege to the contributor.

I think it's ok to change all of this process to what you are describing. The only difference is using the template project.

I would like to do it.

Package publishing

The current process is below:

For npmjs:

  1. Admin asks npmjs username from contributor.
  2. Add the user to honojs org in npmjs.
  3. Contributor publishes the package to npmjs manually. Or create GitHub action by oneself.

For Deno:

  1. Admin creates the package in denoland.x, and set up the webhook in the repository.

In this case, I would also like to take the approach you describe. However, there are a few things to consider.

Repository name

As you said, I thought it would be easier to understand if we added a prefix such as middleware-. Deno packages refer to a GitHub repository, but it has no problem changing the repository name. So we can rename them.

These are my thoughts. If there is no one who has an opinion, we will process the way you describe it a little by little with helping you @riywo :)

cc: @Code-Hex Even if you don't have any plan to make a middleware, I want to know your opinion.

Code-Hex commented 2 years ago

Hi 👋

Thanks for mentioning me. My opinion is almost the same as @riywo -san's opinion which is distributing any secrets via CI/CD.

But I don't have any knowledges for deno package system. Sorry...

yusukebe commented 2 years ago

Thank you @Code-Hex !

I'd like to add something about Deno. There is not much to do when we publish a module in deno.land/x.

To register, enter the module name and set up a webhook in the GitHub repository, referring to the following.

After that, simply create a tag in the repository, and it will be released - no need for commands like npm publish. Once we've done the initial setup, we don't need to do anything else.

riywo commented 2 years ago

@yusukebe @Code-Hex Thank you so much for your feedback!

Can we create a machine user with npmjs? When I looked for a way to do it, I couldn't find it.

People just create a new user with a shared email address and password like this https://www.npmjs.com/~autifyhq-release

I'd like to add something about Deno. There is not much to do when we publish a module in deno.land/x. To register, enter the module name and set up a webhook in the GitHub repository, referring to the following.

My point here is that there is no concept like scope of npmjs on deno.land/x. So, we can't secure a package name for a new middleware if someone already created the same package, just like the global scope of npmjs or majority of package registries like CPAN, rubygems, etc.

yusukebe commented 2 years ago

Hi @riywo !

My point here is that there is no concept like scope of npmjs on deno.land/x. So, we can't secure a package name for a new middleware if someone already created the same package, just like the global scope of npmjs or majority of package registries like CPAN, rubygems, etc.

You are right!! I reconsidered, and also, I think that the "deno.land/x" has a problem it does not have a "scope,". And we can't get the module name if it is already taken.

Now, we have good news. The recently released Deno v1.25 includes an experimental feature that allows npm modules to be imported without using a CDN like "https://esm.sh/"

import { Hono } from 'npm:hono'
import { poweredBy } from 'npm:hono/powered-by'

const app = new Hono()
app.use('*', poweredBy())

app.get('/', (c) => c.text('Hello Deno!'))

Deno.serve(app.fetch)

This code will really work!!

Of course, this is an experimental feature, and it will be yet to be officially introduced. Although that, I think we would like to move to this approach in the future. Until then, using a CDN like "https://esm.sh/" is a good idea.

In addition, if we use the CDN or the new feature of Deno, we do not need to do "denoify". It is used to transform for Deno code. This process was a little bit tricky.

So you are right!!

I would like to discuss this continue in another Issue.

In conclusion, I think we do not need to consider how to publish on "deno.land/x" right now. First, we should make the way to publish to npmjs.