Assume ownership of low level http packages

[wesleytodd]

NOTE: This is a proposal for discussion. The goal here is to help the team have a conversation around if this is something we would ever want. If the team decides yes, then we could move forward with making it part of the goals and charter and discuss more broadly in the community what this would change.

Would it be productive to pull many low level http related packages under the ownership and governance of Node core? Today the most popular ones live in the jshttp org which is managed by the Express team. Due to this relationship it is common for good additions to be held up or declined because of compatibility issues with Express apis. I am also aware that most of the other popular frameworks have re-implemented some of these for this reason.

I think a better long term model could be for a group like ours to maintain these under the governance of node core. This would mean support sustainability, group ownership of technical direction and added legitimacy and visibility to the packages. The biggest downside I see is migrating the ecosystem is difficult, but I am sure there are other issues to address.

Things to discuss:

• Should this even be considered as a part of the scope for the team?
• We could request a charter from Node to manage these
• How would we pull the existing packages into this? Fork, move repos, re-implement?
• What prior art is there on a node team or wg managing specific packages?

[dougwilson]

Due to this relationship it is common for good additions to be held up or declined because of compatibility issues with Express apis.

I'm not aware of any examples of this in jshttp modules, as they are very disconnected from Express.js itself. Are there any specific examples of this for a module in jshttp org?

I am also aware that most of the other popular frameworks have re-implemented some of these for this reason.

Koa, for instance, extensively uses jshttp modules. There shouldn't be a reason, especially since they mainly just operate on strings. I think it can be improved if there is a specific instance you can highlight for a module in jshttp.

I think a better long term model could be for a group like ours to maintain these under the governance of node core.

I think this is a good thought, but I'm not sure this is the correct forum to have this discussion, as the modules already have a governance and should likely be discussed over there first... ?

[dougwilson]

In a general comment, though, I would assume that directly maintaining code / modules is a non-goal of this Node.js team. There certainly doesn't seem to be any mention of module maintenance / ownership in the charter currently. It would likely end up as a distraction for what I think the goals of the team actually are, as I could see the discussions turning into just general module maintenance instead of the much higher-level discussions that the README seems to set up for this team to have. As an example, I read the charter of this team to be more along the lines of web framework architecture discussion vs a team maintaining code in modules, of which I think would end up being a distraction to the actual goals of the team.

[arei]

I would also wonder if there are any examples of other node teams, like this one, or is this over reaching what teams like this are setup to do.

[mcollina]

I don't think ownership is the fundamental issue. I think we should make a goal to foster collaboration between frameworks on fundamentals, and if that results into a shared ownership it would be great but it should not required.

[dougwilson]

Yes, that is s good way to articulate it, and aligns with what I am thinking this team is setting out to do.

[wesleytodd]

There certainly doesn't seem to be any mention of module maintenance / ownership in the charter currently

The goal of this issue was to foster this exact conversation! Should code ownership be part of our charter? I think there are pros and cons (as the discussion has clearly pointed out), and I wanted to make sure the conversation was had so that all options were on the table.

To make it more clear, I added a bullet to the OP "Should this even be considered as a part of the scope for the team?"

Are there any specific examples of this for a module in jshttp org?

There are for sure examples of this in the fastify and restify communities.

Koa, for instance, extensively uses jshttp modules.

I think Koa is a great example of how it should work, but unfortunately that is not how it has been working in other ecosystems. The goal of this team is to improve that, and I think this is one way we could go about achieving that goal.

I think this is a good thought, but I'm not sure this is the correct forum to have this discussion, as the modules already have a governance and should likely be discussed over there first... ?

If it is not a part of our scope as a team I didn't want to waste time in the Express TC meetings to discuss it. Do you disagree with my approach of deciding if it is even a valid conversation to have first?

I think we should make a goal to foster collaboration between frameworks on fundamentals, and if that results into a shared ownership it would be great but it should not required.

Completely agree! The idea I proposed is one way to achieve better collaboration because it would make the collaboration explicit in the charter. Again, there are pros/cons to this approach and I am not strongly advocating for getting chartered to manage this, but I wanted to atleast open the floor to the conversation.

[dougwilson]

Are there any specific examples of this for a module in jshttp org?

There are for sure examples of this in the fastify and restify communities.

What are they?

The goal of this team is to improve that, and I think this is one way we could go about achieving that goal.

I'm not sure that having this team own modules actually helps in that. Those projects would still need to decide to use them, which they could do equally today. If who owns a module is a concern for those projects and changing the owner is what would change that, I think there are larger problems at play that would be unsustainable to solve in that way. But maybe I am wrong and someone can clarify how this team actually owning the modules would move towards this goal?

Do you disagree with my approach of deciding if it is even a valid conversation to have first?

Until you said it there, it was no clear that was the approach in any way. There was no mention in OP that this was to discuss here first then to invoke the TC of those repos later. I think it is understandable that it could easily have been read different than you intended since it was not actually called out and you are a key member of the controlling TC. One could read it as if there was already a decision there since you are a representative of said TC.

[mmarchini]

What problems you believe would be solved by moving the ownership and governance to Node.js core (or to this team)?

What prior art is there on a node team or wg managing specific packages?

There is (sort of). llnode, [node-report]() and node-inspector are all under the governance and ownership of the Diagnostics WG. Other tools I think are owned by WGs: gyp by Build WG, node-addon-api by N-API WG, and readable-streams by the Streams WG.

[bajtos]

I am not familiar with the wider web-framework ecosystem, but there is one important divide I am aware of: Express/Koa/jshttp vs Hapi. AFAICT, Hapi is not building on top of jshttp and instead implements all functionality inside its own ecosystem. For example, Express and Koa depends on accepts from jshttp while Hapi uses @hapi/accept.

From what I understand, it was a conscious decision to let Hapi depend only on modules maintained by the Hapi team. I kind of understand their reasoning and position and it is NOT my intention to persuade them otherwise.

It makes me wonder though if we can find a way how to share the knowledge and improvements, especially those related to security and various quirky edge cases learned from real world usage, so that all Node.js users can reap the benefits, regardless of the framework they use.

[wesleytodd]

it was a conscious decision to let Hapi depend only on modules maintained by the Hapi team

I would love to hear from the Hapi team about this decision and reasoning so I can understand it better. Express is similar in some regards that it treats "externally maintained" modules differently from core ones. But I think that up-leveling to Node and the governance model their-in would hopefully improve whatever caused this decision from the Hapi team. Maybe it wouldn't, and I would love to hear about why.

It makes me wonder though if we can find a way how to share the knowledge and improvements, especially those related to security and various quirky edge cases learned from real world usage, so that all Node.js users can reap the benefits, regardless of the framework they use.

Yeah! This is for sure something I would like to tackle in this group. I think that this is a separate conversation because it should probably move forward even if this question of stewardship is ultimately decided against. Want to open an issue to that effect?

[wesleytodd]

What are they?

I have heard from individuals, and I will let them share their experiences if they choose.

Until you said it there, it was no clear that was the approach in any way. There was no mention in OP that this was to discuss here first then to invoke the TC of those repos later.

Great feedback! Sorry I was unclear, I will edit my original post so that it is more clear!

What problems you believe would be solved by moving the ownership and governance to Node.js core (or to this team)?

I think that having a track for some of this to land directly in core would be greatly improved by bringing the teams closer together. I think that it would help avoid some of the fragmentation because of the trust the Node team has built with the community, that the apis which land in core will be supported and governed by open and inclusive methods.

I also hope that it would mean distributed load of maintenance. I have talked about the current situation the contributor base to the body of modules maintained by the Express team, and rebuilding that contributor base is hard and will take time. Shifting some of that responsibility to an organization with a proven track record of this seems like an easy out for us on the Express team 😛

[hueniverse]

hapi doesn't use any external code because I simply do no trust it.

I refuse to accept the common wisdom that open source is safe simply because it is open and widely used. It is far less work for me to maintain every single line of code above node itself than to have to review every change made in every module I rely on.

And don't kid yourself - if you are using npm modules and are not reviewing every change made before you upgrade (even a patch release), you are being completely negligence.

When someone installs hapi they know with very high certainty (short of the npm registry being hacked) that the entire code base is safe, well maintained, well tested, and with a single point of ownership for any issue they have. That is a huge differentiating factor between hapi and everything else.

In the past I had discussions with other framework maintainers who were looking into using hapi modules in their frameworks. This included some conversations with Doug about Express and Matteo about Fastify. The end result was that these other framework chose not to use hapi modules (some did for a little while then removed them). I fully respect their decision (especially given my own attitude).

The main reasons were: the fast pace in which hapi modules release new major releases, the lack of support for old node versions (especially discontinued ones which Express and Fastify oddly officially support), and to a much lesser degree the move to the hapi npm org which added @hapi/ to the module names and seems to have annoyed some people who don't like being reminded they are using hapi code.

If you look at the code frameworks can share, it is a very small and insignificant set. It is mostly about parsing HTTP headers... and at this point, it is mostly all done and pretty stable across frameworks.

hapi has a much richer feature set for working with cookies which is integrated into the core framework API. It supports chunk-encoding for all requests, not just static file downloads. It defaults to strict HTTP spec compliance and sometimes gives a more relaxed option. Some of these are available in other frameworks but not always and not the same way.

I think we should stay away from efforts to share actual code between frameworks. An application framework is much more than just sugar on top of node's HTTP APIs.

[jsumners]

...snip...

If you look at the code frameworks can share, it is a very small and insignificant set. It is mostly about parsing HTTP headers... and at this point, it is mostly all done and pretty stable across frameworks.

...snip...

... An application framework is much more than just sugar on top of node's HTTP APIs.

These are two points I can agree with. It is unclear to me how much benefit this proposal would offer outside of some very minimal implementations.

To reference actual code, in fastify-cookie we use the baseline cookie and cookie-signature modules because they do the easy (hard?) work of parsing/signing the headers. But we do not use Express's cookie-parser because the Fastify plugin model is different.

[JozefFlakus]

I can agree with @hueniverse but I would add to this my two cents. I believe that reusability of third-party packages depends on the purpose, the layer of usage and design decisions of course.

Marble.js approaches this problem in a similar way as hapi does - trying to depend on minimum set of essential third party libs when it comes to core modules. When looking at underling, very specialized mechanisms Marble doesn't want to reinvent the wheel. The best example are path-to-regexp or qs libs that are used also by Express. There is no sense to solve this kind of problems over and over if the offered features are just enough for you. Just use the battle-tested solution and forget. But from the other hand as an author I would like to maintain the core mechanism that are essential to the framework - having dedicated implementations that can be easy reachable and adaptable to needs. Body parsing - I would like to use a custom, tailored solution. Content negotiation / Multipart-parsing - use for the beginning a community supported solution and if needed migrate to custom one. As an example - in the future I would like to migrate the underlying ws package, that @marblejs/websockets module depends on, to a custom solution. Just to have a full control over the underlying mechanisms.

[wesleytodd]

All of the above arguments sound in favor of moving some set of behaviors into node apis. While I might disagree on some details, if your goal is to review and ship a solid package of dependencies, then the more things node core can do for you the less code you have to maintain directly as a framework author.

I think you all mentioned cookies, I opened #32 to discuss that in particular. I think this fits inline with what @hueniverse mentions about "It is mostly about parsing HTTP headers", and just because frameworks have solutions today does not exclude room for improvements.