CITGM abuse

[vkurchatkin]

Hi everybody.

I'd like to bring up an issue of potential abuse of current CITGM process. As it is, it allows maintainers of included packages to abuse Node core and internal APIs without the risk of future breakages, thus strong-arming Node core to maintain compatibility with their code.

I don't think that there is a precedent here, but is seems that author of https://github.com/standard-things/esm is planning to do just that. His code contains usages of the following internal APIs (list is not comprehensive):

• new Module
• Module.wrap
• Module._compile
• Module._extensions
• Module._resolveFilename
• Module._cache
• Module._nodeModulePaths
• path._makeLong
• process.binding
  • fs
  • getStatValues
  • stat
  • internalModuleReadFile
  • internalModuleStat
  • natives
  • util
  • arrow_message_private_symbol
  • decorated_private_symbol
  • decorateErrorStack
  • setHiddenValue
• process._tickCallback
• process._preload_modules
• REPLServer.prototype.createContext

While this is fine for a standalone obscure package that nobody uses, he intends to use in his other package, lodash, which is one of the most dependent upon packages on npm and also a part of CITGM testing. Furthermore, it seems that the only reason he wants to do this is to exploit CITGM and to force Node core to support these APIs indefinitely.

Here is a direct quote from here:

And from twitter:

All of this obviously being done in bad faith. That's not to mention that @jdalton has blocked me on Twitter immediately after that, preventing even a possibility of constructive discussion.

I'm not sure what should or can be done here, but I thought it was important to bring this issue to attention of CTC. We already had a crisis with graceful-fs and npm in the past, and this one has a potential to be much worse, so we should do everything possible to prevent it, until it's too late.

[vkurchatkin]

@ljharb by private I personally mean something that is not documented as being public and shouldn't be used. If you apply "pseudo" to this definition, you'll get "something that you kinda shouldn't use, but you can, if you want to". That's why term "pseudo private" seems inappropriate to me.

[jasnell]

The process.binding() and _-prefixed APIs in Node.js are private by convention only. Historically, these were largely Private-only-because-we-said-they-were-private, but now we have a documented policy around it.

Specifically from that document:

Due to the nature of the JavaScript language, it can often be difficult to establish a clear distinction between which parts of the Node.js implementation represent the "public" API Node.js users should assume to be stable and which are considered part of the "internal" implementation detail of Node.js itself. A general rule of thumb has been to base the determination off what functionality is actually documented in the official Node.js API documentation. However, it has been repeatedly demonstrated that either the documentation does not completely cover implemented behavior or that Node.js users have come to rely heavily on undocumented aspects of the Node.js implementation.

Also from that document

Breaking changes to internal elements are permitted in semver-patch or semver-minor commits but Collaborators should take significant care when making and reviewing such changes. Before landing such commits, an effort must be made to determine the potential impact of the change in the ecosystem by analyzing current use and by validating such changes through ecosystem testing using the Canary in the Goldmine tool. If a change cannot be made without ecosystem breakage, then CTC review is required before landing the change as anything less than semver-major.

If a determination is made that a particular internal API (for instance, an underscore _ prefixed property) is sufficiently relied upon by the ecosystem such that any changes may break user code, then serious consideration should be given to providing an alternative Public API for that functionality before any breaking changes are made.

Regardless of how these things have been handled in the past, this is the policy now.

The policy makes it absolutely clear that developers that make use of these things do so at their own risk, but it also makes it absolutely clear that core has a responsibility not to break the ecosystem, and to work with the ecosystem on coming up with viable options -- such as making reasonable public APIs to replace the private ones. That, for example, is what we did with the new http module APIs for working with headers to move people away from using _headers, and with the new http2 compat API which requires frameworks to use the public APIs only if they wish to achieve real API compatibility.

Working with ecosystem developers to identify which new APIs are required so that they can do the things they need to do without relying on those internal APIs is exactly the right thing for us to do.

With regards to CITGM, as the person who created the tool in the first place I can with absolute certainty that the goal was never to act as a gate for changes. The purpose is to allow us to know what the impact of a change would be, not to determine whether the change should or should not be made. If something breaks in CITGM we could very well choose to simply go with it any way. That said, refer to the responsibility not to break the ecosystem discussed previously. In other words, CITGM is an early warning system, that is all.

With regards to this thread in particular. I have to say that I'm quite unhappy with the way that @bnoordhuis and @vkurchatkin have engaged in this discussion. jdalton (I'm not going to at-mention him because he unsub'd and I don't want to pull him back in) did not start this thread, he was at-mentioned and he responded, then was promptly attacked with fairly negative, condescending and unproductive comments (e.g. You are not an idiot so what are you then?). To his credit he took those comments in stride but I believe an apology is owed to him.

With regards to whether this particular issue needs to remain open, I'd vote no. There is a specific path forward that has been identified and it's clear that no one is attempting to abuse CITGM in any way.

[refack]

the definition of "private" seems to have multiple interpretations; mine is that the only things that are private are the things that are inaccessible. Labeling something accessible as "private", by this definition, is precisely "pseudo-private".

@ljharb I don't disagree with you, nor with @Trott, that's why I use the term "internal APIs". We have a contract with the users, we designate some APIs as internal, and we cannot guarantee semver-stability or even regular stability if they are used. Same with experimental or deprecated modules.

AFAICT any attempt to do OOP encapsulation is just a contract, nothing is guaranteed. Even in CPP you can read the function table and find the addresses of "private" methods or calculate the memory address of private fields. You just don't.

That's what I told JD, even if he does his best, I have no idea if node is flexible enough to let him do his best since we never designed those API for this use, and don't test them enough.

[@jasnell scooped me, only he wrote it better 🤷‍♂️]

[mcollina]

I think this issue can be closed, as per https://github.com/standard-things/esm/issues/66#issuecomment-323866297.

I'm doing it now, feel free to reopen.

[refack]

With regards to this thread in particular. I have to say that I'm quite unhappy with the way that bnoordhuis and @vkurchatkin have engaged in this discussion. jdalton (I'm not going to at-mention him because he unsub'd and I don't want to pull him back in) did not start this thread, he was at-mentioned and he responded, then was promptly attacked with fairly negative, condescending and unproductive comments (e.g. You are not an idiot so what are you then?). To his credit he took those comments in stride but I believe an apology is owed to him.

With regards to whether this particular issue needs to remain open, I'd vote no. There is a specific path forward that has been identified and it's clear that no one is attempting to abuse CITGM in any way.

@jasnell I'm requesting that @vkurchatkin moderate their comments in accordance with our CoC and moderation policy.

[jasnell]

Ok @refack. Thank you for the heads up.