Better support other programing languages

[fredemmott]

Deleted template as it doesn't really apply. Hack code currently uses Composer (PHP), but needs to move away. Both Yarn and Esy maintainers have expressed interest in supporting Hack code.

I'm linking to Esy issues here instead of re-filing them as it seems like it would be ideal to come to an cross-package-manager agreement on how to handle these problems.

• Provide a way to identify 'system' packages and resolve version constraints: https://github.com/esy/esy/issues/847
• Expose PnP data outside of javascript: https://github.com/esy/esy/issues/850
• Provide a way for scripts to know how they were executed: https://github.com/esy/esy/issues/851

[arcanis]

• Provide a way to identify 'system' packages and resolve version constraints: esy/esy#847

I think we would want to use the engines field for that. Just to be entirely sure: this system package feature you'd like would be purely informative, right (similar to how the node engine simply informs the user when the requirement isn't met, it doesn't download it automatically or changes the environment)? So not high-pri?

• Expose PnP data outside of javascript: esy/esy#850

This type of use case will be solved by custom linkers, which will have the ability (through standardized interfaces) to generate whatever they want from the dependency tree. The PnP linker will keep doing what it currently does (ie generate a .pnp.js file), but it will be entirely possible for the Hack linker to generate a hack-packages.json or whatever it is that Hack uses - even a Hack file if you want to.

[fredemmott]

I think we would want to use the engines field for that. Just to be entirely sure: this system package feature you'd like would be purely informative, right (similar to how the node engine simply informs the user when the requirement isn't met

It treats it like any other unresolvable version conflict. I guess the node behavior is likely the same.

As for engines: as commented on the other issue, I think this was /intended/ to be the right solution here, but if we're using the npm repository, 'node' must be listed as an engine if we specify any other, which doesn't seem like the right thing to do - https://docs.npmjs.com/files/package.json#engines

it doesn't download it automatically or changes the environment)?

No, just reports what's already there.

So not high-pri?

For other languages in general, probably not - but given the rate of change and BC breakage in Hack, I think being able to resolve version constrains on HHVM is an absolute blocker for us moving.

This type of use case will be solved by custom linkers, which will have the ability (through standardized interfaces) to generate whatever they want from the dependency tree. The PnP linker will keep doing what it currently does (ie generate a .pnp.js file), but it will be entirely possible for the Hack linker to generate a hack-packages.json or whatever it is that Hack uses - even a Hack file if you want to.

How do these get installed? Is it something along the lines of "if you're using hack, depend on hhvm, and also always specify hack-linker as a dependency" ?

[fredemmott]

To give another native example which is more complicated (not one that I actively want to support now, just to give some background and hopefully end up with something more general purpose), if we were talking about C++ we'd likely want constraints like:

• 'supports c++17' - possibly divided into language features (GCC5) and stdlib (GCC6)
• 'gcc 6+ || clang 4 || apple clang 1000.11.45.5'

[arcanis]

if we're using the npm repository, 'node' must be listed as an engine if we specify any other, which doesn't seem like the right thing to do

That's npm's behavior. I'm relatively open to make a breaking change on this if there's a valid business case. I'm not even sure Yarn actually enforces this rule now 🙂

How do these get installed? Is it something along the lines of "if you're using hack, depend on hhvm, and also always specify hack-linker as a dependency" ?

Plugins can be built into a bundle (for example Yarn itself will be a bundle containing a predefined set of plugins). Depending on what you prefer you can either make the Hack linker a downloadable plugin (the exact way of activating it is open to debate, my current thinking was something like yarn plugin add hack), or bundle your own package manager (possibly without JS/npm support if you feel like it) and ship it with Hack.

I'd probably recommend the first one, if only to keep a unified interface.

• 'supports c++17' - possibly divided into language features (GCC5) and stdlib (GCC6)
• 'gcc 6+ || clang 4 || apple clang 1000.11.45.5'

C++ requirements are a different beast. They are on my radar, but since they have a lot of moving pieces and build flavors I want to be careful - especially with TS Modules at the horizon. There's still a fine line between a package manager and a build system, and I don't want to accidentally cross it and turn Yarn into a circus. But I see your point 🙂

[fredemmott]

That's npm's behavior. I'm relatively open to make a breaking change on this if there's a valid business case. I'm not even sure Yarn actually enforces this rule now 🙂

Ah, ok, I was assuming this was a server-side limitation.

Plugins can be built into a bundle (for example Yarn itself will be a bundle containing a predefined set of plugins). Depending on what you prefer you can either make the Hack linker a downloadable plugin (the exact way of activating it is open to debate, my current thinking was something like yarn plugin add hack), or bundle your own package manager (possibly without JS/npm support if you feel like it) and ship it with Hack.

If we go with Yarn, it'll most likely be the latter, as we'll also need to start distributing Node on some of the older platforms we support (e.g. Ubuntu 14.04 includes Node 0.10.25). Unlike most other packages of modern node for older distributions, we'd be installing it outside of standard directories like $PATH to avoid conflicting with anything that actually does depend on the OS-vendor version.

[fredemmott]

I think we would want to use the engines field for that. Just to be entirely sure: this system package feature you'd like would be purely informative, right (similar to how the node engine simply informs the user when the requirement isn't met, it doesn't download it automatically or changes the environment)? So not high-pri?

How should system-installed packages inform Yarn that engine vX.Y is available? Or is this also up for to plugins?

[fredemmott]

Edited to add https://github.com/esy/esy/issues/851 - small UX issue: scripts should know how they were executed to give better error messages

[arcanis]

How should system-installed packages inform Yarn that engine vX.Y is available? Or is this also up for to plugins?

I would go with exposing a hook, yep. Something like this:

{
  buildEngineSet: async (engines: Map<string, string>) => {
    // assumption that --version returns a semver-compatible value
    engines.set(`node`, (await execFile(`hack`, [`--version`])).stdout);
  },
}

scripts should know how they were executed to give better error messages

It can be deduced (kinda) via $npm_execpath which contains the path to the binary. Another way, maybe more solid because it doesn't have the potential to change too much, is to use $npm_config_user_agent. A third option would be to expose a dedicated value ($pm_name).

[fredemmott]

Thanks - either of those vairables seem fine. I'll poke esy about adding support for them.