[stage1] Haxelib replacement

[Simn]

Foreword

Issues in this repositories tend to get bogged down in long-winded discussions without much of a result. We would like to try a slightly different approach which is outlined here: https://github.com/HaxeFoundation/haxe-evolution/wiki/Haxe-Projects

We are in Stage 1 here, which means that comments should be short and to the point as mentioned in the document. If you want to discuss the procedure, please open a separate issue.

Now on to the actual project.

---

Haxelib replacement

Project lead: Simn Due date for stage 1: 2017-10-14 (2 weeks)

We are looking to replace haxelib with a better package manager solution.

Please share your ideas and concerns in a concise manner. Please also +1/-1 individual comments that you agree/disagree with.

[Simn]

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

[kevinresol]

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

[elsassph]

Criteria:

• semver dependency management, and solution to handle conflicts as we can have only one version of a lib,
• global cache and local libs similarly to npm/yarn,
• support for git / file dependencies.

[damoebius]

NPM isn't enough ?

[ibilon]

The user should be able to use an alternative package manager (without making change to or replacing the haxe exe).

[ousado]

The compiler should expose an interface for frontends like CLIs, IDEs, vscode-style language servers, and so on.

[ousado]

Whatever ends up as standard mechanism to invoke the compiler for one-shot compilations must be very fast, there should be no noticeable difference to invoking the compiler directly (in other words start-up times like that of nodejs or even the JVM are out of the question)

[ousado]

The standard mechanism to invoke the compiler will be a first point of contact for newcomers to the language. As such it should provide an outstanding user experience and documentation can't be an afterthought.

[ibilon]

We shouldn't change the current workflow and so not introduce any steps between changing a hxml and running it/doing a haxe command by hand.

[ibilon]

A migration plan is required:

• what happens to the already installed libraries? do they need to be reinstalled?
• do we keep the haxelib online repository as is? if not what about usernames?

[ousado]

Beyond building projects the standard mechanism to invoke the compiler should provide a standardized way to run other common tasks like tests, benchmarks, updating dependencies, etc. (much in the spirit of cargo).

[ibilon]

haxelib.json should be extended, proposed new fields (non exhaustive):

• repository field (to go with the current website field)
• listing the metas/defines of the library
• haxe version compatibility

[kevinresol]

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

Because -lib is the only thing that cannot be "consumed directly" by the compiler.

[MangelMaxime]

• Support lock files so we have reproductive build system

[ibilon]

The following fields should be removed from haxelib.json (or its replacements) and dealt on the library repository website:

• description
• website
• contributors
• tags

since it doesn't make sense to update those by making new releases.

[ousado]

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

[elsassph]

I know @damoebius' suggestion to use npm has been firmly downvoted but it isn't that silly - alternatively it would be interesting to have a npm-compatible haxelib API so we could be really interesting to have.

[EricBishton]

Configuration must be thoroughly documented and transparent. On-disk storage such as hidden .dev, and .current files should be discouraged and instead replaced by a documented configuration file.

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

[Simn]

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

[ousado]

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

If what you mean is a common layout for a staging area, as well as for ex- and importing a package database to allow migration between package managers, OTOH, that'd be a good idea.

[ousado]

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

I was under the impression we're talking about all the (future) components required for the workflows that haxelib is currently involved with, not just its role as a package manager.

If this is about package management only, I think we should look at the more principled approaches - so basically read A modular package manager architecture, conclude that we're probably not going to come up with something better, and hence "do whatever opam does".

[EricBishton]

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

It is and it isn't. While I respect the decision of writers of the extended or alternative package managers to create the solutions they think most appropriate, I believe that the canonical package manager (the one delivered as part of the Haxe toolset) must set an open standard that other tools can inter-operate with.

[Simn]

@sangohan @jcward: I removed your comments because they are way too wordy/not actionable. I've pasted them here so they are not lost: https://gist.github.com/Simn/2b13809d830cfd16dd5ba2f9c97cac6b

[Matan]

• Maintain a local cache of downloaded libs.
• Symlink cached libs to projects to save space. (flag to control behaviour and/or force local install)
• Support "globally" installing libs.
• Support licenses sub method to output all dependency licenses much like PHP's composer licenses.

[kiroukou]

I recently discovered https://lib.haxe.org/p/hmm library which does a pretty nice job on that part with current limitations.

For me, the current haxelib version isn't really far from what I need, but here is my wishlist :

• Having a way to abstract the lib source (git, svn, local, ...)
• Having 2 scopes of lib loading : global and local
• Recursive loading (a lib depending itself on an other, should be able to load it automatically). That leads to a version conflict resolver !
• Having/Keeping a comprehensive synthax

[jasononeil]

It should be possible to open a directory that a project is in, and when I type haxe build.hxml it automatically uses the configured version of Haxe for the project, and the right set of haxelibs/dependencies.

(This one might be controversial but I thought I'd throw it in!)

[jasononeil]

If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

[clemos]

I think we should deprecate haxelib run to avoid depending on neko.

[Simn]

I think we should deprecate haxelib run to avoid depending on neko.

Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

[ibilon]

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

Can't put that in version control since it contains computer specific path into it. EDIT: not actually the case, see @jasononeil comment below

As an alternative to a new file (like yarn.lock) maybe lock the versions directly in the hxml?

[jasononeil]

@ibilon here is how lix deals with it, no absolute paths:

# @install: lix --silent download "haxelib:hxnodejs#4.0.9" into hxnodejs/4.0.9/haxelib
-D hxnodejs=4.0.9
-cp ${HAXESHIM_LIBCACHE}/hxnodejs/4.0.9/haxelib/src
-D nodejs
--macro allowPackage('sys')
--macro _hxnodejs.VersionWarning.include()

Here is how yarn.lock deals with it (basically, it doesn't, it just uses relative paths):

angular-cookies@^1.6.4:
  version "1.6.4"
  resolved "https://registry.yarnpkg.com/angular-cookies/-/angular-cookies-1.6.4.tgz#c28f3f6aac7a9826c1e45f1d6807240036e5b26d"

[back2dos]

The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).

[markknol]

From Haxe code it should still be possible to know which libs are used. so doing #if (libname > 4). It would also be useful to be able to know which classes come which library (in macros/ documentation xmls/dependency dumps etc)

[jgranick]

I have many feelings on haxelib, but quickly l want to share some "wish list" items:

• Support for multiple remote repositories
• Support for cascading local library paths
• Support for wildcard versions (IE -lib lime:3.1.*)
• Support for installing shortcut commands for tools
• Self-updating

Multiple remote repositories would allow companies to generate their own internal versions and distribute more easily, or enable "beta" or "dev" channel repositories.

Maintaining a single class path directory would not be reasonable for dealing with source files. Similarly, it would be nice to be able to add multiple custom library paths -- similar to how class paths are resolved.

Support for wildcard versions would really, really help (especially if semver is respected). You can request library version 1.0.*, allowing for patch releases. This is a great middle-solution between "Give me version 1.0.4 exactly" or "Give me any installed version", and a frequently requested feature for OpenFL

As a cherry on top, it would be nice if libraries could install command aliases, such as lime for haxelib run lime or as3hx for haxelib run as3hx

The best package manager in the world will need updates. I would like to see a library that is self-updated when you run a command similar to haxelib update

[elsassph]

@clemos haxelib run isn't a problem per se, but it should be extended to other runtimes, and be configurable in haxelib.json.

[elsassph]

Caching libs is good, using symlinks is bad: someone could mistakenly modify a file and it would affect the cache / other projects. Libs should be copied locally in the project (unless optional CI optimisation to use cache).

[clemos]

@Simn @elsassph I agree it's not so much about about deprecating haxelib run itself, but rather about deprecating run.n in favor of Run.hx ;)

[bablukid]

The package manager should allow custom fields in haxelib.json It would be very usefull for librairies which are tied to a specific framework or runtime. i.e : For externs, we could provide a link to the original library and infos about how to integrate it in your haxe project i.e : a web framework which loads plugins (haxelibs) that have various assets ( like models, controllers, templates and translation files )

[andyli]

I would like to point out that it is important to consider how to handle "native code" (could be compiled ndll, C/C++/JS/... compiled libraries or source files).

One reason is that it is wasteful to download binaries of all architectures. e.g. Lime's ndll folder is 108M, but if you're only using Windows, you actually only need the 5.1M ndll/Windows folder.

We should also have some guidelines regarding to how to store the info of those "native" things, e.g. where is the original source, what param is used when building (use of -PIC/-O3/debug info), any additional dependencies that is not bundled etc.

[kevinresol]

Build tools (hxcpp,hxjava,hxcs) should not rely on haxelib too. They can be invoked by Context.onAfterGenerate or haxe --run

[Simn]

Stage 1 has concluded, I'll prepare the discussion part for Monday.