search

HaxeFoundation / haxe-evolution

Repository for maintaining proposal for changes to the Haxe programming language
111 stars 58 forks source link

[stage2] Haxelib replacement #34

Closed Simn closed 6 years ago

Simn commented 6 years ago

Foreword

Issues in this repositories tend to get bogged down in long-winded discussions without much of a result. We would like to try a slightly different approach which is outlined here: https://github.com/HaxeFoundation/haxe-evolution/wiki/Haxe-Projects

We are in Stage 2 here, which means that discussion should be limited to the items from stage 1. Below is the result of stage 1, ordered descending by rating (= Upvotes - Downvotes - Off topic votes). I am only listing items which had more than 5 rating. This doesn't mean that all other aspects should be completely ignored (especially late entries which might not have had enough time to get votes), but the focus should still be on the list below.

Now on to the actual project.

Haxelib replacement

Project lead: Simn Stage 1 issue: https://github.com/HaxeFoundation/haxe-evolution/issues/30 Due date for stage 2: 2017-11-6 (3 weeks)

We are looking to replace haxelib with a better package manager solution.

Feature set

General guidelines (higher level)

Simn commented 6 years ago

@back2dos: Which items of this feature set does lix not cover (at the moment)?

The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).

We will look into supporting https in the interpreter. From @ncannasse regarding that:

<nicolas> now that we have properly working ssl support for HL/Neko we should use these for OCaml/Interp as well
<nicolas> should be straightforward to backport HL ssl.hdll to ocaml FFI

Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

A problem with that is that it would require all the dependencies. One nice thing about run.n is that you don't need anything else. For instance, with Dox we provide a run.n so that users don't need all its dependencies to run it, only to compile it.

I'm currently working on a Haxe binary output which could help regarding that.

ibilon commented 6 years ago

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

Either that's built-in in the compiler, and we better make it right the first time considering the compiler's release cycle (even if that's better now).

Or that falls to the user call a separate tool to obtain the information required for compilation. Hardly an improvement in my opinion, how many time will one think "why doesn't it work, everything is good, oh I forgot to call the tool to tell the compiler I've changed my hxml file"?

Support lock files so we have reproductive build system

If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.

Not too familiar with the concept of lock files, do they allow that dependency change tracking?

--

@Simn it would probably be easier if the items from stage 1 had a number id.

jcward commented 6 years ago

Either that's built-in in the compiler (ocaml? slow release cycle?) ... Or that falls to the user call a separate tool ... Hardly an improvement in my opinion.

I had the same concern, but this "interpreting an .hx file" idea is interesting. It could alleviate these concerns.

HTTPS

Now this I don't understand. Of course you want HTTPS on the package authoring / upload side, but on the distribution / download side? HTTPS inhibits possible caching layers (reverse proxies, intermediate caches) and causes extra dependencies for the package manager. This is a public package repo, right? No login / secret info required. So what's the motivation for HTTPS?

benmerckx commented 6 years ago

So what's the motivation for HTTPS?

MITM attacks for one. However unlikely for anyone to spend their time on it, you just shouldn't transfer any code that will be run on your pc over http. It's just too easy to interfere with plain http data on public wifi or similar situations. Https should not be optional. I also don't see any reason for it to inhibit caching layers.

back2dos commented 6 years ago

Which items of this feature set does lix not cover (at the moment)?

Many of the features are actually covered just by haxeshim's stupid simple conventions, that could be trivially baked into the compiler (and should be IMO). From top to bottom, theses are the points that are yet open:

kevinresol commented 6 years ago

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

This should apply on build tools as well (e.g. hxcpp, hxjava, hxcs)

Simn commented 6 years ago

I have extended stage 2 time by a week because I won't be available until then anyway.

jcward commented 6 years ago

I've watched the London meetup, and tried the MVCoconut example, and I'm trying to digest the concepts.

To start, I think haxelib is a simple tool, it's easy to understand, and it does one thing (somewhat) well. Once you understand it and its limitations, you'll create tooling around it and live with it.

The idea of haxeshim seems fine enough -- a convention with a scoped directory structure containing library references, and a global cache of (referenced) versioned libraries. With a place for neko and haxe versions to boot. To me, the best things this provides beyond haxelib is: my dependencies are defined (and committed) with each project.

I'm less clear on lix, node, and npm joining the party. Seems like an awful lot of (JS-leaning) tooling prerequisites. Now I have to understand how NPM works and package.json and whatnot. It's a lot to take in, gotta look at it more.

Just to play devil's advocate, what could I do today with a simple script, checked into my repo, that gives me a reproducible setup with haxelib as it stands:

init_local_haxelib.sh

# Use a project-local haxelib
haxelib setup ./haxelib # or ./haxe_libraries
# [copy or symlink or unzip or wget, my choice] some cached libs to ./haxelib
# Clone some git haxelibs at exact commits:
haxelib git hxcpp-debugger https://github.com/HaxeFoundation/hxcpp-debugger
jcward commented 6 years ago

All this thinking about the very basics of Haxe lib management makes me wonder -- would it make haxelib better to simply remove features? What if...

As for "version of Haxe" -- I consider this slightly out of scope. I use very simple, conventional, (and sadly OS-specific) symlinks to select versions of Haxe and it works like a charm.

elnabo commented 6 years ago
back2dos commented 6 years ago

I'm less clear on lix, node, and npm joining the party. Seems like an awful lot of (JS-leaning) tooling prerequisites. Now I have to understand how NPM works and package.json and whatnot. It's a lot to take in, gotta look at it more.

FWIW understanding NPM is not such a bad choice ;)

What I have maintained for the past three years is that the conventions implemented in haxeshim should be built into the compiler. Beyond that point, I don't think we have to agree on anything. That's the beauty of it. People may or may not choose to use lix. If someone comes forward with a better package manager, I'm more than happy to use it, because I don't particularly enjoy working on such a thing. It's just that I have a desperate need for such a tool (and I believe that our community does share that need, even though they might not necessarily realize it ... more on that later).

(only read this if you're interested in lix) To understand lix, it's important to note that its first and foremost goal is reliability. The second goal is - yet again - reliability. And the third goal - believe it or not - is reliability once more. With those three goals in mind, after spending a few weekends exploring the many options that Haxe presents, I settled nodejs as a runtime because: - HTTPS: many companies and individuals increasingly value security - compared to the nodejs APIs, haxe's `sys` package is dismal. It doesn't even provide symlinking, permissions, proper control over child processes (in particular their environment and their I/O). Not a showstopper, but definitely a PITA. - nodejs is a mature piece of technology, with wide adoption and serious backing. It is battle tested and is relatively sure to still exist 5 years from now. I guess the next best thing we have is neko, which is being deprecated. There's other sys backends, but [without any tests](https://github.com/HaxeFoundation/haxe/issues/6461) there's no way to know how well they will work. Writing a package manager (and thus wrestling with three different OSes) is hard enough without having to worry about the APIs that you're using. - (bonus) everything is streaming. This *perceivably* improves user experience. To quote @thomasuster on the matter: > So I noticed this actually > Like the libs were installing crazy fast > But I wasn't sure how > then I was scared it was like cacheing my private repos on some server (hint: there's no caching going on) The first point is something the compiler team could fix. As for the rest, I guess lix could also be written against `sys` with a few concessions and a lot of debugging (the best solution by the way would be the libuv bindings for eval I keep pestering @Simn about). IMO *for the time being* the pragmatic thing to do is to use nodejs. There's more than enough design decisions to figure out and that's my primary focus. Now, with node as an interim (?) runtime, the obvious choice for installation is `npm install lix.pm -g`. This choice immediately proved useful, because the handful of people who started using lix are scattered over all three major OSes, which makes npm the most common package manager (of course nothing stands in the way of adding a package for brew or chocolatey or whatever). That said: beyond using npm to install lix, you don't *need* npm. You don't *need* a package.json. It provides an additional layer of scoping and reliability, so I guess it's advisable to use (at least until such time that lix is properly tested and thus sufficiently guarded against regressions). I also used that setup for the workshop, because I wanted a self-contained setup that installs all tools locally (including haxeshim, which otherwise would override your `haxe` command). Still, you can limit any contact you have with npm just to installation of haxeshim (which hopefully will be assimilated by the compiler), switchx (which is likely to merge with lix) and lix itself. If we make progress here, installing lix is the one thing you'll have to do to use lix, which I suppose is fair enough. I'm sorry about the somewhat Spartan nature of lix. It is a tool solely driven by the need of those who use it. Fortunately, Jason set out to [improve the documentation](https://github.com/lix-pm/lix.client/pull/21).

Just to play devil's advocate, what could I do today with a simple script, checked into my repo, that gives me a reproducible setup with haxelib as it stands:

[...]

Uhm, nope. This is not reproducible in the slightest. Let's set aside the fact that if your machine has no git it fails and if it has cygwin git, it fails - better yet - for totally obscure reasons. The main point is: if you run that script a couple of months apart, you get two different versions of the code, where the old one might only be Haxe 3 compatible and the new one might only be Haxe 4 compatible. And that's just with one library. If you have 100 of them, the chance of winding up with compatible versions after running the script is practically 0. So you will have to write all the particular tags/commits into your setup script by hand and maintain it by hand and remember to rerun it, whenever you switch branches.

Sure, one can live with it. I documented a pretty equivalent approach years ago. Unfortunately, it just turns out that if something is optional and not automated, it doesn't get done (at least not sufficiently well).

That is why lix fully automates this part, for all dependencies, including the transitive ones.

All this thinking about the very basics of Haxe lib management makes me wonder -- would it make haxelib better to simply remove features?

I am generally a big fan of removing features. That said, if the alternative is to rely on git only, my advise would be to use git submodules as Rob does. If I din't always forget to update submodules when switching, I guess that's what I'd go with myself. But it might work perfectly for you.

As mentioned though, git is not universally supported. What's more: it's not a reliable source of 3rd party code, because the owner may alter/delete the history or the whole repo. In the coming months, lix will gain a create-only (no deletion, no alteration except deprecation) registry intended as the primary source of packages, which will prevent things like the npm fiasco to unfold in its core ecosystem.

As for "version of Haxe" -- I consider this slightly out of scope. I use very simple, conventional, (and sadly OS-specific) symlinks and it works like a charm.

Sure. It works for you, on your machine and I'm quite optimistic that on the whole many people have found some way to arrange themselves with haxe and haxelib. That's great, but it still leaves room for us to progress as a collective.

The main reason why I do see library and language version management as an indivisible unit is usage.

First, there's the matter of having a replicable setup, that will work the same on your CI and on every team member's machine (regardless of OS, regardless of which branch they are on, etc.). Big teams care about this. It's fair to say that the reason this debate was tabled is that at the foundation meeting on the summit the foundation partners agreed on such a need. And it involves the compiler version too. If a patch is landed for something we need, we want to be able to use it across all machines without much hassle - and only on a specific branch to see what other problems might occur. With lix's support for nightlies, we can.

Second, and I think this is just as important (if not more), I see much value in a solid basis for creating tutorials that don't just completely break within months. There's always talk about how Haxe needs more tutorials and what not, but very little thought is given to the technical prerequisites to make them useful, such as making sure that the code actually works for the user. I believe there's a huge difference between a code example that is slightly outdated but compiles just fine, because setting it up was straight forward, and a code example that is slightly outdated and greets you with 500 different errors, half of which come deep out of some macro, that was broken between haxe 3.2.1 (that the tutorial was written for) and 3.4.4 (that the user has on their machine). It might even be detrimental to lock versions in such a case, because the library that the example is based on most likely had a haxe 3.4.4 compatibility update in the mean time that just didn't get installed due to version locking.

So, if you look at these two version management problems not from a purely technical perspective, but through the prism of meaningful use cases, they superpose almost wholly. Given how often concerns are voiced about reducing friction for newcomers, this could make an important difference. My advise is not to waste such an opportunity.

Adding just the library convention into the compiler with fallback to haxelib borders on trivial. It's easy to do and breaks nothing, providing a clear migration path. Because of that, it's probably even worth adding to Haxe 3.

As for Haxe 4, I think the idea of supporting parallel Haxe versions at least deserves thorough consideration. If it's too hard to add, that's one thing. But it shouldn't be dismissed merely on the grounds that "it's out of scope". It isn't.

jcward commented 6 years ago

Thanks for the thoughtful responses, @back2dos.

This is not reproducible in the slightest. ... if you run that script a couple of months apart, you get two different versions of the code.

Oh, I meant to specify a specific commit URL.

the conventions implemented in haxeshim should be built into the compiler. Beyond that point, I don't think we have to agree on anything. That's the beauty of it.

Yes, I think I was just realizing that. So haxeshim (hence, Haxe itself) would expect those conventional directory structures, and you're free to populate them however you choose. I like this a lot (this was the core of my "simple script" idea -- just tell Haxe to look in some conventional folder, and beyond that, I'll script a reproducible setup.)

There does need to be some name / version conventions on the lib side, though, doesn't there? I suppose a library will choose to support lix, thus using lix conventions? It might be nice to be as generic as possible in that specification, so other tools can parse versions and dependencies.

Alright, well anyway, just trying to follow along. I appreciate the dialog. Cheers! 🍺

desttinghim commented 6 years ago

Is this still open? The due date was 8 days ago. I think I'll stick my $0.02 in anyway.

It's important to note that having reproducible builds is important to me, so I'm going to advocate anything that I think would bring haxe closer to having reproducible builds. I also have no clue how the internals of the haxe compiler or haxelib work, so bear with me.

EDIT:

aliokan commented 6 years ago

Who can push this subject on the next step? @ncannasse

The package manager is one of the basics in the tool chain. We regularly reach the limit with haxelib on projects with a lot of dependencies.

I really hope it will be part of the next major release. :)

jonasmalacofilho commented 6 years ago

I would also like to bring back a point previously made by @andyli:

it is important to consider how to handle "native code" (could be compiled ndll, C/C++/JS/... compiled libraries or source files).

ncannasse commented 6 years ago

The topic has been quite discussed, thank you to all the participants. We are currently recruiting for HF (see https://haxe.org/blog/hf-is-recruiting/) and new haxelib will be one of the tasks to prioritize in our upcoming developments.

Simn commented 6 years ago

Sorry about that, that was bad timing on my part. I'll close this issue for now.

... to be continued

kevinresol commented 6 years ago

Just to make sure this is not forgotten

kevinresol commented 5 years ago

Is this happening in haxe 4?

aliokan commented 4 years ago

@ncannasse, @simn and @back2dos isn't it the good time to start the stage 3? ;)