support for git dependencies

[clemos]

It doesn't seem possible to have git dependencies currently :

"dependencies": {
        "hxnodejs": "https://github.com/HaxeFoundation/hxnodejs.git"
}

The above seems to fail. Or maybe there is another syntax ? If not, then I think it would be nice to support git dependencies, pretty much like npm does.

[nadako]

note that it would be great to also be able to specify commit number/branch/tag (like npm does I think)

[clemos]

Yes, it's like npm install git+ssh://git@github.com:npm/npm.git#v1.0.27 see https://docs.npmjs.com/cli/install

[hexpunk]

+1 for this

[nadako]

As mentioned in https://github.com/HaxeFoundation/haxelib/pull/135#issuecomment-163228477, I think we should adopt NPM dependency format, see https://docs.npmjs.com/files/package.json#dependencies.

[ncannasse]

I'm fine with following anything that is quite standardized as NPM seems to be, as long as we keep backward compatibility with our previous format, so users can continue downloading and submitting libraries using the old format.

[boozook]

1. In this case haxelib.json must be exists in root of repo;
2. What about another (potential) supported systems such as Mercurial or Subversion? Should we detect what use by url? No :).

[porfirioribeiro]

Maybe something like Cargo from rust lang? http://doc.crates.io/manifest.html#the-[dependencies]-section It accepts git repo's as a depedency. The Cargo.toml is either used on Lib's or app's to specify options including dependencies.

[nadako]

That is interesting stuff, thanks!

[boozook]

I like the Cargo impl too. but this is new format and will change the our current format: From {"libname" : "ver"} to {"libname" : {"version":"", "git/hg/svn":"", "branch/tag/rev":""} }.

But we can really preserve backward compatibility. We can easy support old(current) and new formats - detect format via Reflection and use the special provider (parser) for it (Or EitherType).

[boozook]

Another one variant - namespace-like:

"dependencies": {
        "libFromHaxelibA": "",
        "libFromHaxelibB": "10.2.4",
        "libFromGit": "git::https://github.com/HaxeFoundation/libFromGit.git",
        "libFromHg": "hg::https://bitbucket.org/HaxeFoundation/libFromHg"
}

But without tags/revs/branches.

[boozook]

I love the NPMs format but it doesn't support any other control version system.

[nadako]

The npm tool itself doesn't support anything but git, but the URL format looks extensible enough (e.g. hg+http://... similar to supported-by-npm git+http://....

[nadako]

As discussed in #105, it would be also nice to support http://... urls, which is also supported by NPM. I propose we borrow npm's url format and use it for both "haxelib install" and dependency urls. We can implement it using some kind of "providers" architecture as it's done in @ibilon's sources related to #217

[nadako]

This raises more questions: should we / how to store multiple versions of a library checked out from some vcs.

For example, library A declares "somelib": "git://some/lib#rev1" dependency which is a rev1 commit from some/lib while library B wants "somelib" to be e.g. git://other/lib#rev2.

Technically, we can support that by generating some unique version identifier from a vcs url (like git__some_lib__rev1), but do we want to go that route?

[ibilon]

It depends if we can make the hypothesis that the user won't modify the content.

In that case when installing we only fetch, and when compiling we do a checkout of the required commit/branch/tag.

[grosmar]

Hello,

We have the same need so I've investigated the sources and this thread, and have some conclusion and some code now.

• As @ncannasse said we should be backward compatible, so even if we extend the syntax we should keep working with the existing one. I've considered it through all the points.
• I've found a strange concept at this moment. When we use a library, we can install it's dependencies two ways: if the library has it's own build.hxml we can install them from it with haxelib, or if it has a haxelib.json and we install it through haxelib install or haxelib git commands then it read up the dependencies. But for the haxelib.json there is no implementation to install it when is it on my local drive (like when you use npm install and it reads up the local package.json and install all the dependency from it) I think we should be able to follow the concepts of other package managers, and we should be able to express our dependencies with versions in the haxelib.json locally, even if this is our main project. And after we should just express the need in the build.hxml. But for keeping the backward-compatibility, i would implement also the git dependency support in the build.hxml So for this i've tried the following syntax: haxelib install <haxelib.json path> So the haxelib install itself looks for the haxelib.json file in the current directory. If it find it, start it's installing. If it get as a parameter the path for the haxelib.json than it will install that one.
• About multiple git dependencies: I would follow the concept that @nadako said: generate git_somebranch syntax. But only as a second step, because it raise a lot of questions. Anyway, now i would change that behavior in the code when install git dependency install it as a dev dependency. There is no point in it, it's more than enough to just set the current version to the git. It works perfectly and no need to force to that version.

To make it backward compatible, i've tried with the following syntax: build.hxml: -lib something:git:https://github.com/grosmar/something.git#commit-ish This is fully compatible with the existing haxe compiler, because it splits at ":", so it will understand to look for the "git" version folder. If we later introduce support of multiple versions, than it can be git_mybranch:https://...

commit-ish would mean anything that can be passed to the git checkout command

haxelib.json dependencies: { "something: "git:https://github.com/grosmar/something.git#commit-ish" }

Later we can introduce maybe a structured syntax for the haxelib.json, to describe all the features of git dependency (like subfolder).

So for now i would not follow fully the npm style git+..., but i would use git:... to keep the compatibility.

According to these conclusions i've made some implementation here: https://github.com/grosmar/haxelib/tree/library_git_dependency

This branch implement the following features that i've mentioned above:

• haxelib install
• build.hxml describe git dependency
• haxelib.json describe git dependency

I still need to write the tests if we can agree on the things.

Also we could think about some extra syntax to describe subdir like git:https://github.com/grosmar/something.git#commit-ish@subDir And we should be able to express the version (tags). The #commit-ish part covers it, but maybe it would be nice to directly express the need only for tags. But i have no good idea for this. Any suggetions?

What do you think guys?

[nadako]

I'm yet to look at the code, but the proposition looks nice.

I agree about expanding haxelib.json purpose so it's not only used for "libraries", but for projects in general, similar to npm's package.json. We can even add more functionality to it, like haxelib test that reads from haxelib.json that is has to run haxe test.hxml, but that's another topic.

git:https://github.com/grosmar/something.git#commit-ish@subDir

another option would be to add another colon here, e.g. git:https://github.com/grosmar/something.git#commit-ish:subDir :)

as for tags, I think commit-ish is sufficient.

[grosmar]

Let me know if you have time to check the repository. I happily finish the devs if we can finalize the concept.

[nadako]

I'm looking at the code right now. Can't say I like it very much, but that's not about your part, but haxelib codebase in general. I think if we decide on the design here, any code will do as long as it works, then we can refactor it from there.

Also, I remembered one issue with losing dev-mode for vcs dependencies - using a subdir inside a repository for the haxelib (see https://github.com/HaxeFoundation/haxelib/issues/263#issuecomment-164477060). I wonder though if that feature used at all and worth caring about...

[nadako]

Anyway, I can't think of any issues regarding the proposed syntax.

[grosmar]

I also cannot say that i like the code, but i didn't want to change deeply the codebase because of this implementation.

I can look into the subdir devmode problem later. But for me this subdir stuff sounds a bit nasty anyway. I prefer one repository for one lib, so why should i need a subdir?

So can we agree on the followings as a final decision:

• support haxelib.json file also for local dependency descriptions, and use the following syntax to install them: haxelib install <haxelib.json path> (if no path given it tries to read haxelib.json in the current directory)
• support git dependencies in haxelib.json with the following syntax: dependencies: { "libname: "git:https://github.com/grosmar/something.git#commit-ish:subdir" }
• support git dependencies in build.hxml with the following syntax: -lib something:git:https://github.com/grosmar/something.git#commit-ish:subdir

commit-ish means anything that can be passed to git checkout command subdir is reference for the subdirectory of the git repository

And later we can improve the implementation and use versioned git folder like this: git_commit-ish_subdir

This way we can have multiple git version checked out and used. But I don't want to mix it now.

So can we agree on this syntax? If yes, i finish the development and create the tests.

[nadako]

I can look into the subdir devmode problem later. But for me this subdir stuff sounds a bit nasty anyway. I prefer one repository for one lib, so why should i need a subdir?

No idea, I think I have never needed that.

support haxelib.json file also for local dependency descriptions, and use the following syntax to install them: haxelib install (if no path given it tries to read haxelib.json in the current directory)

Fine with me :+1:

support git dependencies in haxelib.json with the following syntax: dependencies: { "libname: "git:https://github.com/grosmar/something.git#commit-ish:subdir" } support git dependencies in build.hxml with the following syntax: -lib git:https://github.com/grosmar/something.git#commit-ish:subdir

Not sure about the subdir syntax, but if we lose that feature, we don't need it anyway :)

I don't feel like I can decide for the HF here though, so let's summon @ncannasse @Simn @andyli. Check it out guys, this feature is quite important IMO.

Also ping @fponticelli and @andywhite37, since they are using the hmm tool they wrote, so they may have an opinion on this!

[andyli]

I really want this feature too. However, I don't think we will have enough time to make sure the design and implementation is solid enough for the upcoming haxe 3.3 release. So I will suggest looking into it after that.

[nadako]

Well, the good thing is that we can release haxelib separately. In fact, there's an issue about that.

[nadako]

Hmm, but we want some release bundled with haxe though. However I don't see a problem with including the current proposed stuff in the release, the changes are minimal. IMO the only thing we have to think of is the devmode/subdir issue.

[grosmar]

I also agree the changes are minimal, and kind of new, that won't break any existing behavior.

Later i will look into devmode/subdir stuff and can talk about it. But IMO it's pretty strange to support this bad practice.

[nadako]

Later i will look into devmode/subdir stuff and can talk about it. But IMO it's pretty strange to support this bad practice.

I agree, but that's an existing feature, so removing it is a breaking change, so someone has to make a hard decision about that.

[grosmar]

Your right, so i will check it anyway :)

[grosmar]

I've looked into it. The problem is simple: There is no way to tell haxelib which subdirectory used for the current checkout, only if you set a dev path.

I know a few solutions for this:

• Implement support in the .current descriptor file the path referencing, but it has also an effect on the haxe compiler, so i would avoid it now.
• Use spare checkout feature of git, but i think it's more tricky feature, i don't know it's limitations.
• After checkout the git repository use only the given subDir, and delete everything else. This way we loose the possibility to manually update a git repository, but we can use the subfolder without dev dependecy.
• Only use library as a dev dependency when the subfolder is given. I think this is the easiest solution. Only con that I can mention is causes some inconsistent behavior with and without subDir parameter

And one more note: Because before noone had the possibility for configured git dependency in haxelib.json and build.hxml, does it really matter if we don't implement it in these? So if someone really need a subfolder he can manually install it in the oldschool way: haxelib git something https://github.com/grosmar/something.git master subDir and it will be solved with the dev dependency

What do you think?

[nadako]

When I was thinking about this while working on #263, I mostly thought about the first solution. It shouldn't affect Haxe compiler itself, since it just calls haxelib path, but still I'm very reluctant to change any file formats (especially if it's a breaking change, like this one), because if we do that, we might as well redesign the whole thing.

Solutions 2 and 3 look very ugly, let's not do that! :)

Solution 4 is a great idea IMO. Especially if we all can agree that the subdir feature is not worth it, we can deprecate it, support it with the devmode hack for some time, and then simply remove it (however I'd just remove it completely, TBH).

[grosmar]

When I was thinking about this while working on #263, I mostly thought about the first solution. It shouldn't affect Haxe compiler itself, since it just calls haxelib path, but still I'm very reluctant to change any file formats (especially if it's a breaking change, like this one), because if we do that, we might as well redesign the whole thing.

Yes, that was my other complain about it, it's a change that is not backward compatible.

Solution 4 is a great idea IMO. Especially if we all can agree that the subdir feature is not worth it, we can deprecate it, support it with the devmode hack for some time, and then simply remove it (however I'd just remove it completely, TBH).

For me it would be the best, it causes the least pain and the least impact on the system. Even more it's fully backward compatible.

And also don't forget about that, we agreed to not implement the tag support (called version in the source), because the #commit-ish property fulfills this need.

So can we agree on this solution?

[nadako]

I don't feel like I can decide for the HF here though, so let's summon @ncannasse @Simn @andyli. Check it out guys, this feature is quite important IMO. Also ping @fponticelli and @andywhite37, since they are using the hmm tool they wrote, so they may have an opinion on this!

Let's wait for these guys for a couple days and if there won't be any reaction, just do it :)

[andyli]

I haven't read the whole thread yet and I'm going to do so today. My concern is that, even if it wouldn't break existing functionalities, we still care forward compatibility. I am worried if we introduce this and later found bugs in the design/implementation, it may be hard to patch it without breaking existing code which depended on the buggy behaviour.

we can release haxelib separately

I had bad experience with selfupdate in the past and haven't used it for a long time. I hope it has been improved since then... If you've good faith in selfupdate, we don't need to rush for the haxe 3.3 release either, right?

[nadako]

I am worried if we introduce this and later found bugs in the design/implementation, it may be hard to patch it without breaking existing code which depended on the buggy behaviour.

Well, that's a fair concern, but how do we deal with that then? We won't know if it has issues until it'll be actually used in the field. I'm getting tired of issues with a long discussions but nothing done :)

If you've good faith in selfupdate, we don't need to rush for the haxe 3.3 release either, right?

You're right, we need to test/fix selfupdate (#172), though I had an idea we still have a couple months before haxe 3.3 release which should be enough to deal with remaining issues. Isn't 3.3 gonna be released at WWX?

[grosmar]

Well, that's a fair concern, but how do we deal with that then? We won't know if it has issues until it'll be actually used in the field. I'm getting tired of issues with a long discussions but nothing done :)

Yep I agree. I really don't want to push anyone, but how to deal with fears of future? I will do my best and make all the tests to be sure it's working properly. If we will face the wall after it, then that's life, and we have to deal with it then. But until we should go forward. IMHO this is a must-have feature for a package manager system.

And even more, we should be shortly able to manage semver version comparisons (i've seen the pull request for it), and also private git version handling (proposed here above)

[andyli]

On Thu, Mar 10, 2016 at 6:20 PM, Dan Korostelev notifications@github.com wrote:

Isn't 3.3 gonna be released at WWX?

We should ship Haxe 3.3 RC1 on 2016-04-03 according to our plan: https://groups.google.com/forum/#!searchin/haxedev/Haxe$203.3$20release$20plans/haxedev/Wqo2MeyhnY8/TTHTmhVBAQAJ

[andyli]

On Thu, Mar 10, 2016 at 6:27 PM, Balázs Zubák notifications@github.com wrote:

Well, that's a fair concern, but how do we deal with that then? We won't know if it has issues until it'll be actually used in the field. I'm getting tired of issues with a long discussions but nothing done :)

Yep I agree. I really don't want to push anyone, but how to deal with fears of future? I will do my best and make all the tests to be sure it's working properly. If we will face the wall after it, then that's life, and we have to deal with it then. But until we should go forward. IMHO this is a must-have feature for a package manager system.

And even more, we should be shortly able to manage semver version comparisons (i've seen the pull request for it), and also private git version handling (proposed here above)

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxelib/issues/238#issuecomment-194779556 .

I'm not stopping you to design or implement it. I just suggest to do it on another branch and have it merged after the next release ; )

[Simn]

I agree with Andy, let's focus on getting selfupdate stable so we can take our time with features like this.

[grosmar]

Ok, then I'll work on it according to "docs" that i've wrote above, and i'll let you know guys when it's finished.

[nadako]

Fine with me! We'll focus on making haxelib easily updateable and then will introduce this feature.

[nadako]

Just letting you know that we've implemented the new self-update mechanism and new version notification. So we can deliver this at any time.

[ibilon]

I've been thinking more about this and I don't think this is a good idea.

• Breaks our ability to get the full recursive list of dependencies, and therefor breaks the "no conflict" check
• Adds git/hg as a requirement besides haxe/neko (not for all libraries, so random from an user perspective)
• If the url is allowed without a specific commit/tag goodbye reproducibility (if you install it at a bad time can be broken)
• Can't do the things described in #267 and #290

Also what happens when two libraries add a different git url under the same name?

"dependencies": { "theSuperLib": "https://github.com/someguy/superlib.git" }

"dependencies": { "theSuperLib": "https://github.com/someotherguy/haxe_lib_super.git" }

And since I fail to see the point of this (if you are sending your lib to haxelib with not the dependencies?) that looks like a nightmare for nothing as far as I see (could be extremely useful and I just don't realize it, but still the issues apply).

[nadako]

@ibilon These concerns are indeed valid for libraries published on lib.haxe.org, but publishing a library with git dependencies are just bad style and is headache for library's author (if they want their library to be used by someone anyway). I doubt many authors would do that. At least it's not a problem on NPM.

OTOH, as discussed above, I still think it makes sense to expand the purpose of haxelib.json file to something similar to package.json in NPM world. I.e. it'd be a manifest file file for your application, even if you're never going to publish it. It can describe dependencies, run configuration and everything available for libraries, but it'll do it for your project. In this use case, git dependencies make a lot of sense IMO.

Aside from that, it can be useful when working in a company with private repositories, when different teams create different modules and it's often much less annoying to just use git instead of doing releases on some private haxelib server.

Also what happens when two libraries add a different git url under the same name?

One option also discussed above is generating an unique version directory name like git_<repo_path>_<commit>.

[ibilon]

I doubt many authors would do that.

I think it shouldn't be allowed, if we can make the hypothesis that libraries on lib.haxe.org only have dependencies on lib.haxe.org then there are several things we can do like #290.

it'd be a manifest file file for your application

I see, I'm not sure haxelib is the best place to have this, but I don't want another file format either.

In that case the format should be relaxed, all the fields (except the name?) should be optional, for instance having at least a contributor for your application that'll never go on lib.haxe.org doesn't make much sense.

Should it be picked by haxelib install all? And/or used with haxelib install haxelib.json?

[back2dos]

I see the advantage of having published haxelib only depend on other published haxelibs.

At the same time, not all libraries are on haxelib for various reasons, like lack of stability or dependency on an unstable Haxe version.

I do think that haxelib git should be able to deal with other git dependencies.

Among other things, this makes it easier to use haxelib with an ecosystem of private repos.

On Fri, Mar 18, 2016 at 3:54 PM, Valentin Lemière notifications@github.com wrote:

I doubt many authors would do that.

I think it shouldn't be allowed, if we can make the hypothesis that libraries on lib.haxe.org only have dependencies on lib.haxe.org then there are several things we can do like #290 https://github.com/HaxeFoundation/haxelib/issues/290.

it'd be a manifest file file for your application

I see, I'm not sure haxelib is the best place to have this, but I don't want another file format either.

In that case the format should be relaxed, all the fields (except the name?) should be optional, for instance having at least a contributor for your application that'll never go on lib.haxe.org doesn't make much sense.

Should it be picked by haxelib install all? And/or used with haxelib install haxelib.json?

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxelib/issues/238#issuecomment-198395754

[ibilon]

I do think that haxelib git should be able to deal with other git dependencies.

I agree, as long as it's not on haxelib you should be able to use any type of dependency: haxelib, git, http/local zip or even a path.

Though about something, why is the git url used as version? That's clearly the library "name" (if you think of name as a uri it makes more sense) and commit/tag/branch the version. Same for a zip file or path (which will always have an empty version).

That way there's a lot less name conflict between libraries, especially between published and non published. With the exception of path all name would uniquely define a library.

[back2dos]

On Sat, Mar 19, 2016 at 11:32 AM, Valentin Lemière <notifications@github.com

wrote:

Though about something, why is the git url used as version? That's clearly the library "name" (if you think of name as a uri it makes more sense) and commit/tag/branch the version.

Well, not necessarily. The same library can be in different places, for some reasons. For example assume you use a lib from http://github.com/joe/libx and you need to make a fix and use it right away, then you can fork to http://github.com/ibilon/libx and use that as a dependency while waiting for your PR to get accepted.

Same for a zip file or path (which will always have an empty version).

This also depends. The following url defines a version rather clearly: https://github.com/HaxeFoundation/haxelib/archive/3.2.0-rc.1.zip

If we had https support (which doesn't seem worthy of inclusion in the standard lib, because clearly it's not as fundamental as a routing mechanism or an ORM layer or a template engine), then I would say GitHub releases would actually be the preferable way to depend on libs from GitHub, as it doesn't rely on git and is generally much faster.

That way there's a lot less name conflict between libraries, especially between published and non published. With the exception of path all name would be unique.

Well, firstly, I think -lib https://github.com/joe/libx is a bit cumbersome to write. Second of all, being able to shadow a haxelib with a git lib is a feature to me. If that's not what you want, you can define the git dependency under a different name.

[nadako]

IIRC our repository server currently validates that all dependencies are present in the database. I think we could as well check for absence of git (or, rather, anything but "normal") dependencies when submitting to lib.haxe.org. That makes sense IMO.

[grosmar]

Sorry guys, i didn't disappear but became extremely overloaded in this time. As the things getting better I will get back to this issue.

[grosmar]

Hi!

Finally I'm back on the issue. I've updated with all the changes from the master since, and here is now the branch that contains the modifications: https://github.com/grosmar/haxelib/tree/git-dependencies

As I see there are still a few open questions, also I added some.

Let's talk about them and make a decision:

• Tag support. So I've checked, it works well with just saying #tags/mytagname because these works as a branch in git, so i would not add more complexity to syntax.
• "Also what happens when two libraries add a different git url under the same name?". I still not manage different branches under different versions as we said before, like git_branchname. Is it okay to have a first version without this feature?
• Check subdependency branch changes. When I try to install the dependencies from haxelib.json, and I find a repository that has been checked out, I try to git update, and I do the same with all the subdependencies. But still not managing if a subdependency branch changed. I don't know if I should switch the branch automatically, or it should be part of the second step, when we manage multiple branches with different names?
• Subdir support. So we agreed to not support it through git dependencies, but as i see from the code now someone implemented to set devmode if subdir has been defined, so this was if some still want to use git dependency with subdir he can do it through haxelib git ...
• IMO we should not support any git dependency on lib.haxe.org I think it would be crazy as @ibilon and @back2dos said. So we should have a closed ecosystem there.
• "If the url is allowed without a specific commit/tag goodbye reproducibility". I think your right, but normally it's the responsibility of the creator to make the decision. It's the same with normal haxelib dependencies: if you do not define any version it will use the latest one. So i would not change on this concept, even because when we develop something with private repos with lot of subdependencies then we always want to have the latest one from everything.
• "Should it be picked by haxelib install all". I think no, because it would break backward compatibility. Someone who get used to use that method will see different behavior now, and even more because every package manager or build system have a very simple way to express to setup in the current directory. For example npm install, ant (without any parameter!), etc... So haxelib should also have a simple way to express setup.

Guys, please look through the problems and if we can agree what to do we should finish this long topic. I think many people would be happy with this feature!