improved dependencies

[dimsmol]

moved from here

extended dependency syntax

I agree that allowing links to git repos covers concern about beta releases.

Here I want to discuss how we want to specify such "link to something" dependencies (in future they can be not only links to git, but also links to tarballs, local dirs, etc.). There are at least two possible options here:

• Use the link as a constraint, like this: "user/lib": "git://github.com/user/lib.git#commitish" (this is the way npm works)
• Use the link as a name, like this: "git://github.com/user/lib.git#commitish": "1.0.0 <= v < 1.1.0"

While the second approach is something people are already trying to use, I think it isn't what we really want. There are some reasons against it:

• It's very logical to use a link as a constraint, because it is actually a constraint saying "I want exactly the version I'm referring to"
• Link destination is already an exact version, so having additional constraint on the version has no benefits besides of ability to double-check author's intentions what effectively means author should specify her intentions twice, which is tedious
• We can have other dependencies in the graph referring the same library and if they will have different name we won't be able to detect versions conflict

The third reason is the most important one. Here is an example. Let's assume we have the package x/main which depends on libraries y/a and z/b, but y/a also depends on z/b. The author of x/main wants to experiment with beta version of z/b, so she updates the dependency to refer git://github.com/z/b.git#beta. But she is not aware that y/a also has a dependency on z/b, e.g. "z/b": "1.0.0 <= v < 1.1.0".

Deps graph may look like this:

x/main ---> y/a ---> z/b
     \---------------^

If x/main will have updated dependency as "z/b": "git://github.com/z/b.git#beta" we will immediately find the conflict, because there is no guaranty thatgit://github.com/z/b.git#beta required by x/main is compatible with 1.0.0 <= v < 1.1.0 required by y/b.

But if x/main's updated dependency will be "git://github.com/z/b.git#beta": "1.0.0 <= v < 1.1.0" there is no guaranty that it has any relation to z/b referred in y/a and we cannot find they are different versions of the same lib.

So, I propose to go the way npm goes and stick to the first option, namely "user/lib": "git://github.com/user/lib.git#commitish". Maybe we can invent some other approach, but the second option doesn't look good anyway.

dependency conflicts

In the case described above, we have the conflict. But we can fork y/a, update it's deps to use "z/b": "git://github.com/z/b.git#beta" and use github link for forked version of y/a in x/main. This will require a lot of actions, but anyway we can do it and then experiment with beta of z/b as we need.

But the deps graph can be more complicated. There can be a number of libraries depending on z/b or we can have something sitting very deep into dependency graph depending on z/b. In this case we will need to fork and patch a lot of libraries and it becomes unreasonable. We cannot ask all the authors of those libs to do something for us because we aren't even sure yet that beta version of z/b is what we really need.

Example of complicated graph:

x/main ---> y/a1 ---> y/a2 ---> y/a3 ---> z/b
     |----> z/k1 -------------------------^
     |----> z/k2 -------------------------|
     |----> z/k3 -------------------------|
     \------------------------------------/

In this graph we need to patch all the packages y/a1 - y/a3 and z/k1 - z/k3 to have a chance to try git://github.com/z/b.git#beta with x/main.

How does npm deal with this? With npm there will be no conflict at all, x/main can use one version of z/b and the other libs can use other versions at the same time. This is a way how npm works, but it causes very bad problems, so we cannot use it.

Instead, I propose to add an ability to force top-level constraints over all the constraints dependency packages may have. For instance, having "z/b": "force git://github.com/z/b.git#beta" could make all the dependency packages use git://github.com/z/b.git#beta regardless of their own constraints for z/b.

While this is a potentially dangerous feature, it can be very helpful for experiments and can be a savior if you need to adopt an urgent security fix or quickly workaround a bug in someone's package dependencies. Of course elm-package should output all kinds of warnings when dealing with this option and it would be better to prohibit packages with forced deps from getting into official repository.

caches and downloaded packages dirs

Currently, downloaded packages are saved in the package cache and elm-stuff directory under the paths like user/name/version. But what should be the path for a github link?

Surely it cannot be user/name/version because the package can still have no proper version assigned (as @evancz said you may think it will be major change but end up with minor improvement or vice versa). So we cannot guess the version of the beta package without potentially clashing with some future release version.

npm solves similar problem by translating links into directory names, so git://github.com/user/lib.git becomes something like git_github.com!user/lib (see details).

Going this way we could have path like user/lib/translated_url. Is this a good way to go?

getting package constraints

Currently elm-package gets deps for the package from the store, checks everything and only then starts to download files. For github links there will be no entry in the store, so the only way to get package deps is to download it's elm-package.json and parse it.

For git, however, there can be a problem with downloading a single file from a repository. For instance, there is no way to download a single file from github using git+ssh protocol (which is preferred because it allows to use ssh certificates transparently). The same problem is present for tarballs as well.

Will it be ok to download the whole package just to get it's dependencies list and probably fail with conflict right after that? I think for certain situations we just haven't any other way, but maybe there can be some other ideas?

dev dependencies

@rtfeldman, according to your proposal to use "../": "1.0.0 <= v < 2.0.0" dependencies for tests subfolder to indicate it should inherit all the dependencies of parent folder. I think it would be better to have syntax like "inherit": "../" for this, because there is no reason to have version constraint here (as explained above) and inherit keyword will more clearly indicate what's going on.

But it can be even better to adopt npm's idea of dev dependencies. Those are dependencies that are meant to be installed in dev environment which may need additional libs for tests, build process or something. This way you will just add everything you need for tests into dev dependencies of the main elm-package.json.

What do you think?

[dimsmol]

@rtfeldman, found your first mention of local testing project vs dev dependencies. It looks like in fact you wanted just a dependency on the project in some (e.g. upper level) directory. Then we don't need any special keyword for it and according to my proposal above it will be just a regular dependency referencing a directory above, like "user/pkgname": "..".

[rtfeldman]

just a regular dependency referencing a directory above, like "user/pkgname": ".."

Yeah, this seems like a better way to go for local dependencies. :+1:

Also totally agree on "user/lib": "git://github.com/user/lib.git being better than the reverse. Changing the package name will just cause problems down the line when you go from the experimental version to the release and suddenly realize there's a problem now that it has its "real" name and a concrete version number.

having "z/b": "force git://github.com/z/b.git#beta" could make all the dependency packages use git://github.com/z/b.git#beta regardless of their own constraints for z/b.

It seems like such a feature should not be allowed for anything other than experimental packages, but it also seems like it should be automatically enabled for experimental packages (they never don't want it; either it's a harmless no-op or else you can't possibly build without it).

Putting these two thoughts together, it seems like rather than adding a force keyword, it would be more straightforward to say "URLs and local file paths are considered to satisfy any version bounds." If it turns out that after installing those packages your build breaks, well, caveat emptor on using nonstandard packages. :wink:

it would be better to prohibit packages with forced deps from getting into official repository.

As is currently the case, I think to get into the official package repo, a package should only be able to depend on its own local files plus other packages in the official repo. No bare Git URLs allowed, ever.

Going this way we could have path like user/lib/translated_url. Is this a good way to go?

Makes sense to me! :+1:

Will it be ok to download the whole package just to get it's dependencies list and probably fail with conflict right after that?

Doesn't really seem like there's any way around it, and these aren't terribly large downloads on average anyway. I wouldn't worry about it, honestly.

But it can be even better to adopt npm's idea of dev dependencies.

I don't like that dev dependencies couple package management with build environment. They don't solve any use cases that can't already be solved using the tools of remote git URLs and local file paths, so I'd rather we just stuck with those. :smile:

[dimsmol]

I don't like that dev dependencies couple package management with build environment. They don't solve any use cases that can't already be solved using the tools of remote git URLs and local file paths, so I'd rather we just stuck with those. :smile:

Agree. Even if we'll find any cases that really need dev dependencies, we can always decide to add them in future.

[dimsmol]

Summary

Link Constraints

• Git link can be used as a constraint for a dependency
  • example: "user/lib": "git+ssh://git@github.com:user/lib.git#commitish
  • in future more kinds of such link constraints can be allowed, e.g. path to directory, link to tarball, etc.

Conflicts

• Link constraint conflicts with any version constraint for the same package
  • (link constraints can be used for experiments and the link target may have no properly assigned version yet, so we cannot compare the versions)
• Link constraint conflicts with any other link constraint for the same package unless the links are equal literally

Forced Constraints

• Forced constraint ignores any conflicts with the other constraints any of dependency packages (direct or indirect) may have for the same package
• Link constraints are always forced, version constraints are never
  • in future more flexibility can be easily added if needed

Warnings and Restrictions

• A warning is issued for any usage of a link constraint
• A warning is issued for any conflict ignored due to usage of a forced constraint
• Packages using link constraints are prohibited from getting into official Elm repository

Implementation Details

• Link constrained dependency path name used within the cache and elm-stuff directories is the link itself, but transformed to have better look and no unsafe characters
  • example: git+ssh://git@github.com:user/lib.git#commitish becomes something like git_github.com!user/lib!commitish
  • exact transformation rules to be defined
  • see also npm implementation
• During dependency resolution process, elm-package.json of the link constrained dependency package must be obtained to get it's own dependencies. For this reason the package can be downloaded on the dependency resolution step.

[jamesmacaulay]

If non-version constraints are going to be introduced, I propose that we keep things explicit and use objects:

{
  "user/published-lib": "1.0.0 <= v < 1.1.0",
  "user/git-lib": {"git": "git://github.com/user/lib.git#commitish"},
  "user/local-lib": {"path": "../.."}
}

This way there is no ambiguity possible from introducing new string formats for new source types. If it's a string, that means it's a version constraint, which would be sugar for {"version": "..."}. Otherwise it's an object whose constraints are specified explicitly by the keys.

Edit: alternatively it might be good for the "commitish" of a git dependency to be in its own property of the constraint object, e.g. {"git": "git://github.com/user/lib.git", "rev": "commitish"}...and/or for the #commitish suffix to be sugar for that.

[grncdr]

I'm mostly a casual user of Elm, but have spent quite a bit of time in my life working on dependency management tooling, and I'd like to humbly suggest starting with a much smaller feature/increment.

@rtfeldman said way back in November:

I think "arbitrary Git URL" covers every missing case except "relative local file path"

I think it's at least interesting to consider the inverse: support for Git URL's is really support for local paths, with the local path being predefined and some convenient automation of git clone. Support for packages in arbitrary local paths support is strictly more powerful if you're willing to accept using another tool for the Git part. (Not to mention that Git URLs only support Git, what if my company is using some other SCM?).

Practically speaking, support for relative local paths seems like it would be much simpler to implement, and there are already a myriad of ways to get arbitrary Git URLs to show up at a desired specified local path. Two of them of them (submodules & subtree merging) ship with Git itself. It's true that git submodule is hardly at the level of Elm's tooling in terms of "Just Works", but it's a passable solution for teams working with private repos or "I want to try out the master branch of some dependency" type situations. Everybody seems to agree that packages shouldn't be published with git URL dependencies (and I'd also exclude local packages as well personally), so external management of non-registry dependencies would only be taken on in private teams/experimental situations.

To be clear, I'm not trying to discourage anybody from building support for git links! But I know @evancz is a big fan of actionable issues :smile:. Local path support (in the form proposed by @rtfeldman) seems immediately actionable, and gives some breathing room for figuring out how best to support Git URLs.

[opsb]

A concrete scenario where local paths would be really useful is with elm-test. Currently you have to manually copy dependencies from your project's elm-package.json into your test elm-package.json. If you could simply point to your project as a dependency then you'd automatically inherit the packages.

[naddeoa]

Has there been progress on this lately? I work at a company with a very large and mature build system and I think I have a few scenarios that I can offer that would prevent me for getting some substantial adoption if they weren't considered.

[rtfeldman]

@naddeoa could you start a thread on elm-discuss explaining those scenarios and linking to them here? It would be useful to know about them. 😃

[naddeoa]

Sure thing. I'll spin up a thread about it tomorrow

[naddeoa]

As I was typing it became pretty obvious that I need to use the package system a little more before making suggestions. I'm going to attempt to create a library out of the thing I've been writing to learn Elm before posting so I can speak a little more confidently about potential solutions.

[naddeoa]

Ok, I just submitted a thread for approval. Feel free to let me know if you're looking for something a little different. I just tried to explain my issue and propose a potential solution.

[Ljzn]

I really want something like elm-package install --local