Project-local package dirs with precedence

[ozra]

Having worked a couple of years with iojs for the money, I really enjoyed the modularity of projects vs the C++ world. I've already had several problems with Nim, while hacking the compiler etc. that ~./.nimble/.. pkgs and global nim stdlibs are preferred over the ones from the dir I'm working in.

This is mostly a nim mod-lookup issue, and partly nimble.

Basically a project-local mod deps dir - like "node_modules/" in iojs/nodejs - it should have precedance. Also, those package dirs, should easily be pure github checkouts, for deving on deps, hand in hand with a project.

This way version matched packages can be stored per project - they could still be cached in home-dir/nimble. I'd prefer 'proj-local' pkgs as default. But an option would be ok, "nimble install local..."

https://github.com/nim-lang/nimble/issues/114#issuecomment-98890630 https://github.com/Araq/Nim/issues/2819#issuecomment-106007043 https://github.com/Araq/Nim/issues/2819#issuecomment-106015725

As for the module lookup, I guess that's in nim's domain.

[josephwecker]

Copying a discussion from another thread here where it belongs:

---

josephwecker:

BTW- my comment glossed over nimble's package versioning. Aside from that though it does allow for nimble in the future (if desired) to optionally install packages local to a project and just have everything work as usual. It'soff-topic but it has been extremely useful to me in just about every language I've used- js, ruby (rails especially), c, erlang, ...

---

dom96:

@josephwecker In what way is that useful?

---

josephwecker:

@dom96 I don't want to hijack this thread- so warning: somewhat offtopic from this issue.

And it is a minority case compared to the semi-global package cache as nimble correctly implemented, but generally it is useful as a more clear & abrupt separation of a projects personal space vs shared space; an extension of the reason nimble probably uses ~/.nimble/... instead of /opt/nimble or /usr/include/nimble.

There are various deployment / replication / compiler-farm / multi-project-editing-workflows / debug scenarios where it's very useful to have all or some of the dependent packages that the package manager acquired right there in the main project's subdirectories- neither affecting nor being affected by anything else on that machine. npm, rebar (for erlang-OTP), and rails are the ones I've actively used in the past that come to mind right away that have the ability to optionally install local to the project. In C I've done essentially the same thing in order to experiment with different stdlib "overrides" for super-high-availability zero-copy servers back at twitch.tv etc.

---

josephwecker:

@dom96 similarly- Ruby, perl, and to a lesser degree python have rvm/perlbrew/virtualenv for allowing any project to have its own version of the VM/compiler as well as its own "package space"/gemset/etc. separate from the shared one...

[dom96]

I think the current way to develop (dependency) packages is good enough (TM).

Clone the dependency repo. Work on it. Then run nimble install in its directory. Switch to your project and test it.

[ozra]

Ah, yes. Is there anyway of setting a "temporary" version / dev-version or something then - so it doesn't shadow the "legit" version of it? (The sandboxing we so dearly want)

So, if working on dep "foo" of version "0.4.7" - I don't wanna do the version bumping - it's up to upstream, but I don't want it to shadow foo@0.4.7, so I want my changes at foo@0.4.8 (patch-version +1), but without affecting the source...

What do you thing, some idea?

[dom96]

If you don't want it to overwrite the older non-dev version then you will need to bump the version.

[cdunn2001]

I think the current way to develop (dependency) packages is good enough (TM).

But could be better. Let's make it better.

@ozra and @josephwecker, the new --nimbledir flag is a step in the right direction, but I don't want to have to type that every time I run nimble in a particular workspace directory. (A "workspace" may contain multiple git directories.) My idea is for nimble to search for the nimble.ini first in the current directory (and maybe in the parent) which would then serve to define a workspace. If not found, then it could look in $AppDir/nimble/.

This is pretty easy to do. The first step is this change, which was rejected. It's worth reconsidering. For now, I'll experiment with my temporary fork, but I will continue to hope that it can eventually be merged back into nimble.

I'm very tired and maybe not thinking clearly enough. Has anybody else used Go? I think its build-system is impressively flexible. We can borrow ideas from there. An environment variable like GOPATH might be the way to go, rather than a configuration file.

[dom96]

So, in addition to --global (still to be implemented, install to /usr/lib/nimble, or somewhere more appropriate) and --local (default, install to ~/.nimble), you want to also have --project (install to $PWD/.nimble, or something like that)?

That's something I would be willing to implement, as it plays nicely with and will create a beautiful trifecta of --global, --local and --project.

I still feel like I don't understand the advantage this brings. Do you want to have multiple copies of the same git repository on your system, updating each separately as-needed, developing the copy that you want to test via some other package?

[cdunn2001]

Yes, multiple copies of git-repos. These are separate workspaces. I don't want to litter my "normal" Nim installation with any WIP.

I also think a local nimble.ini file makes sense, to over-ride the global one in ~. That's how many tools work, including git itself. Or an environment variable could define the workspace, which is basically how GOPATH works. However, command-line flags are almost equivalent to environment variables since I can use an alias within a given shell:

alias nimble=\nimble --nimbledir=/my/local/workspace/path

I can use different shells for different workspaces. That's good enough for me. So I guess --nimbledir works for me after all.

I don't know anything about --local/--global... I guess that comes from #80. Well, I think those flags are confusing when --nimbledir exists. Do they over-ride it? Do they only alter a default? If so, what happens when I specify both? It's confusing. In UNIX, most installers rely on --prefix, which might be /usr by default. For local installs, people always specify the local prefix. I guess you don't want to type --nimbledir=~/.nimble all the time. So how about this:

• If --nimbledir is provided, use that always.
• If not, then look for any parent directory called .nimble. If found, use that.
• Otherwise, use the global system default.

However, I prefer this simpler logic:

• If --nimbledir is specified, use it always.
• Otherwise, use ~/.nimble.

I don't see a value in --global. Explicit is better.

[cdunn2001]

Oh, I see. In #80, some people want nimble packages to come from multiple installations, particularly including the system root, /usr/lib/nim or whatever. I'll comment there.

[dom96]

I would prefer not to have Nimble search any parent directories. I think that searching for a directory called .nimble will only create confusion.

--global will act like an alias for --nimbledir:/usr/lib/nimble, if the user specifies a --nimbledir then the global will be overriden.

[ozra]

I prefer command line flags over PATH. But I'd really prefer the local .nimble conf. That way "nimble install" wouldn't accidentally overwrite system wide version if I'm tired and do the wrong thing (which incidentally does happen from time to time). This way deps stay stable system wide, while the WPC version is sandboxed and only "installed" project locally. It makes sense. It could require it to be in cwd (aka, calling nimble when in src/ dir won't work - no parent searching..) [ed: - would this make nimble install still work, and thus cause the problem I mentioned, or does that too require to be 'in the right dir', yes?]

[dom96]

I suppose I have the same question for you as I asked here: https://github.com/nim-lang/nimble/issues/80#issuecomment-110473481

What happens if I have the same package (of two different versions) in my GOPATH? How does Go resolve which one it is that I want?

i.e. how can Nimble disambiguate the local package vs. the system package?

[cdunn2001]

@dom96, Let's establish how to deal with multiple, unrelated workspaces first. The issue of precedence is better discussed in #80, in relation to system installations. (With Go, only standard packages are in GOROOT. If you want to edit them, you kinda need to check-out the entire Go compiler. I think that's fine.)

@ozra, A local nimble-conf file is an interesting solution for telling nimble where to install. However, if it contains absolute paths, then it cannot be in the repository. Also, if it doesn't look at parent directories, why not simply install in the CWD? I think this really gets at the fundamental problem with today's nimble. It's way too complicated (in different ways than we need). If I run nimble install in ~/workspace/foo, it goes into ~/.nimble/.... That's counter-intuitive. ~/.nimble has source-code already. It should use only the repos in its own sub-dirs.

With GOPATH, a workspace contains:

GOPATH=/home/user/gocode

/home/user/gocode/
    src/
        foo/
            bar/               (go code in package bar)
                x.go
            quux/              (go code in package main)
                y.go
    bin/
        quux                   (installed command)
    pkg/
        linux_amd64/
            foo/
                bar.a          (installed package object)

The repositories are in src/. When it builds any given package, it builds into bin/ and pkg/. If something is needed from the web, goget will pull into src/ and then build into bin/ and pkg/OS/PKG. That's simple. It's very easy to understand what came from where.

If you want to remove a repo, then remove it manually. Nimble does not need to clone into /tmp/. Repos are way smaller than binaries.

• http://golang.org/pkg/go/build/
• http://golang.org/cmd/go/

[dom96]

@cdunn2001 In your example, isn't /home/user/gocode equivalent to /home/user/.nimble?

[cdunn2001]

It would be equivalent if the repos are in ~/.nimble/src/mypkg/.git, e.g.

If the package is checked out in NIMBLE_DIR/src, then I don't want nimble to examine packages.json from the web. Use my repositories whenever possible!

And I want nimble to infer the NIMBLE_DIR if I run from a sub-dir of a valid NIMBLE_DIR.

[dom96]

If the package is checked out in NIMBLE_DIR/src, then I don't want nimble to examine packages.json from the web. Use my repositories whenever possible!

If I understand you correctly then that is what Nimble already does (as long as the package that you're building doesn't depend on a newer package than is available in NIMBLE_DIR/src.

[ozra]

If there could be some way to flag to nimble that a repo is locally deved - and so get a warning on nim install without bumping version (that is, nimsuggests sees that an existing package will be installed over, and then it warns when it knows I'm doing install from local repo.

How could this best be solved?

[cdunn2001]

But if I have checked out the repo into 2 places, they will each install into ~/.nimble, right? That's what we do not want. I need multiple workspaces.

[dom96]

I'm assuming you want to avoid installation since it's not required in Go (you just add the repo to your $GOPATH). So isn't this something to implement in the Nim compiler then?

[cdunn2001]

What do you mean by "installation"? go build installs into the workspace.

I want to build this package and all dependencies in my workspace, automatically pulling from the web whatever is missing. Isn't that what nimble is supposed to do?

[dom96]

Nimble already does that. It doesn't pull dependencies into your workspace but into ~/.nimble.

[cdunn2001]

That is the problem! It over-writes the contents of ~/.nimble. I want to bounce between workspaces. My day might go like this:

• Update a nim package. Build and test, integrated into a larger system. Let that run for a while. Do not modify it in the meantime.
• Oops. Bug report came in. Create a new workspace for that bug. Fix and submit changes for code-review.
• Get some basic work done. Use the stuff in ~/.nimble. Leave a long-running task running. Do not alter the ~/.nimble directory.
• In the meantime, switch to a different set of nim packages. Build and test them.
• Return to first set of packages. Oh, I see another crash. Fortunately, I can debug it because nothing has changed there.
• I see responses on the code-review. Return to that workspace and alter that code based on those comments.

I'm not sure that explains it well enough, but maybe you can get some idea of the complexity of my work. I need the filesystem to help me keep track of various ongoing tasks.

Please tell nimble to use the current workspace unless explicitly told otherwise. Please add the concept of a workspace. It's needed.

[ozra]

That describes a not to uncommon scenario! Which is the same reason I want the "sand boxing".. In my case 'workspaces' are just local git repos, they're all "re-synchronized" via my github origin, after these parallel dev situations.

[bluenote10]

@cdunn2001 What you describe was pretty much my motivation to suggest this.

In my opinion the cleanest solution is to move away from installing and switching packages altogether -- also because these are low level primitives. In general, package installation should not be a "global state". What I have proposed is the Maven/SBT/Ivy/Cargo-like solution: I would make --noNimblePath the default and in order to build against a certain version of a package the build tool has to bring in the appropriate path. If ProjectA needs LibX in version 0.1, ProjectB needs git hash 4e73f8 of LibX, while ProjectC needs a local modification of LibX -- no problem, you just declare these dependencies in their respective project build definitions. Running nimble build would take care of everything by assembling the build path in each case individually ("installing" a package if necessary). With global package installations / build paths such a scenario quickly becomes a mess. Switching between various nimbledirs is an ugly work-around since there is still "global state" (Example: Why is my build failing? Oh wait, I probably have to switch to another of my 7 nimbledirs...). There is a reason why most other build tools avoid this.

[cdunn2001]

@bluenote10, Great discussion at that link.

@Jehan says:

The problem with "nimble build" is that currently it has a hardcoded -d:release and no way to pass options other than through a config file.

@dom96 says:

I have actually recently tried Cargo myself and was surprised by just how similar it is to Nimble.

Yes, nimble build is like cargo, and it's not bad. But I need to build A, B, and C, where:

• A depends on B and C
• B depends on C
• I own A, and I am modifying A locally.
• B and C are in the nim-lang nimble registry. (Or whatever it's called.)
• I am modifying B locally.

See the problem? I'm not sure Cargo handles this either. gobuild handles it easily.

@bluenote says:

The "global" install of nimble is a non-issue. Nimble stores the packages in ~/.nimble/pkgs/_. If a project depends on a specific package version (could even be a git hash!) which does not yet exist, nimble will just create a new "global" copy.

"Non-issue"? That is not true! Diamond-dependencies make that system an absolute nightmare! An installation needs precisely one contour of packages, or nobody will have any idea what's going on.

@Varriount says:

@bluenote I definitely like where you are going with these ideas. I think, with a bit of planning and research, we could really make something!

Using Nim to build Nim is an interesting idea, per @Araq's comments. But that all distracts from the need for workspaces.

[Jehan]

Hello,

I think you have me mixed up with someone else. I think this is another "Jehan" in this forum you linked (or else I have big memory holes! :p) Just saying, since I received an automatic email and was wondering what this discussion I could not remember at al was about. ;-)

[bluenote10]

"Non-issue"? That is not true!

@cdunn2001 I fully agree, what I was trying to say was: It is mainly an issue because it is "in scope" by default. I currently don't see the issue if the build tool assembles the build path individually. Let's consider the following fictitious syntax in your example. Project A's build definition before edit:

deps += "B @ 0.1.2"
deps += "C @ 0.8"

Since the version specifier is a plain version string, running build now would pull the respective versions from the remote repositories (build path would contain e.g. ~/.nimble/pkgs/B_0.1.2/). If you want to use a local version of B in A, you can modify A's build definition to:

deps += "B @ /path/to/modification"
deps += "C @ 0.8"

The build tool now could follow the rule: If the version specifier is a local path, use the package from the local path. Running build now would not add any version from ~/.nimble/pkgs/B_*/ but just /path/to/modification to the build path.

Wouldn't such a behavior solve your issue? Or are you referring to the problem of conflicting transitive dependencies (e.g. B depends on "C @ 0.9")? This is obviously a much more difficult problem.

[cdunn2001]

Sorry, Jehan! I hope you can unwatch this somehow. Or maybe dom can remove you from the list in the sidebar.

[cdunn2001]

If it's that hard to explain, it's too complicated. I could describe scenarios which I think could lead to bugs, but I want to stay focused.

I want workspaces. That's all. The repos in my workspace are the repos to use. If deps are missing, auto-clone, but never over-write what I already have.

Write to me privately if you want to discuss dependency conflicts. Or just add to the forum discussion.

[ozra]

@bluenote10: The path as version would solve my problems.

But having a .nimble dir (in the root of my project) that is searched first would be ideal - it could easily be added to .gitignore (for example) and make it very swift to work on one dependency module by simply checking it out in the .nimble dir, or just make a symlink to it - if parallel developing on it and several projects depending on it (for parallel testing and fixing of the module for instance).

Or are you referring to the problem of conflicting transitive dependencies

This could be solved to an extent if Nimble demands semver versioning - which I see no reason why not to. In that case you know the boundaries of version numbers. Iff two different dependencies are within the same "version boundaries", they can be resolved to the latest accepted by both. With semver a specific version reference, A @ 0.3.6, should be less common, rather one would limit it to, say, only patches: A @ 0.3.*, additional features ok: A @ 0.*.*. I don't mind breaking changes: A @ *.*.* or A. Obviously typing out all three parts of the version was just an elaborate exercise in this example. It should be said that below major version 1, the rules are not hard, so '0.*' versions may introduce breaking changes even in patch versions.

[cdunn2001]

Yes, a nimble-dir in the root of my project workspace is what I want. I don't know about all the '@' stuff. I think you're making it way, way too complicated. If you need a specific version of some registered package, just git-checkout that version.

Honestly, I don't think dom96 is going to agree. I have not seen him acknowledge the importance of workspaces, and I can't spend forever arguing about this. The GoLang folks have put a lot of thought into this, and Go has exploded in popularity, not because it's a wonderful language (it's ok), but because of its wonderful ecosystem.

I think we have to fork nimble.

[bluenote10]

If you need a specific version of some registered package, just git-checkout that version.

Well, but how will you publish your project if it depends and a large number of libraries all with specific hash versions? Do you want to give your users a list of git hashes and tell them to check out all these repositories manually to a specific version? The just git-checkout that version is part of the information required to build the package. Consequentially it should be part of the build definition, right?

What do you mean by version specification makes things too complicated? In order to provide the build information required (=ensure reproducibility of a build), a dependency definition must be package + version in one way or the other? And I'm just proposing that a version specifier can be either (1) a version tag, (2) a commit hash, or (3) a local path. Other syntax/options are possible of course, if that is what makes it complicated.

(*) btw: a pretty common situation in Rust the last time I was using it, and I was always pretty pleased that all that information is encoded in the build definition allowing to build even the fanciest dependencies with a single standardized command. And by avoiding the concept of an "installed" package, I don't see how the notion of a workspace would be required in Rust.

[dom96]

@cdunn2001 says:

@dom96 says:

I have actually recently tried Cargo myself and was surprised by just how similar it is to Nimble.

Yes, nimble build is like cargo, and it's not bad. But I need to build A, B, and C, where:

• A depends on B and C
• B depends on C
• I own A, and I am modifying A locally.
• B and C are in the nim-lang nimble registry. (Or whatever it's called.)
• I am modifying B locally.

See the problem? I'm not sure Cargo handles this either. gobuild handles it easily.

I am probably repeating myself at this point, but you can already do this. After modifying B, cd into its directory and execute nimble install. Then build A, cd into its directory and execute nimble build. This may not be the most convenient, but it already works, and is also the way that Cabal works (IIRC).

---

Honestly, I don't think dom96 is going to agree. [...] I think we have to fork nimble.

You're more than welcome to implement this. I can by no means guarantee that I will merge your changes in, I am still undecided about this whole workspaces idea and want to explore as many aspects of it as I can and think about it carefully first. But let's not create the situation where there are two different package managers for Nim.

Perhaps a separate tool for this would be more appropriate?

Maybe it would be enough if you could just put the dependencies you're working on in $yourPackage/workspace manually, and create a yourPackage.nim.cfg file with --nimblePath:./workspace in it.

[cdunn2001]

You're more than welcome to implement this. I can by no means guarantee that I will merge your changes in ...

I understand, but it's hard to rebase my changes onto yours without the earlier PR that you rejected. That was the whole point of that change. Your config-reader is tightly coupled to the rest of your code. If it weren't, then I could simply rebase my changes onto yours whenever I pull an update. No big deal. But as it is, we might diverge. Honestly, I was astonished that you rejected a simple refactoring. It was to allow experimenting. If I don't use a local config file, then I'd use an environment variable, like GOPATH. There are pros and cons to each, and I'm agnostic.

the way that Cabal works

Cabal solves a different problem than I have. I'm not an expert, but here is my understanding: It wants each package to specify the versions of its dependencies. It can install multiple versions of the same package because different transitive closures can use different versions, and because Haskell applications are static binaries.

But I don't care about version numbers. I think that's too complicated. Packages need to maintain backward-compatibility, and if they can't, you might be better off cloning them. Here is some discussion of how Golang ignores this alleged problem. All the links in that answer are worth reading.

Golang does it right. Very little else does. I know that's a bold statement, but it's a big part of the reason Go has exploded. It keeps things very simple. I can criticize the Go language in many ways, but its ecosystem is very easy to work with. I know you want simplicity too, but Go is extremely simple. E.g. if you main concern is reproducible builds, look at the two-step process involving go get and godep. They are two separate tools for two separate concerns, and they both rely on the GOPATH workspace.

You're trying to make Nimble solve multiple problems, and I'm saying we need 'goget' and we need 'godep', separately. We might also need a change in the compiler so we can specify github paths in source code, rather than relying on a global package registry. But the one problem that must be solved first is that of workspaces.

Just consider this food for thought. I think I have to work on things other than Nim for a while now, so I won't fork Nimble anytime soon...

[cdunn2001]

• https://ivanmiljenovic.wordpress.com/2010/03/15/repeat-after-me-cabal-is-not-a-package-manager/
• https://wiki.haskell.org/How_to_write_a_Haskell_program
• https://www.haskell.org/cabal/
• https://wiki.haskell.org/Cabal-Install

I see. Yes, Nimble is very similar to Cabal. But have you heard the phrase "Cabal hell"? And have you ever used "Cabal sandboxes"?

• http://coldwa.st/e/blog/2013-08-20-Cabal-sandbox.html

[bluenote10]

But I don't care about version numbers. I think that's too complicated.

Dependencies are complicated. Omitting the version information simply does not make it any easier.

Packages need to maintain backward-compatibility, and if they can't, you might be better off cloning them.

Forcing backwards compatibility in a build tool is in my opinion detrimental for a language as a whole. Good design requires breaking changes. There are already so many examples of a bad design which are just a result of maintaining backwards compatibility. And practical issue: Packages need to maintain backward-compatibility is pretty vague. In order to make such a policy reliable it would be necessary to check for backwards-compatibility formally, which is a tough problem.

Golang does it right. Very little else does.

I've been working for a company where we managed our internal (Python) libraries by exactly this approach of taking local copies in our python include path. In my opinion it was a huge mess. We constantly had problems with reproducing a build of a colleague, and it was a very common to ask "which version of lib X do I need". That is why I think omitting such a vital information is really not a good idea.

In the worst case it means that in order to ensure 100% build repeatability, I will have to add a local copy of all libraries to my project repository. And this brings us back to a version tag/hash, which is a unique identifier of a copy. So, it is essentially the same as adding a copy, just much more simple and concise. The only real advantage of a copy is that there is the unlikely chance that a remote can go offline.

Update: I just spend a few hours studying the links in this article, and they pretty much sum up all the concerns I have regarding omitting version information. Go itself recommends to 'vendor' packages, i.e., copy all dependencies to the project itself. Like I said above, this is essentially the same as specifying a version -- so not that much different at all. I just think that typing a few characters like "B @ 0.1.2" is in fact the simpler solution. I think all the articles make it actually pretty clear that omitting version information does more harm than good:

source1

The Go Authors recognize that this is an issue, and suggest that if you need this level of control over your dependencies you should consider using alternative tools, suggesting goven as a possible solution.

Over the past two years no fewer than 19 other tools have been announced, so there is clearly a need and a desire to solve the problem. That said, none of them have achieved significant mind share, let alone challenged go get for the title of default.

source2

Others have simply forked all their dependencies to control when upgrades occur. Some have built tools to checkout dependencies with SHA revision identifiers. Still others utilize DVCS features like submodules or subtree merging. Experiment, find what works for you.

If there is such a plethora of additional tools to get versioning back I think it is pretty obvious that it is not a good idea to omit it in the first place. For me personally, it was a salvation to move from a version-free dependency management to systems like Maven/Ant/SBT/Cargo where versioning is explicit.

Regarding "cabal-hell" (I assume that "cabal-hell" refers to the same as jar-hell or dependency-hell in general, i.e., conflicting transitive dependencies): Note that this is not the result of explicit versioning: The problem also exists with the simple copy-your-dependencies approach. It is just a difference of how to resolve them. And again I think it much easier with to handle them if I (and the build tool) actually know the version dependency exactly instead of having to guess it.

[ozra]

But I don't care about version numbers. I think that's too complicated. Packages need to maintain backward-compatibility

Requiring backwards compatibility means no advancments. This is what semver versioning is for, and why I think it should be the standard for the nimble eco system (#130), that I keep nagging about. Then you simply state the major version of a module your project depend on, and backwards compat is ensured, while still allowing updates of features and patches. It's ingeniously simple.

[ozra]

To sum it up - this, imo, would be able to solve everything(TM):

1. Semantic versioning is official policy for nibmle eco system.
2. Version matching in nimble conf can be greatly simplified, while being more specific and safe. Ranges etc. can be kept for the situations where a module has deviated from semver, or the developer has fouled up and broken the rules by mistake - for workaround rescuing.
3. Nimble should keep multiple versions of modules in its cache. A specific purge command can be used to clean up older ones, if one feel the need. (Nimble could keep a cache of project paths that has accessed it, and if they're still in system, it can check them for deps as to not purge packages in use - this is just a bonus bandwidth saving feature, not necessary). If packages are purged that are used, they will be installed again when building those projects as usual, so no problem.
4. Nimble searches for dependency packages first in my-proj-dir/.nimble/ - if not available, uses the regular ~/.nimble, if not there, downloads like usual.

Very simple. Very powerful.

Thoughts?

[yglukhov]

:+1: And a vote for some sort of lockfile. E.g. like ruby gemfile.lock does.

[dom96]

I see. Yes, Nimble is very similar to Cabal. But have you heard the phrase "Cabal hell"? And have you ever used "Cabal sandboxes"?

Yes, I am familiar with Cabal hell. I think that Nimble is just different enough to not suffer from Cabal's hellish problems.

For example, in the scenario described here: http://stackoverflow.com/a/25870174/492186

[...]

Where both B and C depend on A. However, if they were installed at different times, they may depend on different versions of A. For example, A version 1 export type T = Int, but in version 2 it exports type T = Bool.

Only when you try to build D do you expose the problem that B and C were build against different >versions of A, and you can't compare T version 1 against T version 2.

The key difference here is that Cabal will build each installed package (including libraries), whereas Nimble will not.

As long as B and C do not depend on a specific version of A then Nimble will use the latest version of A it can find when building D.

---

Golang does it right. Very little else does.

I'm not so sure. I have read plenty of complaints about Go's lack of sophistication with package management. From @bluenote10's quote, the 19 other tools seems excessive already and shows that people are not happy.

---

@ozra If I understand you correctly then number 3 is already done. Number 1 & 2 I can do. Number 4 I'm still not convinced of.

I'm starting to consider allowing file paths in dependency lists:

[Deps]
Requires: "file:///home/dom/myprojects/jester"

[dom96]

@yglukhov https://github.com/nim-lang/nimble/issues/127

[cdunn2001]

Everyone seems to be missing my point: Workspaces are more important than versioning.

There will never be 100% agreement on how to do versioning, so decouple it. Go uses a 2-step process. That is a very, very good idea.

@bluenote10, "Cabal hell" refers to diamond dependencies, which is a result of explicit versioning. At my last company (a major IT firm), the original designers of the build system added versioning everywhere. Later, that was deprecated because it made updates impractical. Versioning is an enticing idea, but ultimately what you really need are tests. You should be willing to substitute in any version as long as your tests pass. Once they pass, you need to lock down the contour of commits. Forking is one way to do that. ("Go itself recommends to 'vendor' packages, i.e., copy all dependencies to the project itself." Yes. That is forking, not explicit versioning.) Another way is to store the contour in a separate system, specific to your project. (I have an example at https://github.com/pb-cdunn/FALCON-modules, and I'm happy to explain what I've done there.) A lockfile is another way. But you need to distinguish "versioning" (where a package identifies its own version) from "hash-tagging" (where the version is implicit, equivalent to a copy). Those are different things.

@dom96 says:

As long as B and C do not depend on a specific version of A then Nimble will use the latest version of A it can find when building D.

Good! But I'm saying it should generally work that way. As soon as people name specific versions, they will prevent others from updating diamon-dependency A, which is bad. They is why we need to let people use their own systems for versioning. We need to decouple package-dependencies from version-dependencies.

I'm starting to consider allowing file paths in dependency lists.

:-) That seems like a very good idea to me.

But I need workspaces first. And workspaces are trivial.

[ozra]

@cdunn2001 - you seem to miss the point that semver allows specifying a version range that will cause no problems, while still allowing all upgrades that stay within the limits of the semantics. This is not a specific version, but a specific range of "level" of allowed changes. (Only non-breaking changes for instance). This complements tests. And you can simply ignore using the versioning if you want. (Thereby allowing breaking change updates of modules, and if you're tests are exhaustive and you trust them, the breaking changes might be outside of your use case, etc.)

I find this very important, and it's just a matter of declaring this officially, it is like coding style conventions, but for versioning, it's a matter of trust on the module contributors to follow the official recommendation, so it's no implementation cost. Your need for workspaces first might not be the same for everyone else. But I must admit I'm not sure if I understand your workspace notion completely: I don't see a case I couldn't solve with the local ".nimble/" dir, or if must, specifying it with paths in the proj nimble conf (more unnecessary work imo).

@cdunn2001: What is it that you can't solve with those simple constructs? To narrow your use case down a little, so we all can converge a bit on the matter.

There's of course one additional, helpful feature for the local packages:

5. With a command line option, nimble could help in installing a dependency in the local dir, nimble ... --local ..

@dom96: Regarding 4 - the pro of the local dir (apart from 5 above), is that one does not have to change the conf. If for a while I need to tightly develop a dependency with my using app/module, I can simply git clone it, or symlink to it (if machine local), or use 5 above to get it into my local '.nimble/' dir in the project. It will now automatically be used when building. When I'm satisfied, and don't need to co-develop the modules tightly, I can simply remove it when commited, done and published. Now it goes back to automatically using the 'global' nimble cache.

I know I'm repeating some things said earlier by different parts, just trying to converge..

[dom96]

@ozra What about my idea to support filesystem paths in dependency lists? It's more flexible IMO.

---

:-) That seems like a very good idea to me.

@cdunn2001 hrm, so you still want workspaces. Wouldn't filesystem path support in dependency lists be enough?

[ozra]

@dom96 - it's just the simple fact that the placing in .nimble/ makes the intention clear. Having to edit/un-edit it in the file is an extra step. That said, I think the paths idea is great as a complement! Personally, I'd prefer symlinking from the .nimble/ dir to a certain path, rather than enter it in the conf.

But I would survive with the path thing.

Is there a specific reason you're uncertain of the idea of first looking for, and in, a "{cwd}/.nimble/" dir before searching the global?

[dom96]

I clone everything into ~/git, so it makes more sense for me to add file://~/git/jester to my .nimble file than to copy the git repository to (or symlink it in) $CWD/.nimble.

I think that this would be a more flexible solution, and just as convenient. Unless I am mistaken once again about the use case that you guys have for workspaces.

[ozra]

@dom96 I have quite a bit longer paths, and structure things different from that, so there will be more typing, but that's not so much of an issue. I would find it very helpful simply put. So what would be the downside of implementing it in nimble? (I could possibly make time to do it, if it's the effort) (Opting out is as simple as not having a local .nimble-dir)

[dom96]

Are you talking about implementing workspaces or file paths in dependency lists?

[cdunn2001]

@dom96 wrote:

@cdunn2001 hrm, so you still want workspaces. Wouldn't filesystem path support in dependency lists be enough?

It's useful in its own right, but it doesn't address my use-case. I want two overlapping contours of different package trees in my filesystem simultaneously. (Do you understand what I mean by "contour"? It's my word for a specific set of SHA1s for a specific set of repositories.) If these trees are kuldged into the same directory (~/.nimble), then we have the diamond-dependency problem, and it's unresolvable because I am actually using different SHA1s for this parallel development.

@ozra wrote:

you seem to miss the point that semver allows specifying a version range that will cause no problems

I know what you're saying. I thought the same thing when I got to Amazon. But Cary Hall, a Principal at Amazon, set me straight. It is not easy to see the problem caused by versioning, and version ranges have even more subtle problems. For one thing, it doesn't scale. (Your system must handle many thousands of packages.) For another, people will tend to say "> 3.4", but what about 4.0? That's bigger, but a major-version-bump means all bets are off. It's better to use software from people who try very, very hard to maintain backward-compatibility (as we do for jsoncpp), and to test as much as you can. Aside from major version numbers, semantic versioning should be a hint for the human integration-manager, but not a contract for the automated package-manager.

What is it that you can't solve with those simple constructs? To narrow your use case down a little, so we all can converge a bit on the matter.

nimble --nimbledir would suffice, but it's unreliable. I would have to set an alias, and then I have to worry about sub-shells not using my alias. It needs to be either an environment variable or something in the filesystem itself.

Besides, part of the problem I'm explaining is the difficulty of knowing what goes with what when everything is shoved into a single nimble-dir. In a workspace, there is only one version of any piece of software. When you examine the directory, you know exactly what you've got.

[ozra]

@cdunn2001 - so I still don't see how fully local copies of the deps (in "$CWD/.nimble/") would not suffice? Then it is essentially your workspace? What am I missing? Regarding the versioning, for your example, if 3.4 is minimum, a roof should be added too: "< 4". In reality though, if 3.4 is already out, and you enter "mod @ 3", you will get 3.4 or later, but not past major 3. Since nimble don't fetch older versions unless specifically asked for. So the '> 3.4' is not semver idiomatic. Best practises for utilizing semver should be clearly and simply put in the manual. It doesn't take more than a few sentences to explain. And you still don't have to use it.

@dom96 - I will repose my question, since I didn't get an answer, if you don't mind: What would be the downside of implementing a "cwd nimble-dir check first" in Nimble?

[cdunn2001]

in "$CWD/.nimble/"

Yes, that would be a workspace. That's fine, as long as I can continue to commit and push in repos in either $CWD or in $CWD/.nimble/packages, or whatever.