Project-local package dirs with precedence

[ozra]

Having worked a couple of years with iojs for the money, I really enjoyed the modularity of projects vs the C++ world. I've already had several problems with Nim, while hacking the compiler etc. that ~./.nimble/.. pkgs and global nim stdlibs are preferred over the ones from the dir I'm working in.

This is mostly a nim mod-lookup issue, and partly nimble.

Basically a project-local mod deps dir - like "node_modules/" in iojs/nodejs - it should have precedance. Also, those package dirs, should easily be pure github checkouts, for deving on deps, hand in hand with a project.

This way version matched packages can be stored per project - they could still be cached in home-dir/nimble. I'd prefer 'proj-local' pkgs as default. But an option would be ok, "nimble install local..."

https://github.com/nim-lang/nimble/issues/114#issuecomment-98890630 https://github.com/Araq/Nim/issues/2819#issuecomment-106007043 https://github.com/Araq/Nim/issues/2819#issuecomment-106015725

As for the module lookup, I guess that's in nim's domain.

[ozra]

Right, that was the idea: for package/app A, requiring B and C: if package B is in "A/.nimble/B/" - (it may simply be a git repo. local, symlinked, what not) - "nimble build" will see it and use it in build, but C is not there, so that is used from "~/.nimble/". If all packages used are in local nimb dir, they are all 'vendored' so to speak (if I got the term right). You maintain full control of pulling any version you want, local modifications, etc.

So, yeah, I still feel the possible local .nimble/ dir is more or less a must have. I see no downsides to implementing it at all - for those who do not use it.

[cdunn2001]

Here is a nice discussion which mentions Python, Go, and Ruby, all of which have gone through several iterations of package-management. It's worth reading:

• https://caremad.io/2013/07/setup-vs-requirement/

[shaunc]

fwiw ... as a nim newbie, having developed in at least half a dozen language ecosystems over the years, npm is by far my favorite package manager: workspaced is the way to go, falling back on "local" (user) and global locations if not installed locally. Semver is good but should be optional. For one thing you are at the mercy of someone else doing it right.

As a consultant, I often have libraries lie fallow for years, and come back to them when I have a related project. Having everything local in the workspace is brilliant: it will still work no matter what has happened outside (well, modulo changes in the language itself or the os, etc...).

Non-workspace-based package management should be seen as an optimization.

EDIT: Another way to see this: when you decide to use a package in a project, you want to decide just that: "should I use (the current version) of package X in project Y?". Semver lets you specify parameters on what future version you are prepared to accept; (accepting future versions means you trust the package developer). Making your choice implicitly affect your other projects is a memory leak.

[dom96]

Some discussion about vendoring in Go in this post: https://hackernoon.com/gos-alias-proposal-and-all-my-concerns-of-google-controlling-go-a39f6c6046aa#.shw6iaqgc

Looks like the implication is that it was a bad idea.

[cdunn2001]

Good blogpost. It mentions an idea for Go interfaces which is simply not a problem in Nim.

Go-vendoring is meant to ameliorate the problem of storing a copy of an external library within your project, but I don't think it's a very helpful solution. The vendored package is still difficult to update with revision-control. I would prefer relative imports, plus a solution to the interface problem in the link above.

[matt2000]

I heaven't read the whole thread, but it makes me sad that this seems to have not progress in almost a year.

TLDR; I fully Agree with SeanC above.

Please, please, please don't make the mistake of python & PHP & Java & R and end up having to redo packaging several ways over the years. They all started with Global level dependencies and came to realize they were wrong or forked communities. Get it right from the start like Node/npm. Global level dependencies are a constant headache in a sufficiently complex ecosystem, doubly so in a microservices environment. Project-level is the only reasonable thing to do in a world where disk space is so cheap as to be a non-issue.

[dom96]

@matt2000 I still don't think I get the issues that we will run into if we use a global workspace. Can you shed some light on this?

[genotrance]

I'm tracking the proposal for this feature here. Appreciate all feedback to help finalize the design.

[genotrance]

I have a working PR #834 for this that implements the proposal from November last year. Looking for feedback.

Effectively, mkdir nimbledeps is all that's needed for nimble to know to use local deps mode. So:

git clone repo
cd repo
mkdir nimbledeps

After that, all nimble commands will use the local directory for dependencies and avoid all use of any other nimble locations.

Note that Nim is not aware of nimbledeps so this is restricted to nimble at this time. Supporting local deps mode with Nim will be done by generating a nim.cfg in the future. That design is in the works and will be developed once this issue is resolved.