Package / gem system?

[farleyknight]

Haven't worked on anything crystal in a while! Just wondering if you guys have any plans on doing a crystal package / gem system any time soon?

[bararchy]

I would really like if the dir will be an external dir, like in gem, where for each "require" crystal can link the needed deps from the global directory, this way I dont need download all the deps each time for each project, and It's easier to keep all the deps up to date.

[trans]

On Wed, May 20, 2015 at 7:50 AM, Trans transfire@gmail.com wrote:

Why do I need to trust you? I know all about Toml. It's not at all perfect. Hipster is what it is -- all the new kids are doing it. Blah. It's junk.

Okay. That's a bit harsh. Toml is okay. I just think it is overrated, and espoused for the wrong reasons, i.e. new and simple --- new and simple does not necessarily equate to better.

[trans]

If I were able to get this to compile to Crystal would it be helpful to you?

https://github.com/rubyworks/versus

Note it has a version resolver.

[bgdncz]

@asterite Do you mean like if package a depends on b, and b on a, then throw an error? Yes, Cargo does that. Also, here's an example of Servo's (in case you didn't know, Servo is like Gecko written in Rust) Cargo.toml. It's very readable by humans. One thing I wouldn't like is Github being compulsory, because there's Bitbucket, GitLab etc.

[trans]

On Wed, May 20, 2015 at 8:24 AM, Trans transfire@gmail.com wrote:

In the other case, perhaps Crystal can handle multiple versions of the same package? If my package uses version 4 of some library, but another dependency of mine use the same package but version 3, is there any reason that they can't just use their own respective versions and compile without conflict? (Of course it's a good idea for the tool to tell me about the version difference, at least.)

So can anyone tell me if this is possible (or can be made possible) or not?

[asterite]

@trans Dependencies aren't compiled to separate binaries that are later linked. When you compile your program, all of the dependencies are load up and from them and the main program an executable is created. If version 3 defines a method foo that returns 1 and version 4 defined a method foo that returns 2, one of them will win: they can't coexist.

[trans]

@asterite That's helpful. Thanks.

Theoretically, couldn't the two versions be "namespaced" so as not to be confused? So code using version 4 would call v4::foo and code using version 3 would call v3::foo. I once developed such a system for Ruby. With the exception of global variables and core extensions it worked. In short, instead of require 'foo', one used a more javascript styled foo = require('foo', '>4'), so a namespace automatically wrapped the library, which was then reachable via foo variable.

[bcardiff]

@trans In theory for a simple code it can be done, but as @asterite said, right now the compiler loads all the required sources and compiles it as a single unit. Class/method lookups are run over the whole code.

If the lookups are changed in order to use some sort of scoped lookup based on project dependencies each shard would be able to use just the code it's advertised depends on. But then we need to see how to deal with blocks calls, inversion of control, delegation. Monkey patch will be left out (despite not being the something to encourage). Top level funs will definitely clash. I'm not sure if for something more than simple code that will end up working.

I don't see something like that coming. Yet, it seems an interesting problem.

[clord]

I know most languages have their own package management solution and so the temptation is there to invent yet another one from whole cloth, but I'd like to propose adopting a slightly modified version of nix as a way to distribute code and dependencies. And if not that, at least base a new crystal-specific manager on it.

Why?

Most language package managers end up needing other things too. I'm thinking of having a local imagemagick install, or libxml, etc. and everyone always ends up rolling their own solutions to having these installed, usually half-assed and buggy for a decade or more.

Using a real package manager as the foundation means you can easily port these sorts of dependencies without having to reinvent the wheel.

The reason I suggest nix specifically is that it is really well thought out, and solves a lot of problems that a source-code package manager will also have to solve. Aforementioned binary dependencies is one. Also, it's really good at versioning. One project might depend on v3.1 of some library, whereas one of it's dependencies might depend on 2.0. Now the ideal way to deal with this is to allow both to have their own dependent versions used, without conflict. Nix makes that sort of stuff trivial.

Because of the way nix works, it becomes really easy to 'switch' projects. With ruby, RVM and other tools try to solve this problem (badly, imo). nix inspired tooling would allow the whole stack to be specified declaratively at the project level.

Also, because of the way nix lays out packages, it makes it really easy to build stuff. It puts literally everything into one tree (kind of like a mini chroot) and then builds it. So one need only set the appropriate LOAD_PATH environment variable and run the normal compiler without much cooperation from the package manager (the compiler could also be a dependency of the project btw)

So my proposal then is to:

1. Either fork be inspired by nix (at the very least the idea of immutable self-contained packages)
2. Create a crystal-specific front-end to the nix store
3. use the resulting tool to manage source of dependencies
4. use it to publish end-user binaries and compiled libraries as well, ideally with an easy way to port from elsewhere.

This is really a big opportunity to leap-frog gems and RVM. At least studying the design of nix will certainly produce a much stronger package manager that will solve some of the worst problems with the other systems out there, and save some work in the process.

[rishavs]

Hi

Let me provide the perspective of a beginner and a not too serious coder.

coming from node and python, i personally prefer a centralized approach. I personally believe that npm was among the most important factors in node getting so big.

Also, if crystal does implements a package manager, i'd really want it to be part of crystal itself, rather than a separate project/product.

I'd love to use commands like `crystal install fast-json@1.2'

For me all the complex and dirty bits should be under the hood. Global or local namespace is sufficient for me. Heck, even a global default is great. the less i have to fiddle with package mangement, it would be better for me.

Finally, while i love YAML, i really dont think it should be used in any public project with lots of contributors. It is very contentious and a lot of people find its lack of adherence to its own specs to be off putting. JSON is what I'll recommend.

[vyp]

@rishavs You may be interested in https://github.com/manastech/crystal/wiki/Roadmap#package-manager if you haven't already seen it.

@clord I agree, but I don't see the reason for using a fork of nix, why not just nix itself? And besides, unless I'm misunderstanding something, crystal install is not supposed to be a package manager, but just more like a build helper, ala rust's cargo install command, so really you should be able to use whatever you want.

[clord]

@vyp the problems a 'build helper' solves and those that a 'package manager' solve overlap significantly, especially if crystal wants to have something resembling ruby gems.

[jhass]

Shall we close this in favour of #1357?

[asterite]

@jhass Good idea!