Package manager

[asterite]

This is the main issue tracking this feature from the Roadmap.

Crystal will provide a built-in package manager. We really want this to be the only package manager so it's easier to build a collaborative community.

We want a truly decentralized package manager. These ideas could make it work:

• No global directory where all deps are installed: it's local to each project (lib/libs directory)
• Each repo has a special branch (maybe _releases) with metadata for dependencies for each version. These are cummulative, so version 0.2 contains metadata for 0.1 and 0.2.
• Crystal provides a command to release a new version, pushing to that special branch, and creating a tag
• Use Semantic Versioning
• Automatically download recursive dependencies
• Resolve conflicts
• Remove Projecfile, use YAML, both for project.yml and for the metadata file
• Include name and version in project.yml
• The name above is used for the directory in which it is installed, and used by require "...", so for example "webmock.cr" will be installed in "lib/webmock", and that directory will contain the direct checkout of the project (so it has the src directory in it)
• The require logic changes to that if you do require "foo", we check if there's foo.cr in CRYSTAL_PATH, or foo/foo.cr, or foo/src/foo.cr (this last one is the one that will be used for dependencies installed via the package manager). With this logic, the current CRYSTAL_PATH value doesn't need to change
• Probably rename "libs" to "lib"

With the above, when you do crystal deps install, all first-level dependencies are gathered. From there we go to each depednency's repository and check out the special _releases branch to get all metadata for all versions of that first-level dependency. We apply this recursively until we get all the metadata for all involved libraries. The previous process should be fast, because only that metadata branch must be checked out, and only once for each library (and we can parallelize the requests). Then we can solve conflicts and install what's needed.

For discoverability, we can list github/bitbucket repositories that have crystal code and that also have that special _releases branch, which in turn contains all the information for every version of the library.

For all of this, the easiest thing would be to build on top of @ysbaddaden's shards, which already has the desired YAML format, probably has some logic for semver, etc.

[ysbaddaden]

As discussed with @asterite Shards shall supersede the current crystal deps command.

Development on the package manager will continue there for the time being. We'll eventually figure out if it shall be merged into the Crystal source code or be kept separated (as long as it's tightly integrated/embedded with Crystal's distributions).

The main feature still missing is the lock file (https://github.com/ysbaddaden/shards/issues/12) then to integrate it as crystal deps.

[PragTob]

Looking forwards to this very much :+1: :+1: :rocket: :rocket: :dancer:

Comments/questions:

• will semantic versioning be enforced/is that on the roadmap? elm claims that they do it, which I finrd to be fascinating
• it seems common to me that the lib directory contains code you wrote yourself but that is applicable to other projects (that's how I use it in my ruby projects and I know other do too), hence personally I'd prefer another name for the directory. Unless in crystal we'll have another convention for where to put that code (somewhere under src) ?
• Steve Klabnik once told me that he considers toml to be a much better format for that sort of task (compared to yml), which is why it is used for cargo (Rust) - I don't have any details, though and I like yml, just throwing it in here
• Do I understand it right that this will all be based on checking out git repos or is that a first step and in the distant future there might be something else (like gems/packages pushed to some infrastructure) - I understand that own infrastructure would be sort of central but so is relying on github/bitbucket (sort of) in my understanding

also thanks @ysbaddaden for the work so far! :+1:

[ysbaddaden]

• semver: not enforced, but supported and expected (mostly by developers);
• I like lib for dependencies better than libs and it's 3 letters like src so it's nice, but no strong opinion here (it could be vendor);
• YAML is distributed in the stdlib and did the job. I have no strong opinion against TOML and it may prove better. We shall just try it. Maybe it's better than YAML which needs quotes around some version requirements (eg: ~> if I remember correctly).
• There won't be any registry that would distribute packages. Source code shall be downloaded from repositories (either GIT or whatever we want to support). That being said, but there may be a nice registry site to index and search libraries (ala http://crystalshards.herokuapp.com/) to index and search libraries, available versions, etc.

[0x1eef]

probably you won't need many of the features that YAML provides, and I guess you have to think about security when loading YAML because it can deserialize complex types, where as toml is more simple and just supports arrays, hash, string, & number types.

[jhass]

Is there a defined serialization of TOML, given abstract input data? Last time I checked I couldn't find it, but I think it's important to have in order to support tooling that automatically edits the file to bump the version number, add/remove dependencies etc.

Stdlib YAML doesn't have serialization either yet, but it should be comparatively easy to add.

[asterite]

In TOML you have to put quotes around every string, so having to put quotes in YAML from time to time is not an issue.

There's also the thing that TOML isn't stable. We have a parser for it, it's just not in the standard library because it's not standard enough.

We can also compare a shard.yml with a Cargo.toml file. To me, YAML looks much cleaner without all those quotes. Also, those [[...]] look strange. Well, once you learn TOML you understand it, but it's something else that needs to be learned (I'm sure many more people know YAML).

There's also the thing that @jhass mentions, there's many ways to represent a TOML and I can't figure a way to do a toml_mapping because you have to parse things to an intermediate structure before converting that to a model. In JSON that works pretty well, and in YAML it might work well too.

[0x1eef]

@jhass what do you mean by "Stdlib YAML doesn't have serialization either yet" ? When I say serialization I mean turning a Crystal string into yaml, and then back again(deserialize). YAML can just do much more than only a string, and sometimes it's a security issue (like being able to create Ruby symbols from YAML).

[jhass]

I mean that the serialization API is not available yet, but the standard does define a canonical serialization and the backend libyaml does have an API for it.

With TOML, if there's a canonical serialization, we also have to write the complete serializing code. But I couldn't even find a definition for such.

[refi64]

Ugh...please not TOML. It's the worst markup language. Ever.

The issue with YAML is that, if you want auto-tooling, you're going to store a heck of a lot in the tree structure (the indentation the user used, the whitespace surrounding the colon, etc.), as well as use some sort of ordered hash to avoid literally wrecking the order of the file, if you want the tooling to not change the whole file around every time one "bumps the version number".

What about a ridiculously simple INI-ish thing, like:

[meta]
name = shards
version = 0.1.0

[sources]
https://shards.crystal-lang.org/
https://shards.example.com/

[dependencies]
pg >= 1.2.3
memcached = *
openssl = github+datanoise/openssl.cr#master

[dependencies-dev] ## Like Pip's requirements-dev.txt; you don't minitest installed to actually run shards
minitest = git+https://github.com/ysbaddaden/minitest.cr.git ~> 1.0.0

[trans]

I would like to offer a modified approach that would provide a few additional benefits. My main concern is with the _releases branch. I do not think it is a good solution to the problem b/c the mechanism of a branch doesn't really correspond to the need. Another issue is the usability of the project.yml file. Either the user will have to follow a very strict schema or the parser will have to have plenty of heuristics to make the format flexible. The former isn't very user friendly, and the later would generally require use of the same parser library whenever the file needs to be read to ensure proper info. But other tools might want to use that info too, perhaps some not even written in Crystal.

To overcome these issues, I'd like to propose an idea I came up with some time ago that I use for my Ruby projects. I have a humanized form of metadata, in an easily edited YAML file, though it can be a Ruby script or other format, like TOML, or combination of files actually, it doesn't really matter. Than there is a canonical metadata file. A tool takes the editable file(s) and generates the canonical file. The canonical format follows a strict schema with no exceptions. This makes it readable by other programs without any heuristic processing. To solve the _releases issue, this tool can also look back at past tagged versions and pull the relevant dependency info into the canonical file. This approach gives maximum flexibility to the developer while still providing a single strict metadata file for use by the package manager and other tools.

[ysbaddaden]

The _releases branch hasn't been implemented. This is open for discussion, and as I understand it, it would mostly allow to speed things up, yet hide the gory details behind a few tools —that would work as you described.

For now the different shard.yml come from version tags.

The format of the YAML file will indeed be specified and I'm thinking about a validate command (Travis does that for instance).

[vyp]

@trans:

My main concern is with the _releases branch. I do not think it is a good solution to the problem b/c the mechanism of a branch doesn't really correspond to the need.

I don't understand what you mean, could you elaborate?

@asterite I have a question:

With the above, when you do crystal deps install, all first-level dependencies are gathered.

Where are these dependencies 'installed', in ./.crystal?

[ysbaddaden]

The src folder of a foo dependency is installed as libs/foo.

The .crystal is where we cache cloned repositories.

[kumpelblase2]

If I may throw in my 5ct:

• I'd certainly be in favor of something along the lines of yaml or json for the project file. I find it way easier to read than anything ini-ish/toml (because it most of the times looks weird with arrays or sub-properties).
• I also do like the decentralized approach, but having a single registry to look up what's available is still something I'd want to have. Because not everything will be stored on github and we end up having half the software (I'm exaggerating here) on self hosted repositories where you won't easily find them. Probably a hard problem to takle for a registry as well. But I see it on the Shards roadmap, so I'll just hope something like that will be available.
• Liking the Semver versioning approach, would be even more happy if it got enforced, but I don't have a good argument to do that.
• On the name of the source directory for those dependencies I have to agree with the concern of @PragTob , as I think several people would use a lib directory for something else. So I would prefer something like a deps (or dep for that matter) directory or maybe even something completely different.

Otherwise it looks solid and I could certainly work with it, so I'm happily waiting for it. :+1:

[jhass]

If we go for something else than lib I'd vote for vendor, but I think it's fine if we establish lib for third party and src for own stuff as a community convention.

[vyp]

Thank you @ysbaddaden.

I vote for enforcing semver but only because it just ensures to keep all the stuff that handles version numbers simple (or simpler).

About format, I vote yaml because it's 'prettier' to read, and I guess crystal users in general also like nice looking (i.e. readable) syntax, because that's one of the selling points of this language. But I'm really not too opposed to the other options either.

About third party directory name, I don't really see an issue with the community just establishing lib as the name. But if another one has to be chosen, I'd go with @ysbaddaden's vendor suggestion > deps/dep.

@asterite About discoverability... In addition to automatically listing github/bitbucket repos with crystal code (I assume you mean automatically), could we also have it that random people can submit their project to this list (as a 'hardcoded' project)?

Obviously, this would only be for those who do not host on github/bitbucket. And the submission should have their git repo url somewhere, so that the presense of a _releases branch can be detected. There shouldn't be any conflicts even if two projects have the same name because the url should always be different. (As in, conflicts in the list itself. I think the problem of actually using conflicting projects simultaneously still exists, but that's another issue altogether.)

Oh and also, what about gitlab.com? (if they consider crystal as a language)

[jwaldrip]

Question:

The releases branch sounds like a pain to handle. Why not just rely on git tags, and ensure each released version has the correct dependencies specified with each tag?

[asterite]

@jwaldrip The releases branch will be managed by a tool, so it won't be a pain. And you need to have fast access to all dependencies of every version released. It's not enough to have access to the dependencies of a particular tag, you need them all to resolve conflicts.

[ibash]

Hi there, just started to explore crystal recently and wanted to lend a couple of thoughts:

1. While I like the idea of decentralized dependencies, I think a centralized repo is going to emerge whether you want it to or not, at the very least to provide an easy way to discover packages and have reliable installs. The question is really whether you want it to emerge organically or here.
2. I really do not like the idea of the _releases branch. The reason why is that it makes versioning inherently tied to git. I would much prefer not having git be an implicit requirement if it can be avoided. In this instance it could be avoided by by having a _releases folder within the same directory.

[vyp]

@ibash

1. There is going to be a list to help with this as said in the OP.

2. A _releases folder (or better a .releases folder) will not work because it means crystal will need to download the entire project for each dependency just to do dependency resolution. Understandably, this process should not have to do that. Because there's no central registry holding all metadata*, the next best thing is to use a dedicated git branch per project for this information.

   And besides, @ysbaddaden, doesn't shards currently only support git anyway, meaning all crystal projects have to use git if they want to be compatible with shards? I think enforcing using git is an easy and simple solution.

*Of course, there will be a list as said in point 1 (probably something like https://crystalshards.herokuapp.com/), but that's obviously not a reliable source of metadata information because crystal projects don't have to be on this list.

[oprypin]

I am very very strongly against _releases. Let me just leave this here.

/tmp/blaxpirit $ git ls-remote --tags https://github.com/blaxpirit/crsfml
701047a301c09740e1409b0f557212c27373d3c2        refs/tags/v2.2.0
f9253349b25c1f0e01f5839b95c470f334e85d7c        refs/tags/v2.2.1
2ff4bcb2adcbca95ee1b4e4ddd11f7c4f7c03b56        refs/tags/v2.3.0
9b7534de0af941b73da06f65a3a57c1afe65c4be        refs/tags/v2.3.1

/tmp/blaxpirit $ git clone --branch v2.2.1 --depth 1 https://github.com/blaxpirit/crsfml
Cloning into 'crsfml'...
remote: Counting objects: 52, done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 52 (delta 2), reused 24 (delta 1), pack-reused 0
Unpacking objects: 100% (52/52), done.
Checking connectivity... done.
Note: checking out 'f9253349b25c1f0e01f5839b95c470f334e85d7c'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

/tmp/blaxpirit $ cd crsfml

/tmp/blaxpirit/crsfml (v2.2.1) $

1. To list all tags a clone isn't needed at all.
2. To see one specific version only one commit needs to be cloned.

Not to mention GitHub API which turns (1) into a JSON download and (2) into an archive download (URL provided in JSON). GitHub also allows you to download a single file like so: https://raw.githubusercontent.com/BlaXpirit/crsfml/v2.2.1/README.md (replace README.md with project.yml or whatever). I'm sure GitHub is worth specialcasing because of this. With the added benefit that Git doesn't need to be installed!

[vyp]

@BlaXpirit

1. Not only the version numbers, but a dependency needs to be checked if it has any dependencies of its own. So I don't think just listing the tags will work?
2. Even with depth=1, it still downloads all the rest of the files of the project which is unnecessary. Using a _releases branch will be more faster the more dependencies there are, compared to using just cloning a release (even with depth=1).

(@asterite I'm just guessing here though, I may be wrong?)

But why are you very strongly against a _releases branch per se?

[oprypin]

@vyp

People provided multiple reasons to not like the _releases branch. I don't have much to add. And I haven't seen a single compelling reason why it's necessary.

Do dependencies really need to be checked beforehand? I mean, if you're gonna be downloading the whole library anyway, why bother with having a way to get just the release-info files?

If you're concerned about speed. The vast majority of dependencies will be on GitHub. If it's specialcased like I mentioned then this is not a concern anymore.

The size of the _releases branch can also get big, by the way.

[oprypin]

I just realized about a big problem with the _releases branch: it limits you to working with tags only. Can't work with a specific commit or master!

Some people can't even be bothered making tags, let alone using this whole branch machinery.

And how would this branching be implemented? I mean, there are a lot of shady things about it, like making an orphan branch. And some tool would have to switch to a branch, make a commit and switch back? What if I have unstaged changes?

---

Hmm, it took me a long time to understand what @asterite said.

It's not enough to have access to the dependencies of a particular tag, you need them all to resolve conflicts.

This is indeed one compelling reason. But no, you don't need information about literally all versions, just a couple of them that appear in the dependency tree.

[vyp]

People provided multiple reasons to not like the _releases branch. I don't have much to add.

@trans has only said:

I do not think it is a good solution to the problem b/c the mechanism of a branch doesn't really correspond to the need.

But I don't understand why that really has to be an issue?

@ibash suggested using a directory instead of a branch, which will not work.

Do dependencies really need to be checked beforehand? I mean, if you're gonna be downloading the whole library anyway, why bother with having a way to get just the release-info files?

Again, I'm not sure about all this, but I presume it's so that the correct version (and only that version, instead of the full git history) can be downloaded, which is a good enough reason for me. And I don't really see any downside to it other than being forced to use git (which I think is worth it).

If you're concerned about speed. The vast majority of dependencies will be on GitHub. If it's specialcased like I mentioned then this is not a concern anymore.

I guess this is the need for this sort of _releases branch mechanism, because the authors of the language do not want a centralized system, or to have to rely on particular services like Github. Of course, you bring up a good point (I missed this first, I guess you edited it in), and I think a special case for Github projects could be done, but a solution for non-Github projects is still required.

The size of the _releases branch can also get big, by the way.

Iirc, the branch will only have a single metadata text file. And changes to the file are probably only going to be changed version numbers, or an extra line to specify another dependency. Hence the _releases branch will always stay very small compared to the size of downloading snapshots of all the dependencies.

[vyp]

I just realized about a big problem with the _releases branch: it limits you to working with tags only. Can't work with a specific commit or master!

Sorry I don't follow, could you give a small hypothetical example?

Some people can't even be bothered making tags, let alone using this whole branch machinery.

@asterite has said crystal will make it painless to handle the branch. You're right, I don't want to handle this _releases branch by myself either, that would suck.

And how would this branching be implemented? I mean, there are a lot of shady things about it, like making an orphan branch.

Obviously I can't answer the first question, as I would also like to know that, but why is making an orphan branch shady?

And some tool would have to switch to a branch, make a commit and switch back? What if I have unstaged changes?

Good point. I guess the tool could git stash and git stash pop if necessary? But I think the tool would only be needed to be used when making a release, not at any other time. So hopefully it shouldn't interfere too much with people's git workflows. (I still think having a _releases branch for a faster deps install is worth this though, others may disagree?)

[vyp]

I just realized about a big problem with the _releases branch: it limits you to working with tags only. Can't work with a specific commit or master!

Oh sorry, I get what you mean now, you mean if you want a dependency to be on a specific commit or always on HEAD. Great point. (I guess for HEAD, a '*' for the version number could be used?)

[ozra]

• +1 semver
• +1 yaml
• +1 decentralization
• +1 "lib"
• +1 on require logic - and well - most all the points on the list :)
• Regarding the branching discussions, I'm not quite following, but personally I use git for everything (few on github atm though), and I always use a devel branch and compose releases on master branch - this is a scheme I'm happy with and will not change.
• Even though I use git for everything, I think it's a bit heavy handed to make that the only option!

[trans]

Let me try to clarify what I was saying b/c I am not sure it was understood.

1. A _releases branch is not required b/c a tool to stuff the information into that branch can just as easily stuff the information into a file on the current branch. The package manager only needs to checkout the latest tag (or master or specified branch if working with a development branch) and the file will contain all the dependency information for the current tag/branch and all the versions that came before it as well. One can think of it as a registry that is updated with each new release never loosing any of the previous information for prior releases.
2. My other suggestion of separating user edited metadata from the canonical file would allow developers to use whatever format they preferred (e.g. YAML, JSON, TOML, etc.) as long as an adapter existed in the aforementioned tool to convert it to the canonical format. It might feel a little odd to allow this degree of flexibility, but I think it is better than alienating potential developers.

Note that the canonical dependency information and the project metadata could be kept in separate files or just one file, since the data is generated.

[ysbaddaden]

The _releases branch would allow to speed things up dramatically, since it would be a single file to download and load in memory, instead of cloning all dependencies upfront or making lots of requests. It would be handled automatically by a command like "shards release". It won't disallow using refs (commit or tag or branch) but doing so would slow things down (eg: more requests to the server or shallow clones). It would also be retroactive, merging data from all previous releases when we implement it.

Yet, this isn't easy to implement, and there are lots of things to achieve before that. For now, dependencies are cloned upfront (bare clones) and metadata extracted with "git show" using tags for versioning (tags are required). It works, and it's fine, until we reach lots of dependencies and conflict resolution...

Only GIT and local path resolvers are supported by Shards, but adding resolvers for SVN or Mercurial is on the roadmap, just not priority since I don't use anything but Git and then we'll have to maintain them, speed them up, etc.

We can't have different formats for the metadata file, like one project uses TOML, another one is using YAML or JSON... Let's avoid aliases!

[jwaldrip]

I have been thinking a lot about this and I am don't align with the couple of suggestions/decisions thus far. I thought it would be best to list out my suggestions and get feedback. So with that, here they are:

Semantic Versioning

I think we definitely need to go with semantic versioning. If we are going to offer any special version methods such as ~> then we need to adhere to ONE versioning scheme. Remember one of the things ruby taught us "convention over configuration".

The Format

YAML and TOML are both great markup languages in there own right. But... I have to admit, I love that Ruby Bundler's Gemfile is written in Ruby. I think crystal should adapt the same paradigm as its familiar and sticks with the ability for developers to contribute as the contribute to the rest of the language... by writing crystal. Not that the deserializer wouldn't be written in crystal too, but rather that I think this is more native.

Shardfile

A Shardfile should be how any project or library would define its dependencies. Unlike Gems in ruby, I think that a gemspec for libraries is a bit redundant.

Here is what I would imagine a Shardfile would look like:

# Define a source, to remain "sudo decentralized, we can defined a source format to tell crystal how to fetch the shard.
# Shards are looked for in sources in the order in which they were defined.
source git: "https://github.com"

# Latest Version
## takes the latest version tag, if no version tag exists then an error will be raised.
shard "jwaldrip/cool_library"

# Specifying Versions
shard "jwaldrip/cool_library", "v1.2.3"

# With a specified tag
shard "jwaldrip/cool_library", tag: "custom-tag"

# Branch
shard "jwaldrip/cool_library", branch: "custom-branch"

# Sha
shard "jwaldrip/cool_library", sha: "da0406ad212d0e0b6ac2116b1e599d9b6330f558"

# Inline source
shard git: "https://github.com/jwaldirp/cool_library.git"

The Lockfile or Shardfile.lock

Another thing to take from Ruby's bundler is a lockfile to store information local to the project. I think this is better than managing things in a branch and is more portable to other version control systems (or lack there of). One thing I would suggest is that the lock file ALSO be written in crystal. This is a bit different than bundler but I think it aligns with my previous statement.

The lockfile would not be an exact clone of the Shardfile itself, but rather translated into each shards lowest common denominator. In the case of git this would be the expanded source and a SHA.

Example:

Shardfile

source :github, git: "https://github.com"

shard "jwaldrip/cool_libary"

Shardfile.lock

shard git: "https://github.com/jwaldrip/cool_library.git", sha: "da0406ad212d0e0b6ac2116b1e599d9b6330f558"

The Command Line

Although longer hand, I think that the shard commands should live under the crystal command itself. It would look like this: crystal shards [subcmd]

Given the format above I think the following commands should be supported:

command	description
crystal shards install	installs any dependencies into ./shards from the lockfile (runs upgrade -A if no lockfile exists)
crystal shards upgrade -A	updates all the given libraries.
crystal shards upgrade [name]	updates the lockfile for a library and runs install.

Fetching, Speed, and Storage

Like NPM, crystal should be able to do things in parallel... so I don't think network connections should be a problem. In addition all shards should be stored in the shards directory at the root of the project. In addition, the shards directory should only contain DIRECT dependencies, and not any child dependencies. I will go into this next... but a structure should look as follows:

- Shardfile
- Shardfile.lock
- shards/
  - github/
    - jwaldrip/
      - cool_library/
        - Shardfile
        - Shardfile.lock
        - shards/
          - bitbucket/
            - johndoe/
              - jane_libary/

TRULY!! Decentralized

As you can see by the above structure, johndoe/jane_library is a child dependency of jwaldrip/cool_libary. This child dependency should be tracked within its depender, and not at the project root. This allows for true decentralization and means that there wont be conflicting dependencies in libraries. (Note: There may be special exceptions to the rule here that I am open to.)

Releases

Releases should be managed using crystal defined conventions within the user preferred version control system. I think this is where convention comes in handy since we want a decentralized approach. If we do in fact want to support various version control systems, it makes sense for the Shardfile to understand how to fetch from those systems, but seems counter productive or over complex to build a release command/function into the command line too.

[ysbaddaden]

No nested dependencies ala node_modules. That won't happen. It doesn't fit the way Crystal requires dependencies (globally, not scoped). Dependencies are installed flattened in the libs folder.

The lock file is being implemented. It's a flattened list of all dependencies and only applies to the project. See https://github.com/ysbaddaden/shards/pull/27

[ysbaddaden]

Just reading myself: sorry for the tone. I didn't mean to be rude!

[asterite]

@jwaldrip

Semantic versioning: we could enforce that. But does this mean just the "1.x.x" syntax? Elm seems to check changes in the code, although I'm not sure how would you do that.

DSL: The problem with using a DSL is that you'll have to fire the compiler multiple times to compile that. And then it becomes hard to communicate data between these processes. In Ruby it's easier because you can eval, instance_eval and share state in the same single running process. Also, what would the shard method do? It can't do anything by itself, this info needs to be gathered and then used in conflict resolution, so at most that would just register this info in a data structure... which is the same as using JSON, YAML or some other format.

Command line: I also think doing crystal shards or crystal deps will be much more intuitive than just shards (a separate command).

Child dependencies: as @ysbaddaden mentions, in node.js you do some_module = require("...") and then you use it through some_module, so you don't have a global name conflict. But in Crystal it's just require "..." and everything gets clamped together. This sounds good in node, until you find out you have a lot of duplicated, almost the same code, in your dependencies directory for no real reason. I have a node_modules directory for a relatively small app that occupies 200 MB, ugh... Crystal's drawback is that you have a single namespace, but I find that to be more intuitive and simple if used correctly (no need to fill you code with multiple require or import statements at the top of your files).

[PragTob]

semantic versioning: I meant the code checks, no idea how they do it - I just wanted to put it in as an idea as I found that to be amazing about elm. Not sure how it plays out in practice. I'm perfectly fine with putting it on a "future wishlist" or abandoning it as too complex :)

[jwaldrip]

@asterite

_Semantic Versioning:_ yes as the format for semver, nothing more.

_DSL:_ I don't know why the compiler would have to run multiple times. I would see that the only thing that uses the shardfile and its lock would be the shards/deps command itself. Once the dependencies are downloaded they are in the directory tree and can be required using "..." or their name as defined in the CRYSTAL_PATH. When cloning a project, it just customary to run a dependencies command.

_Child Dependencies:_ I see your point, but my counter point is that in bundler it gets annoying when two libraries child dependencies conflict. This could certainly be the exception, but I don't think it lends to as much flexibility... unless there is a way to both.

[waterlink]

On semver standard is described here: http://semver.org/ Format is pretty strict, but allows for labels after patch version: '1.0.0-alpha' for example.

Best Regards, Alexey Fedorov, Sr Ruby, Clojure, Crystal, Golang Developer, Microservices Backend Engineer, +49 15757 486 476

2015-09-06 18:32 GMT+02:00 Jason Waldrip notifications@github.com:

@asterite https://github.com/asterite

semver: yes as the format for semver, nothing more. DSL: I don't know why the compiler would have to run multiple times. I would see that the only thing that uses the shardfile and its lock would be the shards/deps command itself. Once the dependencies are downloaded they are in the directory tree and can be required using "..." or their name as defined in the CRYSTAL_PATH. When cloning a project, it just customary to run a dependencies command.

— Reply to this email directly or view it on GitHub https://github.com/manastech/crystal/issues/1357#issuecomment-138097996.

[jwaldrip]

@waterlink I agree with that too.

[trans]

@ysbaddaden

We can't have different formats for the metadata file, like one project uses TOML, another one is using YAML or JSON... Let's avoid aliases!

There would only be one definitive file, but the developer can work with whatever source format they'd like. The distinction between them is similar to the difference between Gemfile and Gemfile.lock.

[tomchapin]

:+1: for the shard ideas presented by @jwaldrip

[ibash]

On the topic of source formats. I think it's better to have a single format than many. The reason why is that when looking at another library or project, you immediately know what file to pick out and how to read it.

What the specific format is does not matter much, developers will learn it and most of the formats mentioned are human friendly enough to be useable.

I'm partial to json, but if it's yaml, toml, or something else, it's not going to bother me.

[refi64]

Ok...this is kind of off-topic...but I can't help it...

Guess what the full name of Kirby 64 was?

Kirby 64 and the Crystal Shards

(notice the bolded text)

Sorry, but it seemed kind of funny to me. :)

[ibash]

I vote next tool be named Kirby.

[asterite]

Starting from https://github.com/manastech/crystal/commit/b86a5b8e179ca7022b3cc4f15e291c31f39ae19a crystal deps now delegates to shards \o/

The idea is that eventually shards will be distributed with the compiler, so it's always found.

[ozra]

+1 for "Kirby", haha.

@jwaldrip - as already mentioned the hierarchical dependencies á la npm/iojs does not work with Crystal. This is the reason I strongly believe semver should be enforced. The semantics of the scheme makes it straight forward to deduce which latest possible versions are compatible for all dependants - or if there's a conflict in version needs that needs to be worked out!

"Flat namespace" == great need for a good convention of dependency compatibility checks == semver.

The "strictness" of the semver rules aren't a problem at all - they're super straight forward.

[ysbaddaden]

This is the last time I express myself about semver: it's nice, follow it if you can and want, but I won't enforce it.

1. Depending on the project, having 1.2 or 1.2.3.4 scheme can be perfectly valid.
2. You can't trust minor won't break your code. A bug fix will almost always break something.
3. Hell, you can't even trust a patch! A security fix may lead to a breaking change. Will you bump major for a tiny fix?! I won't.

That being said. Semver is very good. It helped focus to keep a sane version scheme and a stable code that can can somehow rely on. You should try to apply its principles as much as possible.

But you can't follow it blindly, you can't enforce it, and I won't. This is developer responsibility.

[ozra]

Fair enough, the point that developers efforts of versioning correctly might not match reality is a good one. I still don't see how a good unification of dependencies can be achieved without that "promise" of a strife to get it as right as humanly possible though. In any event, this will be my last commenting on SemVer too. Having "official" modules managing in crystal is great!

[refi64]

Despite not being a big SemVer fan, you need to define a standard versioning systems if you want dependencies to be managed correctly. If someone adds package B version >=1.2.1 as a dependency to package A, but package B tags the releases in some weird way, it'll screw everything up.

Maybe not as strict as SemVer, but some kind of control.

[jwaldrip]

So what is the status on the package manager?

[jhass]

The current status is that crystal deps delegates to shards, which you can install from http://github.com/ysbaddaden/shards

Maybe we should close this issue now in favour of more specific ones?