Wrong shard version in "shards outdated" command (1.0.0)

[petr-fischer]

I have this in my shard.yml:

...
dependencies:
  neo4j:
    github: jgaskins/neo4j.cr
  amqp-client:
    github: cloudamqp/amqp-client.cr
  raven:
    github: Sija/raven.cr

My shard.lock:

...
neo4j:
    git: https://github.com/jgaskins/neo4j.cr.git
    version: 0.6.0
...

shards list output:

...
  * neo4j (0.6.0)
  * amqp-client (0.6.6)
  * amq-protocol (0.3.24)
  * raven (1.9.1)
...

But when I run shards outdated, this is the output:

Unable to satisfy the following requirements:
...
- `crystal (~> 0.27, >= 0.27.0)` required by `neo4j 0.5.5`
Failed to resolve dependencies, try updating incompatible shards or use --ignore-crystal-version as a workaround if no update is available.

Why there is version neo4j 0.5.5 in the output? If I check lib/neo4j/shard.yml file, there is correct 0.6.0 version. I'm confused.

I deleted ~/.cache/crystal and ~/.cache/shards, but still the same output.

Ubuntu 20.10

➤  crystal --version
Crystal 1.0.0 [dd40a2442] (2021-03-22)

LLVM: 10.0.0
Default target: x86_64-unknown-linux-gnu

➤  shards --version
Shards 0.14.1 [addc26a] (2021-03-10)

[straight-shoota]

I can't reproduce this with a fresh shard.yml + shards install. Can you provide the full shard.lock please?

And try if it reproduces for you when you remove shard.lock and run shards.install.

[straight-shoota]

Oh wait, I had the wrong crystal version in path. It really depends on resolving against Crystal version 1.0.0. With 0.36.1 it works.

[petr-fischer]

FYI: neo4j library/dependency is not prepared for 1.0.0 yet - in their shard.yml, there is still crystal: 0.34.0 - but why is the shards utility talking something about neo4j 0.5.5 (much older version) is still unclear for me - probably bug?

@straight-shoota So you can reproduce the situation in your environment with 1.0.0?

[straight-shoota]

Yes, it reproduces with 1.0.0.

It's really confusing that neo4j installs with crystal: 0.34.0.

[petr-fischer]

It's really confusing that neo4j installs with crystal: 0.34.0.

Installing is possible with --ignore-crystal-version.

[straight-shoota]

Since the release of neo4j@0.6.1 the original configuration doesn't reproduce any more. Adding version: "<= 0.6.0" restriction does.

[straight-shoota]

I missed yesterday that CRYSTAL_VERSION=1.0.0 shards install already fails, not just shards outdated. So this is not specific to the outdated command.

The core of the issue is that shards complains about neo4j 0.5.5 being incompatible when there are three more recent release available: 0.5.6, 0.5.7 and 0.6.0 which are still incompatible, but it would make more sense to report a bout the most recent version than a seemingly random one.

This behaviour reproduces in isolatiotion with the following integration spec:

diff --git i/spec/integration/install_spec.cr w/spec/integration/install_spec.cr
index cc993ef..0a290b1 100644
--- i/spec/integration/install_spec.cr
+++ w/spec/integration/install_spec.cr
@@ -1258,4 +1258,15 @@ describe "install" do
       end
     end
   end
+
+  it do
+    metadata = {
+      dependencies: {
+        "neo4j": "*"
+      }
+    }
+    with_shard(metadata) do
+      run "CRYSTAL_VERSION=1.0.0 shards install"
+    end
+  end
 end
diff --git i/spec/integration/spec_helper.cr w/spec/integration/spec_helper.cr
index 5451583..44c2b19 100644
--- i/spec/integration/spec_helper.cr
+++ w/spec/integration/spec_helper.cr
@@ -185,8 +185,21 @@ private def setup_repositories
   create_git_version_commit "branched", "0.1.0"
   checkout_git_branch "branched", "master"
   create_git_release "branched", "0.2.0"
+
+  create_git_repository "neo4j"
+  create_git_release "neo4j", "0.5.4", {crystal: "0.27.0"}
+  create_git_release "neo4j", "0.5.5", {crystal: "0.27.0"}
+  create_git_release "neo4j", "0.5.6", {crystal: "0.30.0"}
+  create_git_release "neo4j", "0.6.0", {crystal: "0.34.0"}
 end

I've established the solver applies the following order between these versions: 0.6.0 < 0.5.6 < 0.5.4 < 0.5.5.

This suggests that it seems to prefer the version with the lowest crystal restriction. And in fact, all neo4j releases below 0.5.5 have the same crystal: 0.27.0 restriction, 0.5.5 is just the highest version with that lowest Crystal dependency.

Complaining about an old version being incompatible is certainly most unexpected behaviour. But there are two alternatives to that:

• Either the error shows the most recent version and complains about that being incompatible.
• Or the error should say that no version is compatible.

[daliborfilus]

Hi! I have the same problem, but with shards install, shards update (+ shards outdated) for shard that IS COMPATIBLE with Crystal 1.0.0.

So this issue does not only affect the report in the situation none of the shards can be installed, but also when there IS new compatible shard available, but somehow the check stumbles upon the old version first and complains about it before even trying to verify the newest version.

I have the newest version in my shard.lock:

❯ cat shard.lock | grep pg -A2
  pg:
    git: https://github.com/will/crystal-pg.git
    version: 0.23.2

No other transitive dependency exists on the "pg" shard anywhere else.

❯ cat shard.yml | grep pg
  pg:
    github: will/crystal-pg

❯ shards update
Resolving dependencies
Fetching https://github.com/will/crystal-pg.git
[private shards removed from output - they are leaf and have no dependencies whatsoever, so cannot affect pg itself]
Fetching https://github.com/crystal-lang/crystal-db.git
Unable to satisfy the following requirements:

- `crystal (< 1.0.0)` required by `pg 0.21.1`
Failed to resolve dependencies, try updating incompatible shards or use --ignore-crystal-version as a workaround if no update is available.

Output of shards install is different - it complaints any of the listed shards aren't compatible, but the versions match!

❯ shards install
Resolving dependencies
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
[removed]
Unable to satisfy the following requirements:

- `crystal (>= 0.35.0, < 2.0.0)` required by `pg 0.23.2`
...
- `crystal (>= 0.35.0, < 2.0.0)` required by `db 0.10.1`
Failed to resolve dependencies, try updating incompatible shards or use --ignore-crystal-version as a workaround if no update is available.

To the naked eye each of the crystal version constraint listed should match - and confusingly enough, it tries to use the correct version.

Maybe the version constraint specified by "pg" shard isn't somehow accepted properly as matching for 1.0.0 and because of that, the update command skips it as incompatible too?

Newest version of pg shard is 0.23.2 (which I have in shard.lock) which has this crystal version:

crystal: ">= 0.35.0, < 2.0.0"

Env: crystal and shards from snap: crystal 1.0.0 652 latest/stable crystal-lang✓ classic Crystal 1.0.0 [dd40a2442] (2021-03-22) Shards 0.14.1 [addc26a] (2021-03-10)

Tried rm -rf lib, rm -rf ~/snap/crystal/common/.cache ... nothing helped. Confusingly enough, on my second computer, shards install worked. But maybe just because I installed the shards first using non-snap versions first.

[daliborfilus]

Btw this affects even shards install for shards itself (at least for me):

❯ git checkout addc26a 
HEAD is now at addc26a Release 0.14.1 (#481)

shards on HEAD 
❯ shards install
Resolving dependencies
Fetching https://github.com/crystal-lang/crystal-molinillo.git
Unable to satisfy the following requirements:

- `crystal (>= 0.35.0, < 2.0.0)` required by `molinillo 0.2.0`
Failed to resolve dependencies, try updating incompatible shards or use --ignore-crystal-version as a workaround if no update is available.

❯ shards install --ignore-crystal-version
Resolving dependencies
Fetching https://github.com/crystal-lang/crystal-molinillo.git
Installing molinillo (0.2.0)
Shard "molinillo" may be incompatible with Crystal It seems that some libraries are missing in your host.
  The following are the suggested packages to be able to use the whole standard library capabilities.

    $ sudo apt-get install gcc pkg-config git tzdata \
                           libpcre3-dev libevent-dev libyaml-dev \
                           libgmp-dev libssl-dev libxml2-dev

  You can find more detailed information in:

    * https://crystal-lang.org/reference/installation/on_debian_and_ubuntu.html
    * https://github.com/crystal-lang/crystal/wiki/All-required-libraries

1.0.0

I have these libraries installed, but snap cannot see them, which is another issue I'll try to solve.

Could this all relate to that it cannot install the newest shard because of missing library dependencies, then goes to older version and tries that - that fails foo... and so an so forth until it comes to the oldest version?

It "says" the crystal version didn't match and the shard is incompatible, but the real problem is missing system dependency?

Don't know if it's the same as with neo4j, but the symptoms are similar: the shard cannot be installed because of missing something (lib in pg case, crystal version in neo4j case) and it goes down the version list until it cannot and reports the oldest version tried instead of the latest.

[Sija]

@daliborfilus Looks more like a duplicate of #491.

[daliborfilus]

It might, but the cause is different. The #491 talks about issues relating to <1.0.0, but my case fails to install the shard because of other reasons and then complains about crystal version, which is not a problem and is correct.

And indeed, installing the missing required sytemwide library fixed the problem for me.

Looking at it once more it seems to me that my issue and #491 are very related and happens if the shard cannot be installed for some reason, wrong versions are printed out.

shards outdated listing wrong version might be a different problem, but may have the same cause? (Shards cannot be installed, try older version, then again and again and print the final version tried, instead of the latest?)

Anyway, sorry for the spam.

But I must add some to the spam: I just thought of a somewhat weird scenario for shards update. Hypothetically, imagine you have a shard xyz v1.10, which has a dependency on system lib, e.g. libvdpau. Then you have shard xyz v1.05, which doesn't have this dependency. Then you have a project, depending on xyz and in shard.lock have specified version 1.10. Now you want to install the shard, but you don't have libvdpau installed. When you do shards update, this "auto-skip non-installable versions" feature might be a burden, because it could try to satisfy your dependencies using xyz 1.05, even though you specified 1.10 in your shard.lock and 1.05 isn't the latest version and might also be bug-ridden, exploit-full version? I know that shards update doesn't necessarily read shard.lock, but even without that little detail, it still shouldn't install non-latest version, when you depend on "latest" by not-specifying the version in the first place?

Can you see this happening? Could it be an issue, worth creating a separate ticket for it? If not, sorry for even more spam.

[straight-shoota]

@daliborfilus I have no explanation why this is happening to you. It doesn't reproduce with my setup, as it did not on your second computer. So my guess would be it has something to do with your Crystal installation. Maybe there is an additional older Crystal versions installed (snap and non-snap perhaps?) and shards confuses them. I don't know. Maybe you could try to recreate your setup in isolation to see if you can find a configuration that reproduces the error.

Could this all relate to that it cannot install the newest shard because of missing library dependencies, then goes to older version and tries that - that fails foo... and so an so forth until it comes to the oldest version?

It "says" the crystal version didn't match and the shard is incompatible, but the real problem is missing system dependency?

No, shards is a source-based dependency resolver. Binary dependencies do not have any effect on shards' behaviour. So, regarding your secondary issue, there is no "auto-skip non-installable versions" feature. The purpose of shards.lock is just to document the dependency state and make it reproducible. shards update will always discard that information. If you need some version restrictions on a dependency, you have to place them in shard.yml (or in an override).