Unable to resolve dependency with pre-release version

I'm still trying to reduce the issue, but for now to reproduce:

Download Athena monorepo at: https://github.com/athena-framework/athena/tree/bump-components
run SHARDS_OVERRIDE=shard.dev.yml shards install
Notice you get:

Resolving dependencies
Fetching https://github.com/crystal-ameba/ameba.git
Unable to satisfy the following requirements:

- `athena-console (*)` required by `shard.yml`
- `athena-console (*)` required by `athena 0.17.1`
Failed to resolve dependencies

Currently also unsure if I'm just missing something or there is a bug here.

Debug Notes

Tested against shards 0.17.2
The root shard.yml doesn't have any version constraints and sources everything from latest GH release
shard.dev.yml doesn't have any version constraints either and sources everything from the related path within src/components
The console component is versioned as 0.4.0-dev
The framework component's version constraint for the console component is >=0.3.0, <0.4.0, which 0.4.0-dev should be valid against
- Confirmed via matches?("0.4.0-dev", ">=0.3.0, <0.4.0").should be_true in shards repo
Changing the version constraint to >=0.3.0, <=0.4.0-dev produces same error
Changing the version constraint to 0.4.0-dev also produces the same error
Moving the path sources out of shard.dev.yml into shard.yml and trying again produces:

Resolving dependencies
Fetching https://github.com/crystal-ameba/ameba.git
Fetching https://github.com/athena-framework/spec.git
Unable to satisfy the following requirements:

- `athena-console (*)` required by `shard.yml`
- `athena-console (>=0.3.0, <0.4.0)` required by `athena 0.17.1`
Failed to resolve dependencies

So far unsure if the path part of it is the problem, or the fact the component versions have pre-releases
Could it have something to do with the current latest release of the framework component not having an athena-console dep? My assumption was that the override file would make it just use whatever is in src/components/framework/shard.yml.

Reduced it to a test case in molinillo. Definitely seems related to the pre-release shard version.

Steps to reproduce:

https://github.com/crystal-lang/crystal-molinillo#development

Create spec/fixture/index/pre_release.json with:

{
"console": [
{
  "name": "console",
  "version": "0.4.0-dev",
  "dependencies": {
  }
}
],
"framework": [
{
  "name": "framework",
  "version": "0.17.1",
  "dependencies": {
    "console": ">=0.3.0, <0.4.0"
  }
}
],
"athena": [
{
  "name": "athena",
  "version": "0.0.0",
  "dependencies": {
    "framework": "*",
    "console": "*"
  }
}
]
}

Create spec/fixture/case/pre_release.json with:

{
"name": "resolves an index with a pre-release version",
"index": "pre_release",
"requested": {
"athena": ""
},
"base": [],
"resolved": [
{
  "name": "athena",
  "version": "0.0.0",
  "dependencies": [
    {
      "name": "framework",
      "version": "0.17.1",
      "dependencies": [
        {
          "name": "console",
          "version": "0.4.0-dev",
          "dependencies": []
        }
      ]
    },
    {
      "name": "console",
      "version": "0.4.0-dev",
      "dependencies": []
    }
  ]
}
],
"conflicts": []
}

Run crystal spec
Notice it fails:

  1) Molinillo::Resolver(R, S) dependency resolution with the TestIndex index resolves an index with a pre-release version

       Unable to satisfy the following requirements:

       - `console` required by `athena (0.0.0)`
       - `console` required by `framework (0.17.1)` (Molinillo::VersionConflict(Gem::Dependency | Molinillo::TestSpecification, Molinillo::TestSpecification))
         from src/molinillo/resolution.cr:333:9 in 'raise_error_unless_state'
         from src/molinillo/resolution.cr:316:11 in 'unwind_for_conflict'
         from src/molinillo/resolution.cr:698:13 in 'attempt_to_activate'
         from src/molinillo/resolution.cr:266:11 in 'process_topmost_state'
         from src/molinillo/resolution.cr:200:11 in 'resolve'
         from src/molinillo/resolver.cr:33:7 in 'resolve'
         from spec/resolver_spec.cr:78:7 in 'resolve'
         from spec/resolver_spec.cr:96:20 in '->'
         from /usr/lib/crystal/spec/example.cr:45:13 in 'internal_run'
         from /usr/lib/crystal/spec/example.cr:33:16 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:339:7 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:339:7 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:339:7 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:156:7 in 'run'
         from /usr/lib/crystal/spec/dsl.cr:220:7 in '->'
         from /usr/lib/crystal/crystal/at_exit_handlers.cr:14:19 in 'run'
         from /usr/lib/crystal/crystal/main.cr:50:14 in 'exit'
         from /usr/lib/crystal/crystal/main.cr:45:5 in 'main'
         from /usr/lib/crystal/crystal/main.cr:127:3 in 'main'
         from /usr/lib/libc.so.6 in '??'
         from /usr/lib/libc.so.6 in '__libc_start_main'
         from ../sysdeps/x86_64/start.S:117 in '_start'
         from ??

Pretty sure it's a bug at this point given replacing the three instances of 0.4.0-dev with like 0.3.5 in both files allows it then pass, even tho both versions are valid based on the requirements.

Not sure if I'm seeing this exact same issue or not, but I did a soft release of Lucky and Avram 1.0.0 over the weekend, and now my apps won't pull 1.0.0-rc1 anymore.

My shard.yml file has this:

dependencies:
  lucky:
    github: luckyframework/lucky
    version: 1.0.0-rc1
  avram:
    github: luckyframework/avram
    version: 1.0.0-rc1

I run rm -rf lib/ shard.lock ~/.cache/ then shards install. When it's installing, I see this:

My shard.lock file has the correct version:

❯ cat shard.lock 
version: 2.0
shards:
  CrystalEmail:
    git: https://github.com/jwoertink/crystalemail.git
    version: 0.2.6+git.commit.960e1b9ad02288ae56110d39fe9d7ca3bbcdcd1e

  ameba:
    git: https://github.com/crystal-ameba/ameba.git
    version: 1.3.1

  authentic:
    git: https://github.com/luckyframework/authentic.git
    version: 0.9.0

  authly:
    git: https://github.com/azutoolkit/authly.git
    version: 1.2

  avram:
    git: https://github.com/luckyframework/avram.git
    version: 1.0.0-rc1
...
  lucky:
    git: https://github.com/luckyframework/lucky.git
    version: 1.0.0-rc1

Running shards list --tree shows the correct list

❯ shards list --tree
Shards installed:
  * lucky (1.0.0-rc1)
    * lucky_task (0.1.1)
    * habitat (0.4.7)
    * wordsmith (0.4.0)
    * lucky_router (0.5.1)
    * shell-table (0.9.3)
    * cry (0.4.3)
    * exception_page (0.3.0)
      * backtracer (1.2.2)
    * dexter (0.3.4)
    * pulsar (0.2.3)
    * teeplate (0.8.5)
  * avram (1.0.0-rc1)
    * lucky_task (0.1.1)
    * pg (0.26.0)
      * db (0.11.0)
    * habitat (0.4.7)

but the actual versions that show up in lib are wrong...

❯ cat lib/lucky/shard.yml 
name: lucky
version: 1.0.0

crystal: ">=1.6.0"

❯ cat lib/avram/shard.yml 
name: avram
version: 1.0.0

crystal: ">= 1.6.0"

These are installing 1.0.0 and not 1.0.0-rc1 even though it's saying that it is.

Now one very strange thing is, after I run shards install, if I rm -rf lib/lucky/ then shards update lucky that will install the rc1 version. It only installs the wrong version on shards install and shards update.

Edit: using shards 0.17.2

@jwoertink What you describe is pretty certainly unrelated, so I created a new issue for it: #581

The exclusion of dependencies with pre-release versions is deliberate and identical to bundler (https://github.com/rubygems/bundler/issues/2938, https://github.com/rubygems/bundler/issues/4340).

There is a specific case for a pre-release version matching against a dependency restriction that is not a pre-release and defines that as not satisfying the requirement:

https://github.com/crystal-lang/shards/blob/e56c44a5f056b767a34d97ddfcc56d82858d8fe9/src/molinillo_solver.cr#L225-L234

Note: @prereleases is only true when --pre option is passed to shards outdated. In any other use case (including shards install), @prereleases is false.

So I think this should be considered not as a bug report but a discussion about this pre-release behaviour. But considering that bundler behaves the same, I would expect good reasons for it.

@Blacksmoke16 Your reduction into a Molinillo test case is great, but it's actually coincidental: Molinillo has no understanding of versions and pre-releases. That's entirely delegated to the user of the Molinillo API, in form of implementing SpecificationProvider#requirement_satisfied_by?. The reason you see the same behaviour in the Molinillo specs is that the spec helper used to execute these test cases implements the same pre-release logic as shards and bundler.

Interestingly, the most recent bundler release has added support for the --pre option to the update and lock commands: https://bundler.io/blog/2023/01/31/bundler-v2-4.html#new-cli-features

We added a few small CLI features, such as a new --pre flag to bundle update and bundle lock to explicitly opt-in to prereleases of selected (of all) gems without having to explictly change your Gemfile with pre-release requirements such as >= 7.1.0.beta.

(The same release also replaced Molinnillo by PubGrub, but I don't think the option is related to the resolver engine)

crystal-lang / shards

Unable to resolve dependency with pre-release version #572

Debug Notes