[CT-277] [Feature] Improve mismatching package errors

joellabes commented 2 years ago

Is there an existing feature request for this?

[X] I have searched the existing issues

Describe the Feature

An extremely common failure point in the Community Slack is "I'm trying to install a new package (or update a package), and am hitting version errors". Example:

Version error for package dbt-labs/dbt_utils: Could not find a satisfactory version from options: ['=0.7.6', '>=0.6.2', '<0.8.0', '>=0.7.0', '<0.8.0', '>=0.8.0', '<0.9.0'] Code: 10006

NB: Everywhere I say "utils" in this issue, I mean "the package we're trying to install but have conflicts"

Option 1. Minimum viable improvement:

It would be much easier for people to debug if we listed the packages and their requested utils versions:

package_1: >= 0.7.6
package_2: >= 0.6.2, <0.8.0
package_3: >= 0.7.0, <0.8.0
package_4: >= 0.8.0, < 0.9.0

By looking at that, you can see that packages 2 and 3 are in conflict with package 4. Displaying this information in the output would go a loooong way to clarifying + contextualising why there's so many version numbers being listed.

Option 2. Extra for experts

Combine this with the upgrades available notifications in https://github.com/dbt-labs/dbt-core/pull/3759 to call out possible fixes. This is the lazy but probably almost always correct version:

assume package 4's goal is right-est, because it is targeting the highest utils version
assume that if packages 2 and 3 have new versions available, their latest versions import the latest utils version as well

package_1: >= 0.7.6
package_2: >= 0.6.2, <0.8.0 update available: install package_2 v0.7.0 instead
package_3: >= 0.7.0, <0.8.0 update available: install package_3 v0.4.2 instead
package_4: >= 0.8.0, < 0.9.0

Option 3. Extr-est for expert-ests

I don't think this is necessary out of the gate, but I wanted to call out the edge case that would be created by Option 2. We could actually go and validate which utils version the latest version of each requested package can support:

package_1: >= 0.7.6
package_2: >= 0.6.2, <0.8.0 
package_3: >= 0.7.0, <0.8.0 update available: install package_3 v0.4.2 instead
package_4: >= 0.8.0, < 0.9.0

package_2 has no version available which is compatible with dbt_utils [>= 0.8.0, <0.9.0]. 
You can: 
- Remove it from your packages.yml file 
- Install v0.5.3 of package_4 instead (requires dbt_utils [>=0.7.0, <0.8.0])

If package_2 is a package like audit-helper, it's actually reasonable to suggest getting rid of it for a while.

Describe alternatives you've considered

I have dreams of some sort of package shopping cart on the Hub where you list all the packages you have/want and your target dbt version, and it builds your packages.yml file for you. We could delegate this to the Hub site instead

Who will this benefit?

The [~30 people who have posted in the Slack]("Could not find a satisfactory version from options" before:2022-02-25 after:2021-02-24) about this over the last 12 months
@jaypeedevlin, who does the Lord's work answering a lot of them

Are you interested in contributing this feature?

I think I could do option 1, would need some help on the others

Anything else?

No response

oscarlukersmithsc commented 2 years ago

Hi @joellabes, I think this is a great improvement and would avoid my confusion in the future. I am not sure option 3 is necessary from my end. Option 2 is clear, concise and provides all the information I need to resolve.

emmyoop commented 2 years ago

@joellabes this seems like a reasonable and logical request! Listing out the specific packages with the versions (or version ranges) would definitely be helpful!

The error itself is being triggered here:

https://github.com/dbt-labs/dbt-core/blob/0fce63665cd8a45fbc069c02ae174e194ed5892b/core/dbt/semver.py#L377-L385

Right now we don't have access to the name of the package so we would need find a way for that to get passed along with the version.

My only hesitation with option 2 is causing confusion if there is an update available but that update does not actually resolve this problem. Sometimes the solution may be to downgrade a package to resolve mismatching package errors. The same goes for option 3. The solution may be to downgrade package 3 and 4 and then there will be no conflict with package 2. Please correct me if this is not the case!

Perhaps just a note that an update is available for the package but not instructing the specific version to upgrade to would be enough information?

joellabes commented 2 years ago

Agreed that option 2 could cause confusion - option 3 does address the option of saying "you might have to downgrade package 4" - the two bulleted items (remove package 2 or downgrade package 4) are alternatives as opposed to a checklist.

My gut feeling is that although we could knock off option 3 in the future, it would be an excessive amount of work for the incremental benefit.

If we did option 2 and there were no upgrades available, we'd still be listing out all the selected packages, which means they can see which one is causing trouble.

Perhaps just a note that an update is available for the package but not instructing the specific version to upgrade to would be enough information?

If we're going to say an upgrade version available, I think that we might as well list which the latest version is to make the default path easier.

fivetran-jamie commented 2 years ago

i'm wondering how helpful option 2 is without information about the version upgrade....would surfacing the text in the new version's Release Notes be on the table?

saraleon1 commented 1 year ago

+1 dbt Cloud user - are using a package that contains a package, and the error surfaced was for the package version within the other package, which was confusing to parse out.

saraleon1 commented 1 year ago

+1 dbt Cloud user - determining compatibility between package and core versions can feel more difficult than it needs to be for this user

joellabes commented 1 year ago

would surfacing the text in the new version's Release Notes be on the table?

@fivetran-jamie we don't have the release notes information on the hub at the moment, so that would be a prerequisite. With that said, I don't think we'd want to drop potentially-very-long release notes into the console output anyway.

aashishdbt commented 1 year ago

+1 One of my dbt cloud customers did report this today to me as well.

They were using the following package:

package: dbt-labs/logging version: 0.6.0

joellabes commented 1 year ago

Another one: https://discourse.getdbt.com/t/version-error-for-package-dbt-labs-dbt-utils/5746

dave-connors-3 commented 1 year ago

the great @jtcohen6 spun up a working prototype over in #6902

for this packages.yml:

packages:
  - package: dbt-labs/dbt_utils
    version: 1.0.0
    # requires dbt-utils between 0.8.0 and 0.9.0
  - package: dbt-labs/codegen
    version: 0.6.0

🎩

Current behavior

$ dbt deps
14:50:55  Running with dbt=1.3.1
14:50:55  Encountered an error:
Version error for package dbt-labs/dbt_utils: Could not find a satisfactory version from options: ['=1.0.0', '>=0.8.0', '<0.9.0']

New behavior

$ dbt deps
15:05:31  Running with dbt=1.5.0-a1
15:05:32  Encountered an error:
Version error for package dbt-labs/dbt_utils: 
    Required by user: ['=1.0.0']
    Required by dbt-labs/codegen@0.6.0: ['>=0.8.0', '<0.9.0']

millyleadley commented 1 year ago

+1 for this, just ran into it and was about to raise an issue 😄 thank you for addressing it

github-actions[bot] commented 6 months ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 6 months ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

joellabes commented 2 months ago

Return to meeeee

Just saw another discussion in Slack, and I don't have any idea how it could be happening the way they're describing so I wish dbt would have listed out the offending packages

Running with dbt=1.7.16
Encountered an error:
Could not find a matching compatible version for package dbt-labs/dbt_utils
  Requested range: =0.7.6, =0.7.6
  Compatible versions: ['0.0.1', '0.1.0', '0.1.1', '0.1.2', '0.1.3', '0.1.4', '0.1.5', '0.1.6', '0.1.7', '0.1.8', '0.1.9', '0.1.10', '0.1.11', '0.1.12', '0.1.13', '0.1.14', '0.1.15', '0.1.16', '0.1.17', '0.1.18', '0.1.19', '0.1.20', '0.1.21', '0.1.22', '0.1.23', '0.1.24', '0.1.25', '0.2.0', '0.2.1', '0.2.2', '0.2.3', '0.2.4', '0.2.5', '0.3.0', '0.4.0', '0.4.1', '0.5.0', '0.8.0', '0.8.1', '0.8.2', '0.8.3', '0.8.4', '0.8.5', '0.8.6', '0.9.0', '0.9.1', '0.9.2', '0.9.5', '0.9.6', '1.0.0', '1.1.0', '1.1.1', '1.2.0']
  Not shown: package versions incompatible with installed version of dbt-core
  To include them, run 'dbt --no-version-check deps'

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.1

  - package: bqbooster/dbt_bigquery_monitoring
    version: 0.5.0

  - package: brooklyn-data/dbt_artifacts
    version: 2.6.3

https://getdbt.slack.com/archives/CBSQTAPLG/p1718900461904149

dbt-labs / dbt-core