Bug (or "undocumented feature?") in dependency resolution using `--latest true`

pwillis-els commented 3 years ago

I'm not sure if this is expected behavior, but it certainly was unexpected to me, and seems like it might be a bug. Slightly complicated to explain, but I'll try...

Expected behavior:

User A has a list of pinned plugins to install. They have the entire set of plugins (+ dependencies) that are needed. They pass the list of plugins to the install tool.
The tool goes through each item and collects its dependencies.
Seeing as the entire list of dependencies has been provided, and all of them are pinned, the tool should only install what has been provided and nothing more, and return successfully.

But this is what is happening:

User A has a list of pinned plugins to install. They have the entire set of plugins (+ dependencies) that are needed. They pass the list of plugins to the install tool.
The tool goes through each item and collects its dependencies. It then retrieves the latest version of each dependency (due to --latest default of true), regardless of what has been pinned.
The tool fails with an error that it can't resolve the newer dependency it wanted to collect, because that dependency was already provided and pinned to an earlier version. The tool ignores the fact that the pinned version actually satisfies the dependency.

The "bug" here is that the list User A is providing is complete and accurate. All plugins + dependencies have been provided and pinned to their minimum required versions. But the tool is erroring out, seemingly due to ignoring the pinned items in favor of the latest dependencies it wants to install.

Maybe I missed something and something else is going on? But I've combed through all the MANIFEST.MF files of all the plugins and even the plugin-versions.json file, and all the dependencies shown below seem to be correct. It seems to be just --latest true that is failing.

The plugins below has been crafted to provide the minimum versions of all plugins + dependencies in this chain.

Sample failure with --latest true:

jenkins-plugin-cli --no-download --verbose --latest true --plugins active-directory:2.20 apache-httpcomponents-client-4-api:4.5.10-1.0 aws-credentials:1.23 aws-java-sdk:1.11.341 aws-parameter-store:1.2.2 configuration-as-code:1.27 configuration-as-code-secret-ssm:1.0.1 credentials:2.3.5 credentials-binding:1.23 display-url-api:2.3.1 git:4.2.1 git-client:3.0.0 github:1.29.2 github-api:1.90 jackson2-api:2.7.3 job-dsl:1.72 jsch:0.1.55.1 ldap:2.2 mailer:1.32.1 matrix-auth:2.6.2 plain-credentials:1.6 role-strategy:3.1 scm-api:2.6.3 script-security:1.75 ssh-credentials:1.17.3 structs:1.20 timestamper:1.11.2 token-macro:2.8 workflow-api:2.39 workflow-scm-step:2.9 workflow-step-api:2.22
No .txt or .yaml file containing list of plugins to be downloaded entered.
No directory to download plugins entered. Will use default of /usr/share/jenkins/ref/plugins
Using update center https://updates.jenkins.io/update-center.json from JENKINS_UC environment variable
Using experimental update center https://updates.jenkins.io/experimental/update-center.json from JENKINS_UC_EXPERIMENTAL environment variable
Using incrementals mirror https://repo.jenkins-ci.org/incrementals from JENKINS_INCREMENTALS_REPO_MIRROR environment variable
No CLI option or environment variable set for plugin info, using default of https://updates.jenkins.io/plugin-versions.json
No war entered. Will use default of /usr/share/jenkins/jenkins.war
Jenkins version: 2.272

Retrieving update center information
Created cache at: /var/jenkins_home/.cache/jenkins-plugin-management-cli
Cache miss for: update-center-2.272
Cache miss for: experimental-update-center-2.272
Cache miss for: plugin-versions
Couldn't find checksum for active-directory at version: 2.20
Setting checksum for: active-directory to CGamy8Tm1yzx0TkE6/Sj+6UwPWqodup52RAGxP5MWjU=

active-directory depends on: 
mailer 1.32.1
Skipping dependency mailer:1.32.1 and its sub-dependencies, because there is a higher version defined on the top level - mailer:1.32.1
Couldn't find checksum for apache-httpcomponents-client-4-api at version: 4.5.10-1.0
Setting checksum for: apache-httpcomponents-client-4-api to kTaiJaAAbVqBISY2AWNVjkhhl6DbXKVW6sEDKxTYdNs=

apache-httpcomponents-client-4-api has no dependencies
Couldn't find checksum for aws-credentials at version: 1.23
Setting checksum for: aws-credentials to IP5LXBJzruZrKK21BMzQ+7C0TiQLhL3HTvejh0nucm4=

aws-credentials depends on: 
credentials 2.3.14
aws-java-sdk 1.11.854
credentials-binding 1.24
io.jenkins.tools.pluginmanager.impl.PluginDependencyStrategyException: Plugin aws-credentials:1.23 depends on credentials:2.3.14, but there is an older version defined on the top level - credentials:2.3.5
    at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:883)
    at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:493)
    at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:157)
    at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:117)
    at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:76)
Plugin aws-credentials:1.23 depends on credentials:2.3.14, but there is an older version defined on the top level - credentials:2.3.5

Sample success with --latest false:

jenkins-plugin-cli --no-download --verbose --latest false --plugins active-directory:2.20 apache-httpcomponents-client-4-api:4.5.10-1.0 aws-credentials:1.23 aws-java-sdk:1.11.341 aws-parameter-store:1.2.2 configuration-as-code:1.27 configuration-as-code-secret-ssm:1.0.1 credentials:2.3.5 credentials-binding:1.23 display-url-api:2.3.1 git:4.2.1 git-client:3.0.0 github:1.29.2 github-api:1.90 jackson2-api:2.7.3 job-dsl:1.72 jsch:0.1.55.1 ldap:2.2 mailer:1.32.1 matrix-auth:2.6.2 plain-credentials:1.6 role-strategy:3.1 scm-api:2.6.3 script-security:1.75 ssh-credentials:1.17.3 structs:1.20 timestamper:1.11.2 token-macro:2.8 workflow-api:2.39 workflow-scm-step:2.9 workflow-step-api:2.22
No .txt or .yaml file containing list of plugins to be downloaded entered.
No directory to download plugins entered. Will use default of /usr/share/jenkins/ref/plugins
Using update center https://updates.jenkins.io/update-center.json from JENKINS_UC environment variable
Using experimental update center https://updates.jenkins.io/experimental/update-center.json from JENKINS_UC_EXPERIMENTAL environment variable
Using incrementals mirror https://repo.jenkins-ci.org/incrementals from JENKINS_INCREMENTALS_REPO_MIRROR environment variable
No CLI option or environment variable set for plugin info, using default of https://updates.jenkins.io/plugin-versions.json
No war entered. Will use default of /usr/share/jenkins/jenkins.war
Jenkins version: 2.272

Retrieving update center information
Created cache at: /var/jenkins_home/.cache/jenkins-plugin-management-cli
Cache miss for: update-center-2.272
Cache miss for: experimental-update-center-2.272
Cache miss for: plugin-versions
Couldn't find checksum for active-directory at version: 2.20
Setting checksum for: active-directory to CGamy8Tm1yzx0TkE6/Sj+6UwPWqodup52RAGxP5MWjU=

active-directory depends on: 
mailer 1.5
Skipping dependency mailer:1.5 and its sub-dependencies, because there is a higher version defined on the top level - mailer:1.32.1
Couldn't find checksum for apache-httpcomponents-client-4-api at version: 4.5.10-1.0
Setting checksum for: apache-httpcomponents-client-4-api to kTaiJaAAbVqBISY2AWNVjkhhl6DbXKVW6sEDKxTYdNs=

apache-httpcomponents-client-4-api has no dependencies
Couldn't find checksum for aws-credentials at version: 1.23
Setting checksum for: aws-credentials to IP5LXBJzruZrKK21BMzQ+7C0TiQLhL3HTvejh0nucm4=

aws-credentials depends on: 
credentials 2.1.16
aws-java-sdk 1.10.16
credentials-binding 1.7
Skipping dependency credentials:2.1.16 and its sub-dependencies, because there is a higher version defined on the top level - credentials:2.3.5
Skipping dependency aws-java-sdk:1.10.16 and its sub-dependencies, because there is a higher version defined on the top level - aws-java-sdk:1.11.341
Skipping dependency credentials-binding:1.7 and its sub-dependencies, because there is a higher version defined on the top level - credentials-binding:1.23
Couldn't find checksum for aws-java-sdk at version: 1.11.341
Setting checksum for: aws-java-sdk to 6+fMp3VdwCDROO9Tenv3th47utQffw9r5ABJF8B2kjg=

aws-java-sdk depends on: 
apache-httpcomponents-client-4-api 4.5.3-2.0
jackson2-api 2.7.0
Skipping dependency apache-httpcomponents-client-4-api:4.5.3-2.0 and its sub-dependencies, because there is a higher version defined on the top level - apache-httpcomponents-client-4-api:4.5.10-1.0
Skipping dependency jackson2-api:2.7.0 and its sub-dependencies, because there is a higher version defined on the top level - jackson2-api:2.7.3
Setting checksum for: aws-parameter-store to bUDzx5vvdyQ4QYkW3C2UtO46UVR5v0yCWWUNrWFh71w=
Setting checksum for: aws-parameter-store to bUDzx5vvdyQ4QYkW3C2UtO46UVR5v0yCWWUNrWFh71w=

aws-parameter-store depends on: 
aws-credentials 1.23
aws-java-sdk 1.11.264
Skipping dependency aws-credentials:1.23 and its sub-dependencies, because there is a higher version defined on the top level - aws-credentials:1.23
Skipping dependency aws-java-sdk:1.11.264 and its sub-dependencies, because there is a higher version defined on the top level - aws-java-sdk:1.11.341
Setting checksum for: aws-parameter-store to bUDzx5vvdyQ4QYkW3C2UtO46UVR5v0yCWWUNrWFh71w=
Couldn't find checksum for configuration-as-code at version: 1.27
Setting checksum for: configuration-as-code to 3i717v6SHSrnZTGJ7RbUsLa+X8VMegbOCdNg4v0HpCo=

configuration-as-code has no dependencies
Setting checksum for: configuration-as-code-secret-ssm to uxb74m+8kzBnotLH0YjXi4uv8sBeR713izCG+ZKGGbA=
Setting checksum for: configuration-as-code-secret-ssm to uxb74m+8kzBnotLH0YjXi4uv8sBeR713izCG+ZKGGbA=

configuration-as-code-secret-ssm depends on: 
configuration-as-code 1.2
aws-java-sdk 1.11.341
Skipping dependency configuration-as-code:1.2 and its sub-dependencies, because there is a higher version defined on the top level - configuration-as-code:1.27
Skipping dependency aws-java-sdk:1.11.341 and its sub-dependencies, because there is a higher version defined on the top level - aws-java-sdk:1.11.341
Setting checksum for: configuration-as-code-secret-ssm to uxb74m+8kzBnotLH0YjXi4uv8sBeR713izCG+ZKGGbA=
Couldn't find checksum for credentials at version: 2.3.5
Setting checksum for: credentials to tgVI8cfMc/mk8TzxP7yzUeKUXhudHmow5qOym+CvO5Q=

credentials depends on: 
structs 1.20
Skipping dependency structs:1.20 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Couldn't find checksum for credentials-binding at version: 1.23
Setting checksum for: credentials-binding to 0tsPEoS2UEI3UUApcvSLUqkiisETySJxCNGeY9bXejo=

credentials-binding depends on: 
workflow-step-api 2.22
credentials 2.3.5
plain-credentials 1.6
ssh-credentials 1.17.3
structs 1.20
Skipping dependency workflow-step-api:2.22 and its sub-dependencies, because there is a higher version defined on the top level - workflow-step-api:2.22
Skipping dependency credentials:2.3.5 and its sub-dependencies, because there is a higher version defined on the top level - credentials:2.3.5
Skipping dependency plain-credentials:1.6 and its sub-dependencies, because there is a higher version defined on the top level - plain-credentials:1.6
Skipping dependency ssh-credentials:1.17.3 and its sub-dependencies, because there is a higher version defined on the top level - ssh-credentials:1.17.3
Skipping dependency structs:1.20 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Couldn't find checksum for display-url-api at version: 2.3.1
Setting checksum for: display-url-api to FIjOqVuLDR7dgw56pJRYvubMsIGRcZlq/ykbE+m5mjw=

display-url-api has no dependencies
Couldn't find checksum for git at version: 4.2.1
Setting checksum for: git to Xc9x7ta81coOYHFXmic4C3szqD33QMG9VkmS25bXklA=

git depends on: 
workflow-scm-step 2.9
workflow-step-api 2.20
credentials 2.3.0
git-client 3.0.0
mailer 1.23
scm-api 2.6.3
script-security 1.66
ssh-credentials 1.17.3
structs 1.20
Skipping dependency workflow-scm-step:2.9 and its sub-dependencies, because there is a higher version defined on the top level - workflow-scm-step:2.9
Skipping dependency workflow-step-api:2.20 and its sub-dependencies, because there is a higher version defined on the top level - workflow-step-api:2.22
Skipping dependency credentials:2.3.0 and its sub-dependencies, because there is a higher version defined on the top level - credentials:2.3.5
Skipping dependency git-client:3.0.0 and its sub-dependencies, because there is a higher version defined on the top level - git-client:3.0.0
Skipping dependency mailer:1.23 and its sub-dependencies, because there is a higher version defined on the top level - mailer:1.32.1
Skipping dependency scm-api:2.6.3 and its sub-dependencies, because there is a higher version defined on the top level - scm-api:2.6.3
Skipping dependency script-security:1.66 and its sub-dependencies, because there is a higher version defined on the top level - script-security:1.75
Skipping dependency ssh-credentials:1.17.3 and its sub-dependencies, because there is a higher version defined on the top level - ssh-credentials:1.17.3
Skipping dependency structs:1.20 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Couldn't find checksum for git-client at version: 3.0.0
Setting checksum for: git-client to DKFkrtQQCZDhroGQH2Ld7hBc9+vP7AOpcVErPjZgRsA=

git-client depends on: 
apache-httpcomponents-client-4-api 4.5.10-1.0
credentials 2.3.0
jsch 0.1.55.1
ssh-credentials 1.17.2
structs 1.20
Skipping dependency apache-httpcomponents-client-4-api:4.5.10-1.0 and its sub-dependencies, because there is a higher version defined on the top level - apache-httpcomponents-client-4-api:4.5.10-1.0
Skipping dependency credentials:2.3.0 and its sub-dependencies, because there is a higher version defined on the top level - credentials:2.3.5
Skipping dependency jsch:0.1.55.1 and its sub-dependencies, because there is a higher version defined on the top level - jsch:0.1.55.1
Skipping dependency ssh-credentials:1.17.2 and its sub-dependencies, because there is a higher version defined on the top level - ssh-credentials:1.17.3
Skipping dependency structs:1.20 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Couldn't find checksum for github at version: 1.29.2
Setting checksum for: github to P0s/MQxAXzTNZyxZ2NxuRTx4gTgPy0yywZlKbg5z4SE=

github depends on: 
credentials 2.1.8
display-url-api 2.0
git 3.4.0
github-api 1.90
plain-credentials 1.1
scm-api 2.2.0
structs 1.10
token-macro 1.11
Skipping dependency credentials:2.1.8 and its sub-dependencies, because there is a higher version defined on the top level - credentials:2.3.5
Skipping dependency display-url-api:2.0 and its sub-dependencies, because there is a higher version defined on the top level - display-url-api:2.3.1
Skipping dependency git:3.4.0 and its sub-dependencies, because there is a higher version defined on the top level - git:4.2.1
Skipping dependency github-api:1.90 and its sub-dependencies, because there is a higher version defined on the top level - github-api:1.90
Skipping dependency plain-credentials:1.1 and its sub-dependencies, because there is a higher version defined on the top level - plain-credentials:1.6
Skipping dependency scm-api:2.2.0 and its sub-dependencies, because there is a higher version defined on the top level - scm-api:2.6.3
Skipping dependency structs:1.10 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Skipping dependency token-macro:1.11 and its sub-dependencies, because there is a higher version defined on the top level - token-macro:2.8
Couldn't find checksum for github-api at version: 1.90
Setting checksum for: github-api to jsEMmLTMED2Cbi4qEBtL9Es7kbOVwz3nRIADQxunNs4=

github-api depends on: 
jackson2-api 2.7.3
Skipping dependency jackson2-api:2.7.3 and its sub-dependencies, because there is a higher version defined on the top level - jackson2-api:2.7.3
Couldn't find checksum for jackson2-api at version: 2.7.3
Setting checksum for: jackson2-api to bfobo9syrkdjCyOTmcP8jZEC/wtSOlwz3TK8iNmHSDc=

jackson2-api has no dependencies
Couldn't find checksum for job-dsl at version: 1.72
Setting checksum for: job-dsl to UaAZG3Ovv1xl+AZAILiRdkSmiZ+qKW6gfi3xjP9niTQ=

job-dsl depends on: 
structs 1.13
script-security 1.54
Skipping dependency structs:1.13 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Skipping dependency script-security:1.54 and its sub-dependencies, because there is a higher version defined on the top level - script-security:1.75
Couldn't find checksum for jsch at version: 0.1.55.1
Setting checksum for: jsch to k0FRYA3UKwOY6IH7unBlbDj9am8SqHTSP8jcNeNxxNQ=

jsch depends on: 
ssh-credentials 1.14
Skipping dependency ssh-credentials:1.14 and its sub-dependencies, because there is a higher version defined on the top level - ssh-credentials:1.17.3
Setting checksum for: ldap to waxLrSaPc/LvbBgjbgg11WeD2jMXJ3Rg/TGBKxXECvs=
Setting checksum for: ldap to waxLrSaPc/LvbBgjbgg11WeD2jMXJ3Rg/TGBKxXECvs=

ldap depends on: 
mailer 1.32.1
Skipping dependency mailer:1.32.1 and its sub-dependencies, because there is a higher version defined on the top level - mailer:1.32.1
Setting checksum for: ldap to waxLrSaPc/LvbBgjbgg11WeD2jMXJ3Rg/TGBKxXECvs=
Setting checksum for: mailer to CwReqH78KJ/MrVWsUeTJA5RdloO3G0hmfhlEDkKi02g=
Setting checksum for: mailer to CwReqH78KJ/MrVWsUeTJA5RdloO3G0hmfhlEDkKi02g=

mailer depends on: 
display-url-api 2.3.1
Skipping dependency display-url-api:2.3.1 and its sub-dependencies, because there is a higher version defined on the top level - display-url-api:2.3.1
Setting checksum for: mailer to CwReqH78KJ/MrVWsUeTJA5RdloO3G0hmfhlEDkKi02g=
Couldn't find checksum for matrix-auth at version: 2.6.2
Setting checksum for: matrix-auth to ZnV+9pzC4a45xwovy9WZluON6phRi0o5Xf5CZH1S3ls=

matrix-auth has no dependencies
Couldn't find checksum for plain-credentials at version: 1.6
Setting checksum for: plain-credentials to JrwSClZ0r6VZJSfjPF/LCC9BTV9MO13fsvRertn4MHM=

plain-credentials depends on: 
credentials 2.1.16
Skipping dependency credentials:2.1.16 and its sub-dependencies, because there is a higher version defined on the top level - credentials:2.3.5
Setting checksum for: role-strategy to xf/+FgDzUkb3pdruf51USfQUI1dA7pM+x7iRImujFzQ=
Setting checksum for: role-strategy to xf/+FgDzUkb3pdruf51USfQUI1dA7pM+x7iRImujFzQ=

role-strategy depends on: 
matrix-auth 2.2
Skipping dependency matrix-auth:2.2 and its sub-dependencies, because there is a higher version defined on the top level - matrix-auth:2.6.2
Setting checksum for: role-strategy to xf/+FgDzUkb3pdruf51USfQUI1dA7pM+x7iRImujFzQ=
Couldn't find checksum for scm-api at version: 2.6.3
Setting checksum for: scm-api to gyYtQGhirVX/kNNgIKg/iOvXHmb0nHXJpRQPQxdquik=

scm-api depends on: 
structs 1.9
Skipping dependency structs:1.9 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Setting checksum for: script-security to 46ntxSuoQmaVu6ShnDGuQcOk6WwgpH/PX+gOd4UZxLg=
Setting checksum for: script-security to 46ntxSuoQmaVu6ShnDGuQcOk6WwgpH/PX+gOd4UZxLg=

script-security has no dependencies
Setting checksum for: script-security to 46ntxSuoQmaVu6ShnDGuQcOk6WwgpH/PX+gOd4UZxLg=
Couldn't find checksum for ssh-credentials at version: 1.17.3
Setting checksum for: ssh-credentials to Y/5ezKJXbwSYzSg5I+0gxBYHMKLOjjUGZuapxpkjfKs=

ssh-credentials depends on: 
credentials 2.2.0
Skipping dependency credentials:2.2.0 and its sub-dependencies, because there is a higher version defined on the top level - credentials:2.3.5
Setting checksum for: structs to fnhhNWo3qmpydGLXrqcW3ZMHBxJS9zScJybWSnc/6zo=
Setting checksum for: structs to fnhhNWo3qmpydGLXrqcW3ZMHBxJS9zScJybWSnc/6zo=

structs has no dependencies
Setting checksum for: structs to fnhhNWo3qmpydGLXrqcW3ZMHBxJS9zScJybWSnc/6zo=
Couldn't find checksum for timestamper at version: 1.11.2
Setting checksum for: timestamper to Kp52J/R0R6hbWIVsnuBV4m4XhTONp5XN+SQHGYQr8xg=

timestamper depends on: 
workflow-api 2.39
workflow-step-api 2.19
Skipping dependency workflow-api:2.39 and its sub-dependencies, because there is a higher version defined on the top level - workflow-api:2.39
Skipping dependency workflow-step-api:2.19 and its sub-dependencies, because there is a higher version defined on the top level - workflow-step-api:2.22
Couldn't find checksum for token-macro at version: 2.8
Setting checksum for: token-macro to WwaKWNi7yRp0z/0TfqoZYC19jVzpAYzACtZ+M7fDKDw=

token-macro depends on: 
workflow-step-api 2.14
structs 1.14
Skipping dependency workflow-step-api:2.14 and its sub-dependencies, because there is a higher version defined on the top level - workflow-step-api:2.22
Skipping dependency structs:1.14 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Couldn't find checksum for workflow-api at version: 2.39
Setting checksum for: workflow-api to 7r0mqAtz8/K0G4++p4ncAVTDX9qzP9SGBRE6KYZ5wZI=

workflow-api depends on: 
workflow-step-api 2.16
scm-api 2.2.6
structs 1.17
Skipping dependency workflow-step-api:2.16 and its sub-dependencies, because there is a higher version defined on the top level - workflow-step-api:2.22
Skipping dependency scm-api:2.2.6 and its sub-dependencies, because there is a higher version defined on the top level - scm-api:2.6.3
Skipping dependency structs:1.17 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Couldn't find checksum for workflow-scm-step at version: 2.9
Setting checksum for: workflow-scm-step to Y7EF7XduJp9pFQUiGNXNcx2mDKviKRu/0HGOpOOEuYE=

workflow-scm-step depends on: 
workflow-step-api 2.9
Skipping dependency workflow-step-api:2.9 and its sub-dependencies, because there is a higher version defined on the top level - workflow-step-api:2.22
Couldn't find checksum for workflow-step-api at version: 2.22
Setting checksum for: workflow-step-api to 65xkoZQdOvMgRRJn4kj+Og/50jboDD9Xbx06Zm1QKjM=

workflow-step-api depends on: 
structs 1.20
Skipping dependency structs:1.20 and its sub-dependencies, because there is a higher version defined on the top level - structs:1.20
Done

baptistemesta commented 3 years ago

I encountered the same issue. I thought at first that some plugin did really require a higher version of one of my plugin but I then discovered that it was due to the "latest" option.

We update our Jenkins through a versioned/automated build. Given that, we fixed the version of plugins for reproducibility concerns.

So it was a little confusing to have that behavior by default for plugins we explicitly set the version.

MarkEWaite commented 3 years ago

I was also confused by the message that was displayed when I had enumerated each plugin version explicitly. I ran the command:

$ rm -rf plugins
$ java -jar ../jenkins-plugin-manager-2.9.0.jar \
        --war ../jenkins-2.277.1.war \
        --plugins "git:4.6.0 git-client:3.6.0" \
        --plugin-download-directory plugins
Plugin git:4.6.0 depends on git-client:3.7.0, but there is an older version defined on the top level - git-client:3.6.0,

I was confused that the message says "Plugin git:4.6.0 depends on git-client:3.7.0" when I know that git plugin 3.6.0 depends on git client plugin 3.6.0, not git client plugin 3.7.0. In my case, it would have helped if the phrasing of the message made it clear that the explicitly declared dependency on git-client:3.6.0 was being upgraded to git-client:3.7.0 because I had not provided the command line argument --latest false

I had the desired results when I added --latest false to the command line like this:

$ rm -rf plugins
$ java -jar ../jenkins-plugin-manager-2.9.0.jar \
        --war ../jenkins-2.277.1.war \
        --plugins "git:4.6.0 git-client:3.6.0" \
        --plugin-download-directory plugins \
        --latest false

Would it be enough to rework the user error message to say:

Plugin git:4.6.0 depends on git-client.  Using git-client:3.7.0 because --latest is true, but there is an older version defined on the top level - git-client:3.6.0,

timja commented 3 years ago

https://github.com/jenkinsci/plugin-installation-manager-tool/issues/308#issuecomment-792672010

no I think it should allow you to use that version, it should only selecting latest of transitive dependencies if it's defined on the top level AND is compatible then it should select what you have defined

pwillis-els commented 3 years ago

Since multiple people have experienced this bug (i've talked to a few more), can we come up with some possible fixes and have one implemented?

Don't change anything, but clearly mark out the new behavior. The behavior here is clearly different than the old Jenkins plugin install scripts, so I think this should be explained prominently in the docs for this tool. This is basically a breaking behavior in certain circumstances (when you want to use pinned versions and not the latest ones).
Change the plugin manager's default to be --latest false, and instead add a new default option --minimum-valid true (or another name). This option would use the latest minimum version number to satisfy all dependency requirements, when a version was not explicitly pinned. Thus the user can choose which versions of what plugin to pin, and the tool will fill in any gaps with the minimum required functionality. If the user needs the latest of everything, they can use --latest true.
Modify the behavior of --latest true to respect pinned versions and not try to upgrade a plugin version if the user has already pinned a different version. (That is to say, when resolving dependencies, if a pinned version forces a dependency to use a version other than latest, and nothing is pinned for that dependency, then do not use latest, but instead the minimum resolvable version that does not override the dependency trees of any pinned plugins. This would basically be more like --maximum-valid true, and the plugin manager may have to do more work to discover the latest versions of dependencies that don't conflict with pinned dependencies' graphs)

Thoughts?

MarkEWaite commented 3 years ago

I prefer option 1, describe the behavior more precisely so that users understand that they have a choice with plugin installation manager that was not available with the plugin installation shell scripts.

In those cases where the precise versions of all plugins are specified, the --latest false argument provides the desired result.

timja commented 3 years ago

A limitation with option 1 is that when a new dependency is added to a plugin it won’t get updated until someone notices that it’s missing from their plugin list

I still think 3 is the most sane option...

(I don’t have a lot of time for this repo and to get something fixed the quickest way is to send a PR)

gpaciga commented 3 years ago

As someone who just spent yesterday trying to understand this, option 3 is what I expected the behaviour to be before I learned that --latest was required to have pinned versions respected. Given that I can set myplugin:latest if I wanted to always use the latest, it does not make sense that myplugin:1.2.3 also effectively means latest by default.

My expectation is:

if a dependency is not listed, use latest
if a dependency is listed, use whatever is pinned
if a pinned version is lower than something another plugin requires, throw an error and let me sort it out. ("pluginX requires pluginY version B but you have specified version C")

That last point is similar to what currently happens, except the error would be because I pinned something incompatible, not because the tool chose to ignore my pinned versions.

The only case where I would want to pin something meaning "version X or higher" without meaning "latest" is if I want to accept patch or minor updates only (e.g. like npm where pinning ~1.2.3 means 1.2.4 is acceptable but 1.3 is not).

pwillis-els commented 3 years ago

Until someone can submit a PR to provide solution 3, I have opened the following PR for solution 1. https://github.com/jenkinsci/plugin-installation-manager-tool/pull/325 @timja can we get this merged in?

There should really be a better long-term solution, but I think a stop-gap is warranted so users can figure out their issue quickly.

timja commented 3 years ago

above was merged

haminhcong commented 3 years ago

I encountered similiar isssues. Today I need build a custom Jenkins Docker Image from base Docker Image jenkins/jenkins:2.289.1-jdk11, with jenkins-plugin-cli tool to install some Jenkin plugins to Jenkins Docker Image:

installPlugins:
   - kubernetes:1.29.4
   - workflow-aggregator:2.6
   - git:4.7.1
   - configuration-as-code:1.51
   - kubernetes-client-api:4.13.2-1

If I use option --latest true, then Dockerfile will like that:

FROM jenkins/jenkins:2.289.1-jdk11
RUN jenkins-plugin-cli --verbose --latest true --plugins \
        kubernetes-client-api:4.13.2-1 kubernetes:1.29.4 workflow-aggregator:2.6 git:4.7.1 configuration-as-code:1.51

then with the same behavior with PR: https://github.com/jenkinsci/plugin-installation-manager-tool/pull/325/files the Docker build process failed, due to I pinned kubernetes-client-api: plugin version to 4.13.2-1

If I use option --latest false, then Dockerfile will like that:

FROM jenkins/jenkins:2.289.1-jdk11
RUN ls -alh
RUN jenkins-plugin-cli --verbose --latest false --plugins \
         kubernetes-client-api:4.13.2-1 kubernetes:1.29.4 workflow-aggregator:2.6 git:4.7.1 configuration-as-code:1.51

Then Docker build process succeed. But when I inspect installed plugins versions in build log, It showed that some installed plugins versions is incorrect:

https://pastebin.com/zcnmBmB4

$ docker build -f Dockerfile .
Sending build context to Docker daemon  8.192kB
Step 1/4 : FROM jenkins/jenkins:2.289.1-jdk11
 ---> 416c6656c1cd
Step 2/4 : RUN ls -alh
 ---> Using cache
 ---> d80f607455a9
Step 3/4 : RUN jenkins-plugin-cli --verbose --plugins kubernetes-client-api:4.13.2-1 kubernetes:1.29.4 workflow-aggregator:2.6 git:4.7.1 configuration-as-code:1.51 --latest false
 ---> Running in 560577e701c3
No .txt or .yaml file containing list of plugins to be downloaded entered.

Will install new plugin kubernetes 1.29.4
Will install new plugin pipeline-milestone-step 1.3.1
Will install new plugin snakeyaml-api 1.27.0
Will install new plugin trilead-api 1.0.13
Will install new plugin workflow-multibranch 2.20
Will install new plugin docker-workflow 1.14
Will install new plugin handlebars 1.1
Will install new plugin pipeline-stage-tags-metadata 1.3.2
Will install new plugin git-client 3.7.1
Will install new plugin variant 1.4
Will install new plugin scm-api 2.6.4
Will install new plugin kubernetes-client-api 4.13.2-1
Will install new plugin pipeline-stage-step 2.3
Will install new plugin pipeline-model-extensions 1.7.2
Will install new plugin configuration-as-code 1.51
Will install new plugin workflow-cps-global-lib 2.11
Will install new plugin workflow-support 2.20
Will install new plugin mailer 1.32.1
Will install new plugin jackson2-api 2.12.1
Will install new plugin pipeline-rest-api 2.10
Will install new plugin workflow-scm-step 2.12
Will install new plugin pipeline-model-api 1.3.2
Will install new plugin lockable-resources 2.3
Will install new plugin pipeline-input-step 2.8
Will install new plugin metrics 4.0.2.6
Will install new plugin momentjs 1.1
Will install new plugin git-server 1.7
Will install new plugin credentials-binding 1.13
Will install new plugin workflow-aggregator 2.6
Will install new plugin kubernetes-credentials 0.8.0
Will install new plugin pipeline-stage-view 2.10
Will install new plugin pipeline-build-step 2.7
Will install new plugin credentials 2.3.15
Will install new plugin pipeline-model-definition 1.3.2
Will install new plugin caffeine-api 2.9.1-23.v51c4e2c879c8
Will install new plugin workflow-step-api 2.23
Will install new plugin plain-credentials 1.7
Will install new plugin matrix-project 1.18
Will install new plugin bouncycastle-api 2.18
Will install new plugin docker-commons 1.14
Will install new plugin jquery-detached 1.2.1
Will install new plugin ace-editor 1.0.1
Will install new plugin git 4.7.1
Will install new plugin icon-shim 1.0.3
Will install new plugin authentication-tokens 1.4
Will install new plugin workflow-durable-task-step 2.22
Will install new plugin ssh-credentials 1.18.1
Will install new plugin cloudbees-folder 6.15
Will install new plugin durable-task 1.35
Will install new plugin workflow-basic-steps 2.11
Will install new plugin workflow-job 2.25
Will install new plugin jsch 0.1.55.2
Will install new plugin pipeline-graph-analysis 1.1
Will install new plugin junit 1.3
Will install new plugin structs 1.22
Will install new plugin apache-httpcomponents-client-4-api 4.5.13-1.0
Will install new plugin workflow-cps 2.89
Will install new plugin display-url-api 2.3.1
Will install new plugin script-security 1.76
Will install new plugin pipeline-model-declarative-agent 1.1.1
Will install new plugin branch-api 2.0.18
Will install new plugin workflow-api 2.41

In above plugins versions, workflow-cps 2.89 depends on: https://github.com/jenkinsci/workflow-cps-plugin/blob/1768bff98453bfd94e889874dfe8eea0eef1a4cb/pom.xml#L72

Plugin-Dependencies: 
...
workflow-api:2.41
workflow-scm-step:2.11
workflow-step-api:2.23
workflow-support:3.7
...

and workflow-aggregator:2.6 depends on: https://github.com/jenkinsci/workflow-aggregator-plugin/blob/b28dc6c72545a28561ffee43188b983d72594ec7/pom.xml#L81

Plugin-Dependencies: 
...
workflow-multibranch 2.20
workflow-scm-step 2.6
workflow-step-api 2.16
workflow-support 2.20
...

If plugin version resolve correctly, workflow-support plugin must be resolved to at least version 3.7. But with option --latest false, jenkins-plugin-cli resloved this plugin to version 2.20 as build log.

Will install new plugin workflow-support 2.20

due to this behavior, when I start Docker Image after build, Jenkins cannot started successfully:

2021-06-19 13:03:00.072+0000 [id=28]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Groovy v2.89 (workflow-cps)
java.io.IOException: Failed to load: Pipeline: Groovy (2.89)
 - Update required: JavaScript GUI Lib: ACE Editor bundle plugin (1.0.1) to be updated to 1.1 or higher
 - Update required: Pipeline: Supporting APIs (2.20) to be updated to 3.7 or higher
    at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:963)
    at hudson.PluginManager$2$1$1.run(PluginManager.java:550)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Compare with install-plugins.sh, workflow-support plugins version is resolved to compatible version with all depended plugins. With following plugins.txt file and Dockerfile:

kubernetes-client-api:4.13.2-1 
kubernetes:1.29.4 
workflow-aggregator:2.6 
git:4.7.1 
configuration-as-code:1.51

FROM jenkins/jenkins:2.289.1-jdk11
RUN ls -alh /usr/share/jenkins/ref/plugins
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt

When build image process run, plugins versions is resolved like that:

$ docker build --no-cache -f Dockerfile .
Sending build context to Docker daemon   7.68kB
Step 1/4 : FROM jenkins/jenkins:2.289.1-jdk11
 ---> 416c6656c1cd
Step 2/4 : COPY plugins.txt /usr/share/jenkins/plugins.txt
 ---> 852bd8a627ad
Step 3/4 : RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
 ---> Running in 2ecf28e3df3f
WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli
Creating initial locks...
Analyzing war /usr/share/jenkins/jenkins.war...
#...

WAR bundled plugins:

Installed plugins:
ace-editor:1.1
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
bootstrap4-api:4.6.0-3
bootstrap5-api:5.0.1-2
bouncycastle-api:2.20
branch-api:2.6.4
caffeine-api:2.9.1-23.v51c4e2c879c8
checks-api:1.7.0
cloudbees-folder:6.15
configuration-as-code:1.51
credentials-binding:1.25
credentials:2.5
display-url-api:2.3.5
durable-task:1.37
echarts-api:5.1.2-2
font-awesome-api:5.15.3-3
git-client:3.7.2
git:4.7.1
git-server:1.9
handlebars:3.0.8
jackson2-api:2.12.3
jquery3-api:3.6.0-1
jsch:0.1.55.2
junit:1.50
kubernetes-client-api:4.13.2-1
kubernetes-credentials:0.8.0
kubernetes:1.29.4
lockable-resources:2.11
mailer:1.34
matrix-project:1.19
metrics:4.0.2.8
momentjs:1.1.1
pipeline-build-step:2.13
pipeline-graph-analysis:1.11
pipeline-input-step:2.12
pipeline-milestone-step:1.3.2
pipeline-model-api:1.8.5
pipeline-model-definition:1.8.5
pipeline-model-extensions:1.8.5
pipeline-rest-api:2.19
pipeline-stage-step:2.5
pipeline-stage-tags-metadata:1.8.5
pipeline-stage-view:2.19
plain-credentials:1.7
plugin-util-api:2.3.0
popper2-api:2.5.4-2
popper-api:1.16.1-2
scm-api:2.6.4
script-security:1.77
snakeyaml-api:1.29.1
ssh-credentials:1.19
structs:1.23
trilead-api:1.0.13
variant:1.4
workflow-aggregator:2.6
workflow-api:2.45
workflow-basic-steps:2.23
workflow-cps-global-lib:2.20
workflow-cps:2.92
workflow-durable-task-step:2.39
workflow-job:2.41
workflow-multibranch:2.26
workflow-scm-step:2.13
workflow-step-api:2.23
workflow-support:3.8
Cleaning up locks
Removing intermediate container 2ecf28e3df3f
 ---> 634042187e2f

An example, install-plugins.sh resloved workflow-support to version 3.8, which is a compatible version with both workflow-cps 2.89 and workflow-aggregator:2.6

mangitkaur commented 2 years ago

Hi , on marking latest as false, I am still getting dependencies issue, can you please help?

MarkEWaite commented 2 years ago

Hi , on marking latest as false, I am still getting dependencies issue, can you please help?

Best to open a separate issue with precise details that describe the problem you are seeing. Most of the problems that I've seen with dependencies have been due to users providing an incomplete list of plugins.

jenkinsci / plugin-installation-manager-tool

Bug (or "undocumented feature?") in dependency resolution using `--latest true` #250