search

renovatebot / config-help

Please use the Discussions feature of https://github.com/renovatebot/renovate instead
https://github.com/renovatebot/renovate/discussions
MIT License
27 stars 16 forks source link

Renovate not working for SBT private repositories #1021

Closed marius-carp closed 3 years ago

marius-carp commented 3 years ago

What Renovate type, platform and version are you using?

bitbucket, self-hosted

Describe the bug

Private repositories don't work with SBT. Looking into renovate logs I don't see it looking for the private repositories. I just see logs of public repositories.

Relevant debug logs

Dependency extraction complete (repository=sef/renovate-test-scala) "baseBranch": "master", "stats": { "managers": {"sbt": {"fileCount": 1, "depCount": 1}}, "total": {"fileCount": 1, "depCount": 1} }

Config

{
  "endpoint": "https://bitbucket.domain.tools",
  "platform": "bitbucket-server",
  "username": "username",
  "password": "password",
  "extends": [
    "config:base"
  ],
  "repositories": [
    "sef/renovate-test-scala"
  ],
  "hostRules": [
    {
      "hostType": "maven",
      "endpoint": "https://artifactory.domain.tools/artifactory/domain-release-sbt/",
      "username": "username",
      "password": "password"
    }
  ],
  "packageRules": [
    {
      "updateTypes": [
        "major",
        "minor",
        "patch"
      ],
      "managers": [
        "sbt"
      ],
      "depTypeList": [
        "dependencies",
        "devDependencies"
      ],
      "automerge": false,
      "registryUrls": [" https://artifactory.domain.tools/artifactory/domain-release-sbt/"]
    }
  ],
  "sbt": {
    "enabled": true
  },
  "enabledManagers": [
    "sbt"
  ]
}

but I don't see it looking for the private repository

To Reproduce

Additional context

I created a dummy maven project with a single private repository and it worked, it even opened a PR for the outdated private dependency. Is SBT private repository renovation working?

label-actions[bot] commented 3 years ago

This needs to be reproduced in a public repository in order to troubleshoot further. Without a reproduction, it won't be possible to answer further.

rarkins commented 3 years ago

Remove updateTypes and depTypeList from your first package rule and try again.

If that doesn't work, try to reproduce it as closely as possible in a public github repo (e.g. using the fake artifactory URL as above)

marius-carp commented 3 years ago

Thank you for the quick response.

I added a testing repository: stand-alone renovate app with the config sbt project

Removing updateTypes and depTypeList got me a step forward, I do see renovate looking for the private repositories, but I get unauthorized when trying to access the private artifactory. (it worked before with maven, having the same config in the hostRules). I also dropped the enabledManagers and only kept "sbt": { "enabled": true }

Now the config looks like this:

{
  "endpoint": "https://bitbucket.domain.tools",
  "platform": "bitbucket-server",
  "username": "username",
  "password": "password",
  "extends": [
    "config:base"
  ],
  "repositories": [
    "sef/renovate-test-scala"
  ],
  "hostRules": [
    {
      "hostType": "maven",
      "endpoint": "https://artifactory.domain.tools/artifactory/domain-release-sbt/",
      "username": "username",
      "password": "password"
    }
  ],
  "packageRules": [
    {
      "managers": [
        "sbt"
      ],
      "automerge": false,
      "registryUrls": [" https://artifactory.domain.tools/artifactory/domain-release-sbt/"]
    }
  ],
  "sbt": {
    "enabled": true
  }
}

debug message: 

Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala)
       "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"

I see it stange that is trying to look for the same path but the packages are split by a dot(.) and not by a slash(/) Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala) "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"

rarkins commented 3 years ago

This repo does not have the config you are talking about: https://github.com/marius-carp/renovate-test-scala/blob/main/renovate.json. We need a reproduction

marius-carp commented 3 years ago

The config is under https://github.com/marius-carp/tool-renovate/blob/main/config.json

rarkins commented 3 years ago

Please copy those rules across into the renovate.json of your reproduction repo. The aim is that someone can run something like renovate marius-carp/renovate-test-scala --dry-run and reproduce the lookup.

Can you also clarify what is the url you think Renovate should be looking up, versus what it is looking up? Maybe the problem is that the URL is wrong, but that triggers a 403 from your server.

marius-carp commented 3 years ago

I pushed a commit with the configs, please have a look to see if is what is supposed to be.

Where maven looks up for the private repositories: Looking up com.domain: lib-accounts_2.13 in repository "https://artifactory.domain.tools/artifactory/domain-release-sbt/"

Sbt error:

DEBUG: No versions found for com.domain:lib-accounts_2.13 in 2 repositories (repository=sef/renovate-test-scala)
DEBUG: Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala)
       "failedUrl": "https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"
DEBUG: Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala)
       "failedUrl": "https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"
rarkins commented 3 years ago

Please fix the reproduction repo so that a dry run can be run against it. It should just contain repository config options (e.g. hostRules, packageRules).

 INFO: Found renovate config errors (repository=marius-carp/renovate-test-scala)
       "errors": [
         {
           "depName": "Configuration Error",
           "message": "Invalid configuration option: user"
         }
       ]
 INFO: Repository has invalid config (repository=marius-carp/renovate-test-scala)
       "error": {
         "configFile": "renovate.json",
         "validationError": "The renovate configuration file contains some invalid settings",
         "validationMessage": "Invalid configuration option: user",
         "message": "config-validation",
         "stack": "Error: config-validation\n    at mergeRenovateConfig (/Users/rhys/src/renovate/lib/workers/repository/init/config.ts:164:19)\n    at async getRepoConfig (/Users/rhys/src/renovate/lib/workers/repository/init/config.ts:238:12)\n    at async initRepo (/Users/rhys/src/renovate/lib/workers/repository/init/index.ts:21:12)\n    at async Object.renovateRepository (/Users/rhys/src/renovate/lib/workers/repository/index.ts:37:14)\n    at async Object.start (/Users/rhys/src/renovate/lib/workers/global/index.ts:67:7)\n    at async /Users/rhys/src/renovate/lib/renovate.ts:10:22"
       }
rarkins commented 3 years ago

To clarify: we want to work out if it's a wrong URL problem, or a wrong authentication problem.

marius-carp commented 3 years ago

Sorry for that, I should have rechecked the config before pushing it. I fixed it now. It looks like the authentication to the private artifactory is not working in sbt.

rarkins commented 3 years ago

There are three dependencies extracted whereas ideally we'd have 1 for a minimal reproduction. Here's the debug logs:

 INFO: Dependency extraction complete (repository=marius-carp/renovate-test-scala)
       "baseBranch": "main",
       "stats": {
         "managers": {"sbt": {"fileCount": 1, "depCount": 3}},
         "total": {"fileCount": 1, "depCount": 3}
       }
DEBUG: Looking up org.scala-lang:scala-library in repository  https://artifactory.domain.tools/artifactory/domain-release-sbt/ (repository=marius-carp/renovate-test-scala)
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
       "failedUrl": "https://artifactory.domain.tools/artifactory/domain-release-sbt/org/scala-lang/scala-library/maven-metadata.xml"
DEBUG: Content is not found for Maven url: https://artifactory.domain.tools/artifactory/domain-release-sbt/org/scala-lang/scala-library/maven-metadata.xml (repository=marius-carp/renovate-test-scala)
DEBUG: Failed to look up dependency scala (repository=marius-carp/renovate-test-scala, packageFile=build.sbt, dependency=scala)
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
       "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
       "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/typesafe/akka/"
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
       "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"
DEBUG: No versions found for com.domain:lib-accounts_2.13 in 2 repositories (repository=marius-carp/renovate-test-scala)
DEBUG: Failed to look up dependency com.domain:lib-accounts (repository=marius-carp/renovate-test-scala, packageFile=build.sbt, dependency=com.domain:lib-accounts)
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
       "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.typesafe.akka/"
DEBUG: No versions found for com.typesafe.akka:akka-actor_2.13 in 2 repositories (repository=marius-carp/renovate-test-scala)
DEBUG: Failed to look up dependency com.typesafe.akka:akka-actor (repository=marius-carp/renovate-test-scala, packageFile=build.sbt, dependency=com.typesafe.akka:akka-actor)
DEBUG: Package releases lookups complete (repository=marius-carp/renovate-test-scala)
       "baseBranch": "main"
DEBUG: packageFiles with updates (repository=marius-carp/renovate-test-scala)
       "config": {
         "sbt": [
           {
             "packageFile": "build.sbt",
             "manager": "sbt",
             "deps": [
               {
                 "registryUrls": [
                   "https://repo.maven.apache.org/maven2",
                   "https://artifactory.domain.tools/artifactory/domain-release-sbt"
                 ],
                 "datasource": "maven",
                 "depName": "scala",
                 "lookupName": "org.scala-lang:scala-library",
                 "currentValue": "2.13.3",
                 "separateMinorPatch": true,
                 "depIndex": 0,
                 "updates": [],
                 "warnings": [
                   {
                     "depName": "scala",
                     "message": "Failed to look up dependency scala"
                   }
                 ],
                 "fixedVersion": "2.13.3"
               },
               {
                 "registryUrls": [
                   "https://repo.maven.apache.org/maven2",
                   "https://artifactory.domain.tools/artifactory/domain-release-sbt"
                 ],
                 "depName": "com.domain:lib-accounts",
                 "lookupName": "com.domain:lib-accounts_2.13",
                 "currentValue": "15.0.0",
                 "datasource": "sbt-package",
                 "depIndex": 1,
                 "updates": [],
                 "warnings": [
                   {
                     "depName": "com.domain:lib-accounts",
                     "message": "Failed to look up dependency com.domain:lib-accounts"
                   }
                 ],
                 "fixedVersion": "15.0.0"
               },
               {
                 "registryUrls": [
                   "https://repo.maven.apache.org/maven2",
                   "https://artifactory.domain.tools/artifactory/domain-release-sbt"
                 ],
                 "depName": "com.typesafe.akka:akka-actor",
                 "lookupName": "com.typesafe.akka:akka-actor_2.13",
                 "currentValue": "2.6.9",
                 "datasource": "sbt-package",
                 "depIndex": 2,
                 "updates": [],
                 "warnings": [
                   {
                     "depName": "com.typesafe.akka:akka-actor",
                     "message": "Failed to look up dependency com.typesafe.akka:akka-actor"
                   }
                 ],
                 "fixedVersion": "2.6.9"
               }
             ]
           }
         ]
       }

Which of those 3 should we focus on?

marius-carp commented 3 years ago

This one:

{
  "registryUrls": [
    "https://repo.maven.apache.org/maven2",
    "https://artifactory.domain.tools/artifactory/domain-release-sbt"
  ],
  "depName": "com.domain:lib-accounts",
  "lookupName": "com.domain:lib-accounts_2.13",
  "currentValue": "15.0.0",
  "datasource": "sbt-package",
  "depIndex": 1,
  "updates": [],
  "warnings": [
    {
      "depName": "com.domain:lib-accounts",
      "message": "Failed to look up dependency com.domain:lib-accounts"
    }
  ],
  "fixedVersion": "15.0.0"
}
rarkins commented 3 years ago

Is it possible to add the same dependency but for maven package manager (pom.xml) so we can compare the URLs?

rarkins commented 3 years ago

It looks like we tried two URLs:

DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
       "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
       "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"

In this case the connection refused is expected because this is a dummy URL, but in your real case we'd hope one of those URLs are correct and return a response

marius-carp commented 3 years ago

I have updated the code to point to a private dummy artifactory. I added the username and password in the config. You can check now the debug logs. Those are the errors that I see.

rarkins commented 3 years ago

I just noticed one thing that needs changing: remove the hostType setting from your host rule. The scala packages have "datasource": "sbt-package" which means they'll be looked up with hostType=sbt-package and not maven.

marius-carp commented 3 years ago

Ok, removing hostType from hostRules did the job, it works on github. But going back to bitbucket is still not working. I get the same "Dependency lookup unauthorized. Please add authentication with a hostRule", is there something else that I need to add to make it work on bitbucket.

rarkins commented 3 years ago

The core logic of sbt, datasources, and host or package rules are all common between platforms. Any difference is likely because of how you're configuring it (bot vs repository, etc).

marius-carp commented 3 years ago

It worked, removing managers from under packageRules did it. The final config version looks like this:

{
  "endpoint": "https://bitbucket.domain.tools",
  "platform": "bitbucket-server",
  "username": "username",
  "password": "password",
  "extends": [
    "config:base"
  ],
  "repositories": [
    "sef/renovate-test-scala"
  ],
  "hostRules": [
    {
      "endpoint": "https://artifactory.domain.tools/artifactory/domain-release-sbt/",
      "username": "username",
      "password": "password"
    }
  ],
  "packageRules": [
    {
      "automerge": false,
      "registryUrls": [" https://artifactory.domain.tools/artifactory/domain-release-sbt/"]
    }
  ],
  "sbt": {
    "enabled": true
  }
}

Thank you a a lot @rarkins for your patience and help. You are a really nice guy! I'll write a blog post on how to setup renovate for sbt for private repos on seflhosted artifactory.

rarkins commented 3 years ago

Sounds great! Congratulations on getting it up and running