search

renovatebot / config-help

Please use the Discussions feature of https://github.com/renovatebot/renovate instead
https://github.com/renovatebot/renovate/discussions
MIT License
27 stars 16 forks source link

DevOps example not working correctly #971

Closed FrankHoogmans closed 4 years ago

FrankHoogmans commented 4 years ago

Which Renovate are you using?

Renovate Open Source CLI

Which platform are you using?

Azure DevOps

Have you checked the logs? Don't forget to include them if relevant

Every single repo gives the following error (redacted repo names):

       "renovateVersion": "23.74.1"
ERROR: Repository has unknown error (repository=*******)
       "err": {
         "statusCode": 401,
         "result": {
           "$id": "1",
           "innerException": null,
           "message": "TF400813: The user '' is not authorized to access this resource.",
           "typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
           "typeKey": "UnauthorizedRequestException",
           "errorCode": 0,
           "eventId": 3000
         },
         "message": "TF400813: The user '' is not authorized to access this resource.",
         "stack": "Error: TF400813: The user '' is not authorized to access this resource.\n    at RestClient.<anonymous> (/home/vsts/.npm/_npx/2374/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:202:31)\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/vsts/.npm/_npx/2374/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:6:58)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
       }
 INFO: Repository finished (repository=*******)
       "durationMs": 402

What would you like to do?

We are using Renovate in an Azure DevOps pipeline, based on the starter template available at https://docs.renovatebot.com/setup-azure-devops/. This runs multiple times a day, to update dependencies for all of our modules (repositories).

Last friday (6th of november) everything worked great, but yesterday I started getting 401's on the DevOps repository access. I have not changed the configuration in any way, and even tried the template in another devops account to be sure it wasn't account related. Downgrading to the version that last worked did also not provide a working solution (same errors).

I have checked, and do not have any limitations on the System.AccessToken about using it in scripts or across repositories.

I am unsure if this is somehow a bug in renovate, or if i'm doing anything wrong with the configuration, but given that the example doesn't work for me anymore, gives me reason to think something changed, but I seem to be unable to figure out what.

rarkins commented 4 years ago

@JamieMagee do you think this could be related to the transitive dependency updates yesterday?

I also recall previously problems of expired tokens but this error message doesn't warn about that.

FrankHoogmans commented 4 years ago

I also recall previously problems of expired tokens but this error message doesn't warn about that.

I have already tried swapping tokens (switching to a PAT for my own account instead of System.AccessToken). Creating a new pipeline with the same yaml, and trying the template on a different devops account. So expired tokens don't sound likely.

JamieMagee commented 4 years ago

@JamieMagee do you think this could be related to the transitive dependency updates yesterday?

I don't think so. The lockfile updates went out yesterday, but this issue started happening on Friday.

I am also seeing this on my own Renovate jobs running against Azure DevOps (I should check in on those more often!). They started failing on Friday between 19:00 and 20:00 CET, but the Renovate version was 23.73.2 for both. It's still failing right now.

This might be something on the Azure DevOps side. Let me investigate it.

FrankHoogmans commented 4 years ago

@JamieMagee do you think this could be related to the transitive dependency updates yesterday?

I don't think so. The lockfile updates went out yesterday, but this issue started happening on Friday.

I am also seeing this on my own Renovate jobs running against Azure DevOps (I should check in on those more often!). They started failing on Friday between 19:00 and 20:00 CET, but the Renovate version was 23.73.2 for both. It's still failing right now.

This might be something on the Azure DevOps side. Let me investigate it.

I also did some more digging, but have not found anything so far. I'll just share what I checked.

The Azure DevOps release notes seem to indicate nothing has changed on Friday the 6th of November: https://docs.microsoft.com/en-us/azure/devops/release-notes/2020/sprint-177-update Last release of anything seems to have been October 21st.

I checked the run logs between a run that worked on friday, and one that failed on monday. Both runs are using exactly the same software on the build agent (ubuntu image, devops agent software, bash and npmauthenticate task versions). This is the 'Initialize job' logging:

Agent name: 'Hosted Agent'
Agent machine name: 'fv-az101-655'
Current agent version: '2.175.2'
Operating System
Ubuntu
18.04.5
LTS
Virtual Environment
Environment: ubuntu-18.04
Version: 20201102.0
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20201102.0/images/linux/Ubuntu1804-README.md
Current image version: '20201102.0'
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: Bash (3.177.2)
Downloading task: npmAuthenticate (0.174.0)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Finished checking job knob settings.
Start tracking orphan processes.
rarkins commented 4 years ago

So same version of Renovate and (apparent) same version of Azure DevOps, but it suddenly stopped working?

rarkins commented 4 years ago

Can somebody log at debug level in case it gives additional info? I'm guessing it's failing at one of these lines: https://github.com/renovatebot/renovate/blob/ad50398fb3981a7c5d04b96bddce3dbad45a7d21/lib/platform/azure/index.ts#L124-L125

rarkins commented 4 years ago

This question was coincidentally asked yesterday: https://stackoverflow.com/questions/64759524/tf400813-the-user-is-not-authorized-to-access-this-resource-when-using-syste

Someone asked the same question 4 months ago, here was some suggestion from a MSFT employee:

image

Note that the OP said that the problem resolved itself though.

FrankHoogmans commented 4 years ago

So same version of Renovate and (apparent) same version of Azure DevOps, but it suddenly stopped working?

Correct. Although at first the renovate version was different, but reverting doesn't fix it.

This question was coincidentally asked yesterday: https://stackoverflow.com/questions/64759524/tf400813-the-user-is-not-authorized-to-access-this-resource-when-using-syste

Someone asked the same question 4 months ago, here was some suggestion from a MSFT employee:

Note that the OP said that the problem resolved itself though.

I agree it seems to be a DevOps (authentication) issue... We use the Microsoft provided build agents and images, no proxy settings, and (I guess) 0 debugging on network issues there for us. I can't seem to find any other reports regarding these issues...

I still hope it will resolve itself automagically...

FrankHoogmans commented 4 years ago

Can somebody log at debug level in case it gives additional info? I'm guessing it's failing at one of these lines: https://github.com/renovatebot/renovate/blob/ad50398fb3981a7c5d04b96bddce3dbad45a7d21/lib/platform/azure/index.ts#L124-L125

Ok, a debug run seems to confirm that it fails somewhere after 'initRepo'. This is the debug-logging of a single repo (redacted repo name)

2020-11-11T10:12:20.2543975Z DEBUG: Using default gitAuthor: Renovate Bot <renovate@whitesourcesoftware.com>
2020-11-11T10:12:20.2544209Z DEBUG: Using baseDir: /tmp/renovate
2020-11-11T10:12:20.2544385Z DEBUG: Using cacheDir: /tmp/renovate/cache
2020-11-11T10:12:20.2544787Z DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
2020-11-11T10:12:20.2545019Z DEBUG: Commits limit = null
2020-11-11T10:12:20.2545204Z  INFO: Repository started (repository=***)
2020-11-11T10:12:20.2545405Z        "renovateVersion": "23.77.0"
2020-11-11T10:12:20.2545663Z DEBUG: Using localDir: /tmp/renovate/repos/azure/*** (repository=***)
2020-11-11T10:12:20.2545983Z DEBUG: initRepo("***") (repository=***)
2020-11-11T10:12:20.6909588Z ERROR: Repository has unknown error (repository=***)
2020-11-11T10:12:20.6909994Z        "err": {
2020-11-11T10:12:20.6910525Z          "statusCode": 401,
2020-11-11T10:12:20.6910718Z          "result": {
2020-11-11T10:12:20.6910898Z            "$id": "1",
2020-11-11T10:12:20.6911119Z            "innerException": null,
2020-11-11T10:12:20.6912043Z            "message": "TF400813: The user '' is not authorized to access this resource.",
2020-11-11T10:12:20.6912504Z            "typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
2020-11-11T10:12:20.6912984Z            "typeKey": "UnauthorizedRequestException",
2020-11-11T10:12:20.6913193Z            "errorCode": 0,
2020-11-11T10:12:20.6913365Z            "eventId": 3000
2020-11-11T10:12:20.6913509Z          },
2020-11-11T10:12:20.6913925Z          "message": "TF400813: The user '' is not authorized to access this resource.",
2020-11-11T10:12:20.6915288Z          "stack": "Error: TF400813: The user '' is not authorized to access this resource.\n    at RestClient.<anonymous> (/home/vsts/.npm/_npx/2588/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:202:31)\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/vsts/.npm/_npx/2588/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:6:58)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"
2020-11-11T10:12:20.6916019Z        }
2020-11-11T10:12:20.6919136Z DEBUG: Unknown res (repository=***)
2020-11-11T10:12:20.6919594Z        "res": "unknown-error"
2020-11-11T10:12:20.6931906Z DEBUG: Repository timing splits (milliseconds) (repository=***)
2020-11-11T10:12:20.6932204Z        "splits": {},
2020-11-11T10:12:20.6932363Z        "total": 463
2020-11-11T10:12:20.6936525Z  INFO: Repository finished (repository=***)
2020-11-11T10:12:20.6936775Z        "durationMs": 463
JamieMagee commented 4 years ago

I figured out the issue. It's a little bit convoluted but bear with me.

On Friday 6th November, version 1.8.0 of typed-rest-client was published. It included this commit about disabling authentication on redirects.

Looking at the changes on that commit, the authentication is not added to every request, so we're authenticating as an anonymous user for some requests (where have I seen that error 🤔)

azure-devops-node-api has a dependency on typed-rest-client@^1.7.3. So, even though Renovate specifies a different version in our yarn.lock file, because our guide use npx the yarn.lock isn't used. The ^1.7.3 range resolves to 1.8.0, which is currently broken.

Therefore, the fix for this should either be in azure-devops-node-api to set allowCrossOriginAuthentication, or in typed-rest-client to add authentication to all requests.

JamieMagee commented 4 years ago

I continued the discussion here

rarkins commented 4 years ago

@JamieMagee do we need to make any code modifications now?

JamieMagee commented 4 years ago

Yes, small code change required.

I can send a PR out later this evening, or tomorrow. I'm on mobile right now.

FrankHoogmans commented 4 years ago

I can confirm version 23.83.3 of renovatebot does work as expected again. Thank you so much for the help 👍