search

huserben / TfsExtensions

Extensions for TFS 2015+ such as custom Widgets (require TFS 2017) and Build Tasks
MIT License
45 stars 22 forks source link

Access denied. Project Collection Build Service (REDACTED) needs Use permissions for pool REDACTED #195

Closed buckleyGI closed 2 years ago

buckleyGI commented 2 years ago

As of yesterday we are seeing the below error without an effective change on our part.

We recreated the stage (it's a classic release) and the problem persists.

What is also a surprise is that our self hosted agent is using version 3.2.2 of the task even though the marketplace and our installation says that 3.1.1 is the latest. How can our agents be using an unreleased version?

We don't think this version mismatch is the root cause however as we saw it operating for a few days and even weeks without the below error message.

image

image

Using OAuth Access Token
Provided team project was guid.
Context is Release - using Release Environment Variables
Build shall be triggered for same user that triggered current Release: REDACTED
Source Version: REDACTED
Error message: Error: Access denied. Project Collection Build Service (REDACTED) needs Use permissions for pool REDACTED perform the action. For more information, contact the Azure DevOps Server administrator.
Will wait 1 seconds before retrying request...
Error during request (2/5)
Error message: Error: Access denied. Project Collection Build Service (REDACTED) needs Use permissions for pool Azure Pipelines to perform the action. For more information, contact the Azure DevOps Server administrator.
Will wait 2 seconds before retrying request...
Error during request (3/5)

We build up some good experience in debugging DevOps issue but this one baffles us till this point.

buckleyGI commented 2 years ago

(BTW Thanks for this excellent task. We use it extensively in our projects for which we are grateful 🤗 )

huserben commented 2 years ago

Hi @buckleyGI

regarding the version: It's because the extension has a version (3.1.1) while each individual task inside that extension has an individual version (3.2.2). Not ideal and probably it should be better aligned, but apart from being confusing it is not a problem :-)

Regarding to your problem: Essentially it looks like somehow the rights for your build user have changed and it does not have the necessary rights to queue a build.

This could be because some rights have changed on the Agent Pool. Is it possible that something has changed there lately? Is it possible to add the Build Service to the Agent Pool users: image

Be aware: I'm just guessing :-)

buckleyGI commented 2 years ago

Hi @huserben

Ok so the version history is a not a mystery anymore. I didn't know it was possibly the diverge from it.

We haven't changed any rights on the pool. Few people have access and worked on the troubleshooting. We also reviewed the audit log for changes in permissions (currently in preview on DevOps).

One thing I haven't mentioned is that the error message talks about the wrong pool:

Error: Access denied. Project Collection Build Service (REDACTED) needs Use permissions for pool XYZ to perform the action.

XYZ is not the pool that we expect. We ask the job to be run on pool ABC, names have been changed of course.

For now we have mitigated the issue and are using a PAT instead of OAuth. We think we will continue with the mitigation as the fix. We just have to remember to refresh the PAT on time every year.

tobiasbjelk commented 2 years ago

Hi @huserben

Ok so the version history is a not a mystery anymore. I didn't know it was possibly the diverge from it.

We haven't changed any rights on the pool. Few people have access and worked on the troubleshooting. We also reviewed the audit log for changes in permissions (currently in preview on DevOps).

One thing I haven't mentioned is that the error message talks about the wrong pool:

Error: Access denied. Project Collection Build Service (REDACTED) needs Use permissions for pool XYZ to perform the action.

XYZ is not the pool that we expect. We ask the job to be run on pool ABC, names have been changed of course.

For now we have mitigated the issue and are using a PAT instead of OAuth. We think we will continue with the mitigation as the fix. We just have to remember to refresh the PAT on time every year.

I am experiencing the same issue at the moment when using System.AccessToken as method. Error message is refering to a Agent pool we aren't even using, very strange can't find any reason yet. Problems started yesterday for us. I reviewed our auditlogs and no changes have been made what I can see so far.

huserben commented 2 years ago

@buckleyGI Good that you found a workaround at least. I'm still wondering what could be the cause of this though...

@Tobbe121 Interesting, so it's starting to happen on different systems. What I can clearly say is that the task did not change for quite a while now. So maybe some change from Azure DevOps itself could have caused it?

Sorry for the troubles, but currently I'm not really sure what I could do to improve this. If you have any idea I'm happy to look into it or make some change in the task if it could help

huserben commented 2 years ago

Will clsoe this issue due to inactivity. If the issue starts happening again please comment & reopen so we can have a deeper analysis into what could cause it.