SourceLink Authentication error when using Microsoft.SourceLink.AzureRepos.Git

dmuehlenhoff commented 4 years ago

I am trying to enable SourceLink for our internal NuGet packages and am running into a problem where I don't see how I can solve it.

My setup is:

Visual Studio 2019 Enterprise (16.3.8)
An Azure DevOps git repository
A .NET Framework 4.7.2 class library with an SDK style project containing a single HelloWorld class
A PackageReference to Microsoft.SourceLink.AzureRepos.Git, using the latest Version (1.0.0-beta2-19554-01)

This project generates a package that includes the DLL and the PDB (I know this is discouraged, but I would like to get it to work this way rather than using a symbol server; this also does not seem to be the source of the problem, since the debugger finds the correct PDB)

The project file

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net472</TargetFramework>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <AllowedOutputExtensionsInPackageBuildOutputFolder>$(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb</AllowedOutputExtensionsInPackageBuildOutputFolder>
    <RepositoryUrl>https://<host>.visualstudio.com/<project>/_git/Lib</RepositoryUrl>
    <RepositoryType>git</RepositoryType>
    <AssemblyVersion>2.0.0.0</AssemblyVersion>
    <FileVersion>2.0.0.0</FileVersion>
    <Version>2.0.0</Version>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.SourceLink.AzureRepos.Git" Version="1.0.0-beta2-19554-01">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>
</Project>

I commit my changes to the repository and build the package. The package builds fine. When I check the generated .nuspec file, it contains the correct commit hash.
I publish it to a local NuGet feed using nuget add Lib.2.0.0.nupkg -Source C:\NuGet.
I delete my local HelloWorld sources so the debugger does not find these sources and tries to download them from the git repository
I create a console application, reference this NuGet package, and call the HelloWorld function on the HelloWorld class. In the debugger, I hit F11 to step into the HelloWorld call. The "SourceLink will download [...]" dialog box appears and I select "Download Source and Continue Debugging".

This does not work. I get the following error message

Source Link Error:
ERROR: Azure DevOps: Authentication failed for all accounts. Use 'File -> Account Settings...' to add a new account or refresh credentials.

Source Link URL: https://<host>.visualstudio.com/<project>/_apis/git/repositories/Lib/items?api-version=1.0&versionType=commit&version=cf94d5a0f224b6983c3885a2a6ac29aa52cec418&path=/HelloWorld/HelloWorld.cs

Now, the error message is pretty clear, I just don't get it. When I go to the URL stated in the error message, I see the file content I am expecting. I used a browser I never use to check, and when navigating to this page, it asked me to sign in, and I do so using the same account I use in Visual Studio. I also tried signing out of Visual Studio, restarting it, and signing in again to Visual Studio using the same account I used in the browser. Still the same error.

Any idea what I could be doing wrong?

Thanks, David

tmat commented 4 years ago

@chuckries Any ideas?

BTW, you can remove the following lines from the project, the properties are set by Source Link automatically:

    <RepositoryUrl>https://<host>.visualstudio.com/<project>/_git/Lib</RepositoryUrl>
    <RepositoryType>git</RepositoryType>

chuckries commented 4 years ago

@dmuehlenhoff could you use fiddler to capture a network trace of devenv.exe trying to access visualstudio.com? The request headers should contain an Authenticaiton header with a 'Bearer' token.

dmuehlenhoff commented 4 years ago

@chuckries It does not contain an Authentication header. Not even the URL from the error message is used here, just the base URL of the DevOps instance.

Request

HEAD https://<host>.visualstudio.com/ HTTP/1.1 User-Agent: Microsoft.VisualStudio.Debugger.SourceLink.16.0.0.0 X-TFS-FedAuthRedirect: Suppress Host: <host>.visualstudio.com

Reply

HTTP/1.1 401 TF400813: The user '' is not authorized to access this resource.

chuckries commented 4 years ago

@dmuehlenhoff are there any other requests to visualstudio.com? I would not expect the HEAD request to be authenticated. There should be a subsequent GET request.

dmuehlenhoff commented 4 years ago

@chuckries Sorry, I should have been more clear. This is the only request that is being sent by devenv.exe, there's nothing more being recorded in Fiddler (and I did install the root certificate to enable HTTPS traffic decryption; I also have a fresh install since I never had Fiddler on this machine before, so it's the default Fiddler config).

I just tried the following: I logged out of my account in VS so that there is no logged on user in VS. The trace in Fiddler stays exactly the same (just the HEAD request that gets the 401 answer). The error message stays the same as well. It looks to me like what is happening is that whatever SourceLink or VS is trying to do, it doesn't even try to authenticate using the logged in user Is there some way to check what conditions need to be satisfied for SourceLink/VS to find which users it tries?

dmuehlenhoff commented 4 years ago

@chuckries In addition to that, I just tried to use the "Authenticate for SourceLink" link in the error window to see if that gives me something different. When I use this, I get the standard Microsoft authentication window, where I enter my credentials again (same as in VS and as in the browser). I get the same error, but I get a GET instead of a HEAD request in Fiddler.

Request

GET https://<host>.visualstudio.com/<project>/_apis/git/repositories/Lib/items?api-version=1.0&versionType=commit&version=cf94d5a0f224b6983c3885a2a6ac29aa52cec418&path=/HelloWorld/HelloWorld.cs HTTP/1.1 Authorization: Bearer <a long bearer token> User-Agent: Microsoft.VisualStudio.Debugger.SourceLink.16.0.0.0 X-TFS-FedAuthRedirect: Suppress Host: <host>.visualstudio.com

Response

HTTP/1.1 401 Unauthorized Content-Length: 89692 Content-Type: text/html WWW-Authenticate: Bearer .....

And somewhere in the HTML there is this message in the title: 'TF400813: The user '<myusername>' is not authorized to access this resource.', with some helpful watchdogs on the page content.

I swear, that is my user and this works when using any browser. I sniffed the browser using Fiddler and the difference seems to be that the browser uses Cookies instead of a Bearer token. The cookie contains VstsSession SpsAuthenticatedUser UserAuthentication FedAuth FedAuth1

At first I thought that the value of UserAuthentication is the bearer token, but it's not. They start the same for about 30 to 40 bytes, but then start to differ and are not the same length (bearer token 1564 bytes, the cookie part 1095 bytes).

chuckries commented 4 years ago

@dmuehlenhoff Do you have a 'filter' enabled on your Azure DevOps account in VS? For example, do you see 'Clear Filter' in Account Settings next to your identity? Like this:

dmuehlenhoff commented 4 years ago

@chuckries no, I do not see the Clear filter link, just the Remove link.

chuckries commented 4 years ago

@dmuehlenhoff did you set up your repo through Azure Repos or Azure DevOps? I assumed Azure Dev Ops based on the URL, but it looks like you are including the Azure Repos sourcelink package.

Additionally, is your account through an Enterprise organization backed by Azure Active Directory? Or is a Microsoft Account that you made through something like microsoft.com or outlook.com? Essentially, is your account an enterprise account or a personal account?

dmuehlenhoff commented 4 years ago

@chuckries it's an enterprise account from my organization. I do not really know the details, as I'm not involved in it, but the user is the same I'm using to log into windows. I think our internal IT still has their own AD and just synchronizes the accounts, but I'm really not sure about that. Does this matter? If yes, could you tell me what info you need exactly so I can try to get someone from IT to answer it?

Also, I did not set up the project, I just created the git repos in it. Can I see anywhere what type of project it is? If that helps, it looks exactly like the screenshot on here

tmat commented 4 years ago

@dmuehlenhoff What is the domain of the URL that git remote -v prints out?

dmuehlenhoff commented 4 years ago

(note: I did create another repo in the meantime so the HelloWorld is gone and replaced by Lib, but it has the same issue)

C:\Users\<user>\source\repos\Lib (master -> origin) λ git remote -v origin https://<host>.visualstudio.com/<project>/_git/Lib (fetch) origin https://<host>.visualstudio.com/<project>/_git/Lib (push)

tmat commented 4 years ago

@dmuehlenhoff That looks ok. You have the right package (Microsoft.SourceLink.AzureRepos.Git).

chuckries commented 4 years ago

@dmuehlenhoff are you using a proxy server?

dmuehlenhoff commented 4 years ago

@chuckries Not that I am aware of. There's none configured in windows at the moment, and I'm in my home network (also not connected to the company network via VPN) so there is no transparent proxy or anything.

chuckries commented 4 years ago

@dmuehlenhoff Let's try this, just as an exercise:

Open Team Explorer (View -> Team Explorer)
Under 'Hosted Service Providers -> Azure DevOps' click 'Connect'
The 'Connect To A Project' dialog will appear
Use the drop down to select the identity that you are trying to use for Source Link (your user account)
Does the git repo that you are trying to use for Source Link appear in the list? That is, if you find <host>.visualstudio.com in the list and expand it, do you see the repo that you are working with?

dmuehlenhoff commented 4 years ago

@chuckries yes, the repo is visible in the 'Connect to a project' dialog. The way I get there is slightly different from what you describe though:

Team explorer shows the existing connection to the Azure DevOps (the project, not the repo, since I deleted it locally), our internal old TFS that we want to replace with DevOps, and my local git repositories. Above them is a 'Manage connections' link, which opens the 'Connect to a project' dialog that shows the project and, when expanded, the repo. connectdialog

chuckries commented 4 years ago

Next thing to try would be to get the bearer token that is being sent by VS out of fiddler and paste it here: https://jwt.ms/

This will show you several details about the token in use. Hopefully we can determine if it is even using the correct user.

dmuehlenhoff commented 4 years ago

Yes, the token is issued by AAD to my username I use to log into Azure DevOps and VS. Is there any information inside the token that would be useful to further analyze the problem? Do you know of any way I can compare the token information to the one the browser uses?

dmuehlenhoff commented 4 years ago

So I just checked, and the "UserAuthentication" value of the cookie that is used when I access Azure DevOps from the browser is also a decodable token. It seems to be completely different though.

The cookie token does not show "Issued by Azure ActiveDirectory", the VS token does.
The cookie token is issued by "https://sts.windows.net/someguid/", the VS token by "app.vstoken.visualstudio.com"
They both have different values for the subject ("sub") field. The value in the cookie token is a somewhat readable NT-style username (<someguid>\<myusername>), the VS token is some random text. It might be the same value encoded of course, but I don't know how to decode it to check. The unique_name field on the other hand is equal in both (my username)
The cookie token has an Azure tenant id ("tid"), the VS token has a different one. I don't really understand why the cookie one has a tid at all, since it appearently wasn't issued by AAD.

I have no real idea about JWT tokens though, just shout if you need me to compare other things.

chuckries commented 4 years ago

@dmuehlenhoff I have made a small VS Extension that runs through the Dev Ops source link logic and spits out some logging around tenant/account selection. Please install it into your VS, open it through View -> Other Windows -> DevOpsSourceLinkAudit, and paste the URL you expect to work into the text block and click Go.

Between the log it spits out and an associated fiddler trace, we can continue to drill into this. I encourage you to send the results to me directly at chuckr@microsoft.com, and we can continue this discussion off of GitHub. I will report any findings back to this issue.

The extension is available here: https://1drv.ms/u/s!AkMiT-n3a--N9jC0aUIFIFB3aoI2?e=OuxRus

mvonballmo commented 4 years ago

@chuckries: Thanks very much for the extension. Do you have any more information on this issue? I'm getting the same behavior. I've installed the extension and get the following output:

Found cached tenantId: 00000000-0000-0000-0000-000000000000 Using tenantId: 00000000-0000-0000-0000-000000000000 Azure DevOps: Authentication failed for all accounts. Use 'File -> Account Settings...' to add a new account or refresh credentials.

This doesn't seem right, does it? I'm authenticated in Visual Studio (I signed out and back in). I can access the files from an authenticated browser. Fiddler doesn't show much (as it's HTTPS) ... unless you can tell me what to look for. I'm happy to take it offline ... just thought I'd ping here first before hitting your email.

scp-mb commented 4 years ago

@chuckries I'm also having the exact same issue as @mvonballmo with both GUIDs being all zeros. Was there ever any resolution to this with @dmuehlenhoff ?

Exact same symptoms, authenticated in VS with a sign out / back in and works fine from a browser with the same link.

VS version info below:


Version 16.5.0 Preview 4.0
VisualStudio.16.Preview/16.5.0-pre.4.0+29820.132
Microsoft .NET Framework
Version 4.8.03752

Installed Version: Enterprise

ASP.NET and Web Tools 2019   16.5.236.49856
ASP.NET and Web Tools 2019

ASP.NET Web Frameworks and Tools 2019   16.5.236.49856
For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0   16.5.236.49856
Azure App Service Tools v3.0.0

Azure Functions and Web Jobs Tools   16.5.236.49856
Azure Functions and Web Jobs Tools

C# Tools   3.5.0-beta3-20119-02+fdeac238b502eb9a810a287311c7b2040f9c3b4e
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools   1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Fabric.DiagnosticEvents   1.0
Fabric Diagnostic Events

IntelliCode Extension   1.0
IntelliCode Visual Studio Extension Detailed Info

Microsoft Azure Service Fabric Tools for Visual Studio   16.0
Microsoft Azure Service Fabric Tools for Visual Studio

Microsoft Azure Tools   2.9
Microsoft Azure Tools for Microsoft Visual Studio 2019 - v2.9.30207.1

Microsoft Continuous Delivery Tools for Visual Studio   0.4
Simplifying the configuration of Azure DevOps pipelines from within the Visual Studio IDE.

Microsoft JVM Debugger   1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft Library Manager   2.1.25+gdacdb9b7a1
Install client-side libraries easily to any web project

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual Studio Tools for Containers   1.1
Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.

Node.js Tools   1.5.20122.1 Commit Hash:d74ac73e6e22f9e009948cab4535a26194b19b1b
Adds support for developing and debugging Node.js apps in Visual Studio

NuGet Package Manager   5.5.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

ProjectServicesPackage Extension   1.0
ProjectServicesPackage Visual Studio Extension Detailed Info

Snapshot Debugging Extension   1.0
Snapshot Debugging Visual Studio Extension Detailed Info

SQL Server Data Tools   16.0.62002.03150
Microsoft SQL Server Data Tools

TypeScript Tools   16.0.20205.2002
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools   3.5.0-beta3-20119-02+fdeac238b502eb9a810a287311c7b2040f9c3b4e
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools 10.8.0.0 for F# 4.7   16.5.0-beta.20104.8+7c4de19faf36647c1ef700e655a52350840c6f03
Microsoft Visual F# Tools 10.8.0.0 for F# 4.7

Visual Studio Code Debug Adapter Host Package   1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Container Tools Extensions (Preview)   1.0
View, manage, and diagnose containers within Visual Studio.

Visual Studio Tools for Containers   1.0
Visual Studio Tools for Containers

Visual Studio Tools for Kubernetes   1.0
Visual Studio Tools for Kubernetes```

dmuehlenhoff commented 4 years ago

@scp-mb I did talk to chuck a bit via mail, but the issue has never been resolved; his last statement is he needs to wait for people who know more about Azure DevOps and they're not available right now. That has been 2 months ago though, so I'm not sure what's going to happen.

The reason for the error in my case seems to be that we are using users from different tenants (our Azure DevOps is on a different tenant than the users that access it), and this leads to a mix of Azure AD and MSA authentication, which cannot be handled by the current SourceLink implementation.

scp-mb commented 4 years ago

@dmuehlenhoff That's an interesting point about the tenants, I believe we've had issues in the past where our AD accounts didn't seem to work, so we had to create separate Microsoft accounts at some point. That might be a starting point...

scp-mb commented 4 years ago

As a follow up, I've tested it on an AD joined machine using an Azure AD account with the exact same result, so it would appear the issue is elsewhere.

scp-mb commented 4 years ago

Right, I've got to the bottom of the issue on my end. It was an issue with Microsoft vs AD account. Turns out I was looking at it the wrong way around, and assumed my devops account was using my AD account.

I was attempting to authenticate to devops via sourcelink with an Azure AD account. Turns out my devops account is actually pointed at a Microsoft account, so logging in as another user (but then using same email), then selecting the "home" account worked.

So in all, the fix should be as simple as adding the AD account to devops.

@mvonballmo @dmuehlenhoff I'd suggest checking the same on your end.

dmuehlenhoff commented 4 years ago

Good to have this confirmed, thanks for the information. Sadly this is not an option for us for quite some time. I guess we'll have to look for another solution since it doesn't seem there's anything going forward here.

bravequickcleverfibreyarn commented 3 years ago

The same in here. In fact I am not going to verify actual request in browser and from VS. That is job of other guys. Nor I am going to make magic with AD that even I do not have. For Azure DevOps it must be really PITA to add Symbols Server. That is for sure. 👎🏻

wouterroos commented 2 years ago

Hi,

I also have this issue in VS2022. The extension mentioned in this issue to collect logs doesn't work in VS2022. I've verified that I'm able to access the source files through the browser using the same (AD) account that I use for everything. I tried authenticating with a Microsoft Account that also has access to the repo I'm trying to get SourceLink working for and that works fine, but since there are more people in my company who need this I need to get it to work with my AD account.

Any update on this would be greatly appreciated.

UPDATE:

I used Fiddler to see what is going on and I'm seeing the following error (email address ommitted but I can confirm that it's the correct one):

TF400813: The user '**@****.com' is not authorized to access this resource. The user is not a member of the AAD directory that the target Organization is connected to.

The thing is, our DevOps organization isn't connected to AAD:

As mentioned before, A Microsoft account that has permissions to the DevOps organization works fine. It seems that an AAD connection is assumed when an AAD user tries to authenticate, I guess this shouldn't be the case when inviting AAD users to an organization that isn't connected to an AAD instance.

gnimor commented 2 years ago

We are in a similar situation as @wouterroos and are looking for a work-around on this issue. Any update on this?

wouterroos commented 2 years ago

Bump.

KirillSokolovsky commented 2 years ago

Faced the same issue with source link and our private Bitbucket server. Use Internet Explorer on local machine and authorize in your VCS with password saving IE saves password in Windows Credential Manager (Web Credential) And seems like source link uses credential exactly from here (in case authorization is required)

dhtek commented 2 years ago

I have also an issue trying to debug a Razor project with Blazor Web Component using WASM. When I hit the breakpoint, and press F11 to step in the code, I got this

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html lang="en-US">
<head>
    <title>
        Azure DevOps Services | Sign In
    </title>
    <meta http-equiv="X-UA-Compatible" content="IE=11;&#32;IE=10;&#32;IE=9;&#32;IE=8" />
     ...
</head>

<body class="platform">
    <input name="__RequestVerificationToken" type="hidden"
        value="xxxx" /><input
        name="__RequestVerificationToken2" type="hidden"
        value="__RequestVerificationTokenxxxxx" />
    <div class="account signin main-container hide">
        <div class="page-content">
            <div class="header-section" role="banner">
                <link rel="stylesheet" type="text/css" href="/_static/tfs/M209_20220831.4/_content/Combined.css">
                <div id="ux-header" class="FF ltr vsIntegrate" xmlns="http://www.w3.org/1999/xhtml">
                    <span id="isMobile"></span>
                    <div class="upperBand">
                        <div class="upperBandContent">
                            <div class="left"></div>
                            <div class="right">
                                <div class="profileImage"></div>
                                <div id="signIn">
                                    <a class=":SignedOutProfileElement: createProfileLink" href="/go/profile"
                                        title="Anonymous">Anonymous</a>
                                    <a class="scarabLink" href="/_signout">Sign out</a>
                                </div>
                            </div>
                            <div class="clear-both"></div>
                        </div>
                    </div>
                </div>
            </div>
            <div class="content-section" role="main">
                <div class="signin-main-content">
                    <noscript>
                        <span class="error">Microsoft Internet Explorer&#39;s Enhanced Security Configuration is
                            currently enabled on your environment. This enhanced level of security prevents our web
                            integration experiences from displaying or performing correctly. To continue with your
                            operation please disable this configuration or contact your administrator.</span>
                    </noscript>
                    <div class="provider-control">
                        <script class="options" defer="defer"
                            type="application/json">{"providerOptions":{"force":false,"orgIdAuthUrl":"https://login.microsoftonline.com/common/oauth2/authorize?client_id=xxxx\u0026site_id=xxxxx\u0026response_mode=form_post\u0026response_type=code+id_token\u0026redirect_uri=https%3A%2F%2Fspsprodweu2.vssps.visualstudio.com%2F_signedin\u0026nonce=xxxx\u0026state=realm%3Dxxxx.visualstudio.com%26reply_to%3Dhttps%253A%252F%252Fxxxx.visualstudio.com%252Fxxxx%252F_apis%252Fgit%252Frepositories%252Fxxxx%252Fitems%253Fapi-version%253D1.0%2526versionType%253Dcommit%2526version%xxxx%2526path%253D%25252Fxxxx%25252Fxxxxx%25252Fxxxxx%25252FStringExtensions.cs%26ht%3D2%26hid%3Dxxxx%26nonce%xxxx\u0026resource=https%3A%2F%2Fmanagement.core.windows.net%2F\u0026cid=xxxx\u0026wsucxt=1\u0026githubsi=true\u0026msaoauth2=true","user":null,"signInContext":"xxxx"}}</script>
                    </div>
                </div>
            </div>
        </div>
       ...
</body>
</html>

I already check my account settings and everything is ok. I tried to sign out and sign in again in Azure Devops with Edge and chrome before debugging, same error. Our DevOps organization isn't connected to AAD

mostmand commented 1 year ago

Facing the same issue for our on-premise Azure DevOps server. Is there going to be a fix or workaround?

klinckag commented 2 months ago

Is there any news on this? Facing the same issue with Blazor Web Component using WASM.

For the ASP.NET Core website which is hosting the Blazor sourcelink/debugging is working.

dotnet / sourcelink