When can we expect NTLM support?

[Zleep-Dogg]

So... what does "in a month or so" mean? https://github.com/Microsoft/tfs-cli/issues/20#issuecomment-130709079

The month seemed short enough that I didn't want to bother implementing it myself, but we could really use the ntlm support soon... anything we can do to help the progress? I'll be happy to test things as well...

[mmajcica]

Same here. TFS 2015 is out for already some time and still it is not supported by the official tool?

[StevenBorg]

Yikes. Would really like to install the Octopus Deploy task, which is not supported in TFS 2015.1. But we need to hit TFS in a production environment for over 300 dev teams. Changing to accept Basic Auth just ain't gonna happen.

[yaakov-h]

Same issue here. It seems that as much as Microsoft have touted cross-platform builds on TFS 2015, it isn't actually supported out of the box.

Any update on this, @bryanmacfarlane?

[gravufo]

Please implement this feature! Enabling Basic Auth for on-premises servers is just ridiculous...

[bryanmacfarlane]

Update: Extensions will be supported on-prem in QU2. That's the official vehicle for build tasks. That will have a web UI for uploading tasks which of course works with integrated auth.

Regarding other tfs-cli commands. We wanted to implement in core CLR (xplat) but it was not ready when we created this tool. core CLR on windows will support NTLM and integrated auth (domain joined). Therefore we are not doing the node work for ntlm (which wouldn't support integrated). The core CLR port of tfs-cli will support the cmd line options back compat so really it's an internal implementation detail.

[jessehouwing]

Workaround is to use Fiddler to handle the auth for you:

• http://blog.jessehouwing.nl/2016/01/publish-build-tasks-to-tfs-2015-without.html

[Allann]

Same issue. TFS On-Prem should be as important as VSTS as many companies are not able to use VSO.

[bryanmacfarlane]

Good news. We worked with the core CLR folks to ensure core CLR supports NTLM across Linux, OSX (and of course windows). As part of that work, we got our http clients building with core CLR and validated ntlm with linux/OSX/win.

Our new combined agent is building off of that. https://github.com/Microsoft/vsts-agent

That's paving the path for tfx to ported to core CLR. It's just a thin cmd line over our http clients anyways.

Of course we will maintain cmd line arg compat and it will encapsulate and bundler the core CLR with the tool.

Alot of work is leading up to the "right" solution. I know it seems like we're not doing work but it's building toward that.

[brettjacobson]

any update?

[bryanmacfarlane]

Assigning to Trevor to address the plans here for cli.

Couple things have happened since I posted:

• vso-node-api is getting ntlm added right now.
• new agent is working with integrated and ntlm on all platforms so we've confirmed the net core work.

The difference between node api and net core is integrated on windows works. So, if this picks up node api ntlm work, it will prompt and persist domain creds. If it uses net core instead, integrated just works on windows (no need to enter or persist creds).

So, the user question is whether avoiding prompting/persisting domain creds is critical enough to avoid.

[brettjacobson]

I could live with having to enter my credentials when executing tfx login, as long as the credentials are encrypted with NTLM. (Since right now, using --auth-type basic doesn't work with https! That problem makes this problem doubly important)

[bryanmacfarlane]

What's the issue with basic auth over https?

[mmajcica]

There is a workaround. This post may be useful. http://blog.jessehouwing.nl/2016/01/publish-build-tasks-to-tfs-2015-without.html?m=1

Mario On 6 Jun 2016 17:52, "Bryan MacFarlane" notifications@github.com wrote:

What's the issue with basic auth over https?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/tfs-cli/issues/45#issuecomment-224001725, or mute the thread https://github.com/notifications/unsubscribe/AC4ud_NfsytstOw5-xTqENAoX-FkMz1Yks5qJEI3gaJpZM4GOqKV .

[brettjacobson]

@bryanmacfarlane https://github.com/Microsoft/tfs-cli/issues/63 basic auth + https no workie on premises

[shieldsjared]

Agree that NTLM is critical need for those using on-prem instances. Even writing extensions is bothersome because we can't easily automate publishing to our on-prem instance w/o using basic authentication (and we're not turning on basic auth!). As an alternative, providing on-premise with tokens like VSTS has would have sufficed. The code/deploy/test cycle is time consuming for extensions!

[dangmike]

Still waiting on NTLM support for our on-prem TFS servers. Getting close?

[T-Hugs]

Yeah, I actually have done most of the work, but I'm really swamped with other things right now. My goal was to get it out by the end of this week. Thanks for hanging in there :)

[gravufo]

Bump. Any news on ETA?

[StevenBorg]

For the time being, you can always use Fiddler to intercept the traffic and do the authentication for you. The always brilliant JesseHouwing shows you how here: http://blog.jessehouwing.nl/2016/01/publish-build-tasks-to-tfs-2015-without.html

[StevenBorg]

Alternatively, you can package it as a VSIX extension, and upload it to the private TFS marketplace. (Or so I've heard...)

[jessehouwing]

Or use something like; http://ntlmaps.sourceforge.net/ to have something that's always on.

[gravufo]

Listen, I know the workarounds. Why don't you fix the problem at the source? @trevorsg said he was pretty much done with the code and wanted to ship ~20 days ago. I was asking what the status is on that, not "what are the workarounds".

Considering this issue has been open for almost a year now, I think it's quite normal that people are starting to be impatient.

Thanks for providing feedback at least, though.

[T-Hugs]

@gravufo - Unfortunately, the status is still the same. There's just a lot of stuff going on with finalizing TFS Dev15. This is the last week of the sprint, so it will be tricky to get this finished this week, but maybe I'll find a late night to work on it :)

[shieldsjared]

Well the news that dev15 is in its last sprint gives me a little less heartburn on the topic!

[gravufo]

No problem, that's all I wanted to know :) It just helps us plan accordingly. Thanks for the fast feedback!

[StevenBorg]

@gravufo Christian, I don't think Jesse and I meant to imply you didn't know the workarounds yourself. You've been on the thread too long for that. Our replies, however, are now there for those searching for a solution for their first time. As they come across this thread, they'll be able to maybe get past a roadblock that would have stopped them from moving forward. I know I appreciate the workarounds when I end up at a GitHub issue link from Bing / Google / DuckDuckGo. So, think of our comments as more of a community service than an insult to your google-foo. :-)

[gravufo]

Right, I didn't take it as an insult. I just thought you were trying to answer a question that wasn't asked :) I agree that it's great to have workarounds on Github, I love seeing them too. The first workaround was already posted in this same thread, but I guess adding the two other ones is worth mentioning. No worries and cheers!

[T-Hugs]

@shieldsjared Just to clarify, I said we're on the "last week of the sprint", not necessarily that this is the last sprint of Dev15 :)

[shieldsjared]

Ah, you're right! I guess I read "what I hoped" it said.  Excited for the next build, it's a bummer seeing all the goodness making it to VSTS before us on-prem folks.... But I get it! Thanks for putting me back into reality :/

[renatotkr]

any updates?

[swt-das]

Considering that "in a month or so" was a good 20-something months ago I assume we can consider this issue abandoned. Having to hardcode credentials in automated builds is problematic for all the obvious reasons.

Could we maybe get the patch that contains the work done up to now, so that someone else doesn't have to duplicate already done work?

[PeteW]

Adding to the list of teams which invest in TFS on-prem that wont be able to use your CLI creation because of this significant gap in functionality

[bryanmacfarlane]

TFS2017 on-prem supports PAT tokens (without enabling basic auth)

[swt-das]

@bryanmacfarlane And how do I use them in automated builds without hardcoding the token in some script?

As far as I can tokens aren't any more secure than using basic authentication and hardcoding username and password for that scenario (I just use a different method to revoke access and the windows user needs more configuration to be secure).

[bryanmacfarlane]

@swt-das How do you use it in a build without hardcoding in a script?

The build generates a token per build which expires after the build. That token represents the project or collection service account (depending on what you set in the options tab for scope). You can assign permissions etc... to that account. It's available to task authors and ad-hoc scripts (if you check that option on the options tab).

That is the most secure option since it's per build, not persisted anywhere and masked from logs automatically. No domain credentials could be leaked and if somehow compromised it's useless when the build ends.

Example in a task (would be same variable in a script). https://gist.github.com/bryanmacfarlane/154f14dd8cb11a71ef04b0c836e5be6e#file-vsts_sample-ts

To answer your specific question on PAT tokens. That's the second best option. You are correct that you shouldn't hard code in scripts - bad. In the build definition, there's a variables tab. Each variable has a lock icon next to it. If you lock the variables, it's securely stored and available to your build scripts as a variable. Because you locked it, it will be masked from logs. If you do that, then don't do the login option of tfx (will persist) - pass as cli args on invocations. Or use the rest clients from scripts as noted above. That's the second best option because (1) it's a token - not your domain credentials if leaked - scoped to scopes of the token - not everything on your domain as you (2) it can be revoked independently of your account and (3) and it can be masked from logs. But it's not as good as the per job token since if compromised, has a lifetime longer than the build.

Regarding NTLM, we have to separate NTLM from integrated. Integrated means your locally logged on identity. Where as ntlm support in this context means supplying the user name and password. That's less secure because if compromised, it's scope is you on the domain, not revokable independent of the account and would need to be persisted (the node-api typescript module tfx uses has NTLM, not integrated).

The most secure option for automating in a build would be to add an option in the cli to use the per build token we generate. It also works and untrusted domain scenarios.

[swt-das]

@bryanmacfarlane I thought that "NTLM" was referring to using the domain credentials of the logged in user (I was wondering why one would limit this to NTLM and not allow Kerberos though..), similar to what Invoke-WebRequest -UseDefaultCredentials would do. If that's not the case then it's not particularly helpful for my scenario anyhow, good to know.

The solution with a per-build token is perfect and I wasn't aware of that option. Clearly that's the way forward. Is there also a sample somewhere for how this would look in a PowerShell script?

[swt-das]

@bryanmacfarlane So I created a task based on the sample code and the code in app/exec/build/tasks/upload.ts.

The token I get back from tl.getEndpointAuthorizationParameter('SYSTEMVSSCONNECTION', 'AccessToken', false); is 755 characters long, so we're taking the apim.getBearerHandler route. But when trying to upload the task I get a 405 error.

import vm = require('./lib/jsonvalidate');
import trace = require('./lib/trace');
import tl = require('vsts-task-lib/task');
import trm = require('vsts-task-lib/toolrunner');
import * as apim from 'vso-node-api';
import taskagentm = require('vso-node-api/TaskAgentApi');
import path = require('path');
import archiver = require('archiver');

var c_taskJsonFile: string = 'task.json';

function getTaskAgentApi() : taskagentm.ITaskAgentApi {
    let url = tl.getEndpointUrl('SYSTEMVSSCONNECTION', false);
    console.log(`Connecting to ${url}.`);
    let token = tl.getEndpointAuthorizationParameter('SYSTEMVSSCONNECTION', 'AccessToken', false);
    let auth = token.length == 52 ? apim.getPersonalAccessTokenHandler(token) :
                                    apim.getBearerHandler(token);
    let vsts: apim.WebApi = new apim.WebApi(url, auth);
    let agentApi = vsts.getTaskAgentApi(url);
    return agentApi;
}

async function createTaskArchive(taskPath : string) : Promise<archiver.Archiver> {
    let archive = archiver('zip');
    archive.on('error', function(error) {
        console.log(`Archiving error: ${error.message}`);
        error.message = `Archiving error: ${error.message}`;
        throw error;
    });
    archive.directory(path.resolve(taskPath), false);
    archive.finalize();
    return archive;
}

async function run() {
    try {
        let taskPath = tl.getInput('taskDirectoryPath', true);
        vm.exists(taskPath, `specified directory ${taskPath} does not exist.`);
        let agentApi = getTaskAgentApi();
        let tp = path.join(taskPath, c_taskJsonFile);
        var taskJson = await vm.validate(tp, `no ${c_taskJsonFile} in specified directory.`);
        let archive = await createTaskArchive(taskPath);

        console.log('Upload archive.');
        await agentApi.uploadTaskDefinition(null, archive, taskJson.id, false);
        console.log('Success uploading');
    }
    catch (err) {
        tl.setResult(tl.TaskResult.Failed, err.message);
    }
}

run();

[kasajian]

I think that there's a much simpler solution to this whole thing. This isn't about NTLM, because you probably want Kerberos anyway. What you're really asking for single-sign-on, via "Integrated Windows Authentication" -- SPNEGO.

The problem is that with NodeJS, the http stack doesn't support this -- and to use the "kerberos" module is problematic.

What I've done in other projects when I want to use NodeJS code that will do REST to unprem TFS server is to make the HTTP calls not via the HTTP module but via the libcurl module. Works great, and you don't have pass in user-name and password.. it uses the credentials of whatever you're logged in as. Of course you can pass it in. But the point is, you don't need to use insecure "basic authentication" which requires you to update the TFS server IIS box every time you upgrade TFS, nor do you need a "personal access token" -- you're already authenticated.

Here's examples that you can prove to yourself this works with regular curl.exe and with node-libcurl.

Using regular curl: xargs -a url_tfs.txt curl -negotiate -u : where tfs.txt is a text file containing the REST endpoint such as: http://mylocaltfs:8080/tfs/mycol/_apis/wit/WorkItems?ids=42&fields=System.Id,System.WorkItemType,System.Title,System.State&api-version=1.0

(I put the url in a text file instead of the command-line so that I can avoid having to escape the ampersands)

If you want to know why user name is being passed as ":", then read the curl manual (curl --manual). it's a weird reason.

And here's an example of how to do this with node:

var endpoint = urlString;
var url = require("url");
var endpointUrl = url.parse(endpoint);

var Curl = require( 'node-libcurl' ).Curl;
var curl = new Curl();

curl.setOpt( 'USERNAME', '' );
//curl.setOpt( 'VERBOSE', 1 );
curl.setOpt( 'URL', endpoint );
curl.setOpt( 'HTTPAUTH', Curl.auth.NEGOTIATE );
curl.setOpt( 'NOPROXY', endpointUrl.hostname );

curl.on( 'end', function( statusCode, body, headers ) {

    if (statusCode === 200) {
        console.log(body);
        cb(null, { statusCode, body, headers } ); 
    } else {
        cb(new Error(), { statusCode, body, headers } ); 
    }

    this.close();
});

curl.on( 'error', curl.close.bind( curl ) );
curl.perform();

So what you want to do is add a new authentication type (call it "negotiate" or something), and then use this method of making REST calls instead of what's currently done.

I don't have a POST example, but with curl, libcurl, and node-libcurl being popular modules, I'm sure it wouldn't be too hard to come up with a sample that works with on-prem TFS.