Feature request: Add option to ignore untrusted SSL certs

[CamSoper]

curl has a -k command line switch that tells it to ignore untrusted certs. I don't care what shape it takes (command line switch, preference, whatever) but we really need similar functionality in the HttpRepl to use it in Microsoft Learn modules, as we have no way of trusting a developer certificate in the Azure Cloud Shell environment.

cc: @scottaddie @tlmii @bradygaster

[tlmii]

Initial thought, I think we probably want this as both a preference (for scenarios like the Azure Cloud Shell where all/most of your commands will be against untrusted certs) and as an option on all HTTP commands (to allow people to do one-offs without putting them in harms way by making it persistent).

[scottaddie]

This issue is closely related to the discussion in https://github.com/dotnet/HttpRepl/issues/326

[tlmii]

After thinking this over a little bit, here's my proposal:

1. New preference, maybe httpClient.allowUntrustedCerts, default false
2. New option on all HTTP Commands (get, post, etc) as well as connect: -k|--insecure. This does the same as the option, but JUST for that command. Those are the same as curl and I really like --insecure.
3. (Less certain on this) If we detect errors like in #326 we can display a message informing users of the two above options. But we'd probably want to put a big glowing red marquee banner warning people not to use them unless they're really extra sure.

Open to suggestions on the naming of all of the above, but that's less important than the rest of the design.

[CamSoper]

That all sounds pretty good to me. I'm a little ambivalent on the last bullet, but overall I think it's probably okay if the language is scary enough. 🙂

[tlmii]

I'm a little ambivalent on the last bullet, but overall I think it's probably okay if the language is scary enough. 🙂

Perhaps we could limit when we show the suggestion too. Maybe only if they're connecting to something on localhost or if we can somehow detect that they're running in an environment like Azure Cloud Shell.

[tlmii]

OK, one interesting thing I didn't think about until I got into testing how this could work:

HttpRepl currently just uses a single HttpClient and HttpClientHandler instance across the whole session of the tool. Once the certificate is trusted once, it will continue to be trusted in the future (until you restart the tool). That makes sense, but it can feel weird in practice.

For example, if you issue the command get https://self-signed.badssl.com --insecure, the request will succeed. If you then issue the command get https://self-signed.badssl.com (note the lack of the --insecure option), it will also succeed.

We could fix that by throwing away either the HttpClient or the HttpClientHandler (not sure which one does the caching, probably the latter) after every request (or maybe after every insecure request). Since we're not intending to be doing thousands (or even double-digits) of requests per second, we can probably get away with that. Or we could just document the behavior.

This wouldn't be the case in curl since each command is a separate execution of the tool. So this is sort of unique to our REPL way of doing things.

Thoughts?

[tlmii]

I'm also now wondering if, for the sake of security, we might be better off having two preferences - one to disable the check globally and one to disable it just for localhost.

Or even better change it to an allow list of hosts instead of global. So you'd have to be really explicit that you want to trust a cert served by a particular hostname.

[CamSoper]

I like the allow list concept over the global. I think that solves your security concern in the case of preferences, because if the user adds a URL to the allow list, they know they did it. They would expect the same behavior on each request.

In the case of an interactive --insecure, in your example, would it be possible to only throw away the HttpClientHandler in the event that a request with --insecure is then followed by a request that doesn't have the argument? I don't know if that gains you anything though.

[bradygaster]

concur with @CamSoper and think the proposal @tlmii is making is sound.

[bradygaster]

This is what I think sounds best:

Or even better change it to an allow list of hosts instead of global. So you'd have to be really explicit that you want to trust a cert served by a particular hostname.

[bradygaster]

okay at our peril I admit I am wondering what @blowdart would say about this. what's your guidance here, sir security?

[blowdart]

"httpClient.allowUntrustedCerts" where it's global is horrifying, and too wide ranging. What on earth are you doing in azure cloud shell that everything ends up untrusted?

A list of host names (and I'd say you could add localhost to that as a starting point), or better yet, certificate fingerprints would be the way to go.

You'll also need an SDL exemption from someone above my level for any of this.

[bradygaster]

Sounds like he saved us a bunch of work. :)

[blowdart]

Oh and that exemption will need renewing annually :)

[bradygaster]

so how does curl do it without being nefarious? or is curl nefarious?

[blowdart]

We don't publish curl.

An option is an easy argument to make. Defaulting it to be on is not. An option that is global and persists is not.

[bradygaster]

So for us to add the --insecure option that only applies to that request or to that session of the app. Any difference in terms of the exception-acquisition process and complexity [and obviously, the security implications]?

[blowdart]

Nope, we'd need to figure out what the sdl requirement is, email hunter saying you're adding an option to turn off that validation and ask for an exception, then save that email away somewhere safe for when you have to sign everyone off in 1cs. Which of course you're doing, to make sure you pass all your SDL and release requirements.

[bradygaster]

The HuntR has been pung

[CamSoper]

@blowdart

What on earth are you doing in azure cloud shell that everything ends up untrusted?

Microsoft Learn uses the Azure Cloud Shell as a sandbox environment for students. The problem we have is that there is zero way we can find to add a development certificate to the trusted certificates in that environment. Cloud Shell doesn't give users root, so our options are limited.

So in a scenario where we tell the student to dotnet run a web API, the web API's SSL endpoint is using an untrusted certificate. With curl, no big deal, we have the student add -k to the request and go along our merry way.

Tim has put together this great tool in the HttpRepl, and we'd much rather show our students how to use that than curl.

@tlmii @bradygaster For our purposes, just adding --insecure at launch time (like, as a command line parameter and only a command line parameter) and having it apply to the entire session seems to make sense.

[blowdart]

A big fat warning banner when the option is used would be good.

[maciej-izak]

Any news on this? This makes HttpRepl unusable for me.