Insecure Proxy Behavior with Invoke-(RestMethod|WebRequest) on 7.4.0-preview2

[crlogic]

Prerequisites

• [X] Write a descriptive title.
• [X] Make sure you are able to repro it on the latest released version
• [X] Search the existing issues.
• [X] Refer to the FAQ.
• [X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

I use a number of script-based PWSH modules. In addition to using Invoke-RestMethod for a lot of API calls. These calls/modules are executed in a corporate environment with insecure/zero-trust proxies all over the place. The proxies cannot be avoided.

When testing PWSH v7.4.0-preview2, all script-based modules and IRM/IWR throw an error. Eg. https://github.com/dell/OpenManage-PowerShell-Modules

I believe this is part of PR #18595. It seems that in certain scenarios, the change is breaking.

Adding the following to script-based modules and repos would be a large effort with external modules being out of my control. Unclear of best approach to mitigate in order to work before/after the 7.4.0 threshold.

if ($psVersionTable.PSVersion.Major -ge 7 -and $psVersionTable.PSVersion.Minor -ge 4) {
    $querySplat.Add('AllowInsecureRedirect',$true)
}

Expected behavior

[2881.31]ms GitHub> # Works before 7.4.0-preview2
$querySplat = @{
    Method = 'POST'
    URI = $URI
    Body = $bodyJSON
    SkipCertificateCheck = $true
    ContentType = 'application/json'
}
Invoke-RestMethod @querySplat
Id                    : a1bf17dd-d590-4929-8f0a-7357bed828f8   
Description           : abcde.ab@OU=Groups,DC=test,DC=Contosa,DC=com
Name                  : API
...

Actual behavior

[2881.31]ms GitHub> # Works before 7.4.0-preview2
$querySplat = @{
    Method = 'POST'
    URI = $URI
    Body = $bodyJSON
    SkipCertificateCheck = $true
    ContentType = 'application/json'
}
Invoke-RestMethod @querySplat

Invoke-RestMethod: 
Line |
   9 |  Invoke-RestMethod @querySplat
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | This operation is not supported for a relative URI.

### Error details

```console
Exception             : 
    Type       : System.InvalidOperationException
    TargetSite : 
        Name          : get_Scheme
        DeclaringType : uri
        MemberType    : Method
        Module        : System.Private.Uri.dll
    Message    : This operation is not supported for a relative URI.
    Source     : System.Private.Uri
    HResult    : -2146233079
    StackTrace : 
   at System.Uri.get_Scheme()
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
   at System.Management.Automation.CommandProcessor.ProcessRecord()
CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], InvalidOperationException
FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
InvocationInfo        : 
    MyCommand        : Invoke-RestMethod
    ScriptLineNumber : 8
    OffsetInLine     : 1
    HistoryId        : 8
    Line             : Invoke-RestMethod @querySplat 
    Statement        : Invoke-RestMethod @querySplat 
    PositionMessage  : At line:8 char:1
                       + Invoke-RestMethod @querySplat 
                       + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    InvocationName   : Invoke-RestMethod
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 8

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.0-preview.2
PSEdition                      Core
GitCommitId                    7.4.0-preview.2
OS                             Microsoft Windows 10.0.19045
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}       
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[CarloToso]

@crlogic Does this work if you use -AllowInsecureRedirect? (if it does, probably there's a bug and I'll try to fix it ASAP)

[crlogic]

@crlogic Does this work if you use -AllowInsecureRedirect? (if it does, probably there's a bug and I'll try to fix it ASAP)

Affirmative. Adding -AllowInsecureRedirect works on 7.4.0-preview2 but that has to be conditionally applied since below that version (of which I only really tested 7.3.x) doesn't have the param.

[CarloToso]

Could you post the $URI?

The error originates here: response.RequestMessage.RequestUri.Scheme == "https" && response.Headers.Location?.Scheme == "http" It tries to get the scheme from a relative uri (like www.google.com) and fails to get http or https because they are missing.

[crlogic]

Here you go :)

https://ome.contosa.com/api/SessionService/Sessions

[crlogic]

It tries to get the scheme from a relative uri (like www.google.com) and fails to get http or https because they are missing.

I live in proxy central (Axis/ZTNA/others) where all sort of mangling likely occurs.

[CarloToso]

@SteveL-MSFT @iSazonov Maybe we should just revert #18595, if a proxy mangles the url checking for the scheme is useless and we can't determine if a redirect would be insecure. In this case -AllowInsecureRedirect should not be needed, it only skips che check and the error related to get_Scheme()

[iSazonov]

A silent redirect from HTTPS to HTTP is a potential security hole even on the internal network. I don't think PowerShell should encourage insecure scripts and insecure ways to build applications and environments.

[CarloToso]

A silent redirect from https to http is impossible without the -AllowInsecureRedirect parameter, it's not supported by System.Net.Http.HttpClient. Before 7.4 preview it failed silently. I added an error to check if there was an insecure redirect and help the discovery of the new parameter, unfortunately it looks for the Scheme property and fails if the url is relative. I see two options now:

• Remove the error, and go back to failing silently
• Introduce a check for relative uris and skip the error check if the uris are relative

[crlogic]

A silent redirect from HTTPS to HTTP is a potential security hole even on the internal network.

Understandable..

I don't think PowerShell should encourage insecure scripts and insecure ways to build applications and environments.

I don't feel like this is something that PowerShell has any say in; hear me out..

Eg. As an end-user that is mandated to leverage the "security policies" of the corporation, I have no say in those policies and the argument that "PowerShell will no longer work" will merely receive a response from the "security team" that "I should use better tools."

I should have to ability to make the decision to "write an insecure script because I have been mandated to do so". After all, the "security team mandates I do it that way." Even if they are wrong from an ideological perspective.

I don't know how these zero-trust tools/tunnels/redirects work which doesn't help my case. All I know is that, "it works in pwsh 7.3.3 and below" and a lot of tools will break if the preview2 behavior is maintained.

• Remove the error, and go back to failing silently
• Introduce a check for relative uris and skip the error check if the uris are relative

Perhaps a third option is a warning and continuing?

Eg. VMware PowerCLI outputs a warning if deprecated cmdlets/parameters are used.

[doctordns]

The problem I see with the proposed 7.,4 behavior is that it introduces a breaking change. A script that runs in 7.3 now fails. While the script could be fixed in 7.4.,x (ie add the -AllowInsecureRedirect parameter) but then that script no longer works in 7.3.

I agree that issuing a warning and continuing may be a better approach.

[matsmcp]

The problem I see with the proposed 7.,4 behavior is that it introduces a breaking change. A script that runs in 7.3 now fails. While the script could be fixed in 7.4.,x (ie add the -AllowInsecureRedirect parameter( but then that script no longer works in 7.3.

I agree that issuing a warning and continuing may be a better approach.

+1

As it stands -we need the SSH options enhancement that is in 7.3 and we want it in a LTS release (aka 7.4). We do not want breaking changes preventing a smooth upgrade from 7.3 -> 7.4

[doctordns]

@matsmcp Unless I am mistaken, the SSH option enhancements you mention in 7.3 should be in 7.4.

I have not tested it, but what happens when this code is on PowerShell 7.2? Does it work the same way as in 7.3? If so. this also be a breaking change for 7.,2 to 7.,4 upgrades (ie LTS to LTS)? I would not be in favour of adding more breaking changes if it can be avoided (and it feels like it can). Especially in an LTS to LTS update.

[crlogic]

what happens when this code is on PowerShell 7.2? Does it work the same way as in 7.3?

This code has gone through the lineage of 7.0 through latest 7.3. I didn't go all the way back in specific testing, here is 7.2.10

Eg. 1

Eg. 2

[crlogic]

Adding more detail should the PR be revisited in the future.

Further investigation into the symptoms revealed that my focus on tunnel/proxy/fw may have been pre-mature. I primarily work remotely via tunnels that require 10+ hops to the API endpoint. And the way Axis/ZTNA appear to work is that they highjack IP/name-resolution as any type of Resolve-DNSName does not return true address information. Therefore I RDP'd directly to the local (L2) subnet of a suspect API endpoint to repeat the testing above and received the same results.

This made me wonder if it was actually the API doing something funny so I tested a half-dozen other API's and found some that actually did work w/ 7.4.0-preview2. And some that always failed, ala Dell iDrac & Dell OME, regardless of execution proximity to endpoint.

[matsmcp]

@matsmcp Unless I am mistaken, the SSH option enhancements you mention in 7.3 should be in 7.4.

Yes it should be but if we get a breaking change like the one being discussed here, in worst case we cant upgrade meaning we are stuck on 7.3 (at least until the scripts can be fixed). I guess you could do an if on the version and adjust the call based on that but - thanks - no thanks ;)

[daxian-dbw]

@crlogic Now that @CarloToso has a potential fix #19468, can you please try out the private build produced from that PR and confirm whether it actually works for your scenario?

You can download the build.zip package from https://dev.azure.com/powershell/PowerShell/_build/results?buildId=127283&view=artifacts&pathAsName=false&type=publishedArtifacts

After you download, please remember to unblock the build.zip file before unzip it, otherwise you won't be able to start the pwsh.exe:

[crlogic]

@daxian-dbw - three good test results below.

1) using DellOpenMange module which is a script-based module around Invoke-WebRequest

2) Invoke-RestMethod to same Dell OpenManage API

3) Invoke-RestMethod to a Dell iDrac Redfish API

[daxian-dbw]

@crlogic Thank you for confirming the fix works! Appreciate your help.

[ghost]

:tada:This issue was addressed in #19468, which has now been successfully released as v7.4.0-preview.3.:tada:

Handy links:

• Release Notes