search

PowerShell / PowerShell

PowerShell for every system!
https://microsoft.com/PowerShell
MIT License
43.55k stars 7.06k forks source link

Invoke-RestMethod 7.4 not working with charset, multipart/form-data for .pfx certificate and target device doesn't seem to accept charset. PS 7.3 works. #21604

Closed ray-wn closed 1 week ago

ray-wn commented 2 weeks ago

Prerequisites

Steps to reproduce

I have tried several attempts over countless hours, and nothing seems to work. I've also read dozens of Issues (open and closed).

I'm uploading a multipart/form-data certificate (.pfx) from a decoded Base64 Latin1/ISO-8895-1 string. Setting charset just returns an error code. I've had the same issue with another device but setting -ContentType="application/x-pkcs12;boundary=someBoundary;charset=iso-8895-1" worked. but for this device nothing works. Can't upload it as UTF-8 and can't pass charset. I guess e.g. Chrome detects the form-data/file's encoding and submits it accordingly or it uses ISO-8859-1/ASCII.

Edit (May 4): It has to do with the form-data body being submitted as UTF-8 (and not about the .pfx bytes' encoding).

I don't have access to modify the device and I don't see any way to force the old pwsh behavior.

Expected behavior

Any way to force previous behavior or detect charset/encoding automatically.

Actual behavior

No way for to use PS 7.4 when device doesn't accept charset.

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

rhubarb-geek-nz commented 2 weeks ago

A PFX certificates is PKCS12 format data which is ASN.1 which in turn is binary. There should be no character set involved because there is nothing to translate. The Base64 data was text, when you decode Base64 you have an octet-stream or byte array. Your mime type looks correct, what is the charset for?

ray-wn commented 2 weeks ago

A PFX certificates is PKCS12 format data which is ASN.1 which in turn is binary. There should be no character set involved because there is nothing to translate. The Base64 data was text, when you decode Base64 you have an octet-stream or byte array. Your mime type looks correct, what is the charset for?

I pass the file data like:

$boundary = 'someBoundary'
 $body = (
            "--$boundary",
            "Content-Disposition: form-data; name=`"PWD`"$LF",
            $certPw,
            "--$boundary",
            "Content-Disposition: form-data; name=`"FILE`"; filename=`"certificate.pfx`"",
            "Content-Type: application/x-pkcs12$LF",
            [System.Text.Encoding]::Latin1.GetString($bytes),
            "--$boundary--$LF"
        ) -join $LF

        $response = Invoke-RestMethod -ContentType "multipart/form-data; boundary=$boundary" -Uri $certupload_uri -Method Post -Body $body -WebSession $session -ErrorAction Stop

...of course I also tried the same with $mulitpart = [System.Net.Http.MultipartFormDataContent]::new()

I mean, when I see the sent form-data it's now UTF-8 encoded, which doesn't work, so setting it to Latin1 should correctly send the data with ISO-8895-1 encoding. As mentioned, that's also what happens when I manually upload it. For the other devices setting charset=iso-8895-1 correctly POSTed the data with that encoding and in Fiddler I see the content is correctly encoded (ANSI, when viewing in Notepad) which is obviously not the case with 7.4, as it defaults to UTF-8.

Edit: Here the multipart version, same issue

$bytes = [System.Convert]::FromBase64String('MII....=')

$FileHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new('form-data')
$FileHeader.Name = 'FILE'
$FileHeader.FileName = 'certificate.pfx'
$FileContent = [System.Net.Http.StreamContent]::new([System.IO.MemoryStream]::new($bytes))
$FileContent.Headers.ContentDisposition = $FileHeader
$FileContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse('application/x-pkcs12')

$MultipartContent = [System.Net.Http.MultipartFormDataContent]::new()
$MultipartContent.Add([System.Net.Http.StringContent]::new($certPw), 'PWD')
$MultipartContent.Add($FileContent)

So, charset to send the body data as Latin1.

rhubarb-geek-nz commented 2 weeks ago

I must admit I am not familiar with multipart/form-data

This looks wrong

[System.Text.Encoding]::Latin1.GetString($bytes)

The bytes are bytes and should not be converted, they should stay as bytes. I don't know how the length of the mime data within each part is encoded.

My example

$ od -t x1  random.bin
0000000 0f cc eb 1c bc b7 30 8f aa aa bc 40 77 d4 a7 65
0000020 9c 5a fb 35 ee b2 1e bf af 89 ea 58 94 8d 5b 2b
0000040
$ curl --trace - --user guest:changeit --form "file=@random.bin" --fail http://localhost:8080
== Info:   Trying 127.0.0.1:8080...
== Info: Connected to localhost (127.0.0.1) port 8080 (#0)
== Info: Server auth using Basic with user 'guest'
=> Send header, 229 bytes (0xe5)
0000: 50 4f 53 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d POST / HTTP/1.1.
0010: 0a 48 6f 73 74 3a 20 6c 6f 63 61 6c 68 6f 73 74 .Host: localhost
0020: 3a 38 30 38 30 0d 0a 41 75 74 68 6f 72 69 7a 61 :8080..Authoriza
0030: 74 69 6f 6e 3a 20 42 61 73 69 63 20 5a 33 56 6c tion: Basic Z3Vl
0040: 63 33 51 36 59 32 68 68 62 6d 64 6c 61 58 51 3d c3Q6Y2hhbmdlaXQ=
0050: 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 ..User-Agent: cu
0060: 72 6c 2f 37 2e 38 31 2e 30 0d 0a 41 63 63 65 70 rl/7.81.0..Accep
0070: 74 3a 20 2a 2f 2a 0d 0a 43 6f 6e 74 65 6e 74 2d t: */*..Content-
0080: 4c 65 6e 67 74 68 3a 20 32 33 34 0d 0a 43 6f 6e Length: 234..Con
0090: 74 65 6e 74 2d 54 79 70 65 3a 20 6d 75 6c 74 69 tent-Type: multi
00a0: 70 61 72 74 2f 66 6f 72 6d 2d 64 61 74 61 3b 20 part/form-data;
00b0: 62 6f 75 6e 64 61 72 79 3d 2d 2d 2d 2d 2d 2d 2d boundary=-------
00c0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
00d0: 2d 62 39 38 64 33 38 39 63 64 62 34 62 35 32 64 -b98d389cdb4b52d
00e0: 32 0d 0a 0d 0a                                  2....
=> Send data, 234 bytes (0xea)
0000: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
0010: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 62 39 38 64 33 38 ----------b98d38
0020: 39 63 64 62 34 62 35 32 64 32 0d 0a 43 6f 6e 74 9cdb4b52d2..Cont
0030: 65 6e 74 2d 44 69 73 70 6f 73 69 74 69 6f 6e 3a ent-Disposition:
0040: 20 66 6f 72 6d 2d 64 61 74 61 3b 20 6e 61 6d 65  form-data; name
0050: 3d 22 66 69 6c 65 22 3b 20 66 69 6c 65 6e 61 6d ="file"; filenam
0060: 65 3d 22 72 61 6e 64 6f 6d 2e 62 69 6e 22 0d 0a e="random.bin"..
0070: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
0080: 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d plication/octet-
0090: 73 74 72 65 61 6d 0d 0a 0d 0a 0f cc eb 1c bc b7 stream..........
00a0: 30 8f aa aa bc 40 77 d4 a7 65 9c 5a fb 35 ee b2 0....@w..e.Z.5..
00b0: 1e bf af 89 ea 58 94 8d 5b 2b 0d 0a 2d 2d 2d 2d .....X..[+..----
00c0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
00d0: 2d 2d 2d 2d 2d 2d 62 39 38 64 33 38 39 63 64 62 ------b98d389cdb
00e0: 34 62 35 32 64 32 2d 2d 0d 0a                   4b52d2--..

I can see the 32 bytes of my file random.bin in the content from 009A to 00B9 but I don't know how it encodes that there are 32 bytes there. I can see that where is a boundary immediately following.

Given you are trying to pass binary data I don't think you can create the body with string appends. I think you may need something like MultipartFormDataContent Class

Eg how-to-send-multipart-form-data-with-powershell-invoke-restmethod

This is my PowerShell version using .NET

param(
        [parameter(Mandatory=$true)]
        $Path
)

$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'

trap
{
        throw $PSItem
}

if ( Test-Path -LiteralPath $Path -PathType Leaf )
{
        $web = New-Object -Type 'System.Net.WebClient'

        try
        {
                $web.Headers["Authorization"] = "Basic Z3Vlc3Q6Y2hhbmdlaXQ="

                $response = $web.UploadFile('http://localhost:8080',$Path)
        }
        finally
        {
                $web.Dispose()
        }
}
else
{
        Write-Host "Not uploading $Path as it does not exist"
}
jborean93 commented 2 weeks ago

https://github.com/PowerShell/PowerShell/issues/21386#issuecomment-2023863985 Has an example of building form data for Invoke-WebRequest, specifically with raw file bytes.

ray-wn commented 2 weeks ago

Thank you both for the responses and examples!

rhubarb-geek-nz commented 2 weeks ago

it expects the bytes to be Latin1.

The end system doesn't expect the bytes to be Latin1, it expects them to just be bytes. Raw bytes have no character encoding, they are just eight bits. Character-set encoding only applies to text. What has changed is that PowerShell's default character set is now UTF8, and you may have previously been lucky with the encoding and it was a one to one mapping, but really there should be no mapping at all. I suggest following the link above provided by @jborean93

ray-wn commented 2 weeks ago

Thanks for the clarification, that clears that part up for me! What that really means is that the form-data submitted needs to be Latin1 (and the file bytes within the form-data has nothing to do with the issue). The issue still stands but doesn't specifically relate to the .pfx bytes.

Note that when base64 decoding the bytes and saving the file with UTF-8, uploading the .pfx manually doesn't work, this is probably because then Chrome will see the file was saved with UTF-8 encoding and post the form-data with UTF-8 encoding so my confusion was that I thought it had to do with the bytes but it has to do with the form-data itself, regardless of the bytes.

What I still don't understand is what to do with the link, when I wrote in my first answer that it's the exact same issue, the device returns a JSON with the exact same error code. Here my code again, as in the first answer, using mulitpart-form with just bytes, exact same issue. Maybe I should delete the Latin1 part from my answers and just post the multipart-form version, as the issue is I can't send charset but the body may not be UTF-8 encoded:

$bytes = [System.Convert]::FromBase64String('MII....=')

$FileHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new('form-data')
$FileHeader.Name = 'FILE'
$FileHeader.FileName = 'certificate.pfx'
$FileContent = [System.Net.Http.StreamContent]::new([System.IO.MemoryStream]::new($bytes))
$FileContent.Headers.ContentDisposition = $FileHeader
$FileContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse('application/x-pkcs12')

$MultipartContent = [System.Net.Http.MultipartFormDataContent]::new()
$MultipartContent.Add([System.Net.Http.StringContent]::new($certPw), 'PWD')
$MultipartContent.Add($FileContent)

Invoke-RestMethod -Body $MultipartContent ....
rhubarb-geek-nz commented 2 weeks ago

If I was going to try and solve this, I would get to the basics.

Can you do the equivalent with curl ( I mean real curl, not curl the PowerShell 5.1 alias)?

Something like

Set-Content -LiteralPath 'certificate.pfx' -Value $bytes -AsByteStream
curl --trace - --user XXX:YYYY --form "file=@certificate.pfx" --fail http://yoururl

Can you use Invoke-WebRequest to do the post to make sure the Cmdlet is not doing any extra translation of the data body?

ray-wn commented 2 weeks ago

Thanks a lot for all your help so far!

I will definitely try your suggestions, but it'll be on Monday, and comment. I'll try the MultipartForm version too, but am certain that'll work with 7.3. I'll definitely try the Invoke-WebRequest and see about curl, although I don't know curl very well and I will need to pass session data after logging in to the device.

rhubarb-geek-nz commented 2 weeks ago

I notice that the MultipartFormDataContent has a mechanism to convert the whole encoding to a byte stream, I think you need to be sending those bytes exactly as is through Invoke-WebRequest.

mklement0 commented 2 weeks ago

I haven't followed this thread in detail, so there may be a better solution, but note that you can pass a [byte[]] array directly to the -Body argument, so to emulate the 7.3- behavior you can try to ISO-8859-1-encode the entire $body string:

# Create the $body text as an ISO-8859-1-encoded [byte[]] array.
$body = [System.Text.Encoding]::Latin1.GetBytes((
    "--$boundary",
    "Content-Disposition: form-data; name=`"PWD`"`n",
    $certPw,
    "--$boundary",
    "Content-Disposition: form-data; name=`"FILE`"; filename=`"certificate.pfx`"",
    "Content-Type: application/x-pkcs12`n",
    [System.Text.Encoding]::Latin1.GetString($bytes),
    "--$boundary--`n"
  ) -join "`n")

Note that any single-byte, 8-bit, fixed-width text encoding (all of whose code points must be valid) can be used to losslessly encode arbitrary binary data.

ISO-8859-1 encoding (which is closely related to - but not the same as - Windows-1252) is special in that it is a true subset of Unicode, covering the 8-bit range of Unicode code points (� (NULL, U+0000) through ÿ (LATIN SMALL LETTER Y WITH DIAERESIS, U+00FF)

As such, you can cast arbitrary binary data to (the characters of) a .NET string, and the Unicode code points of the resulting characters (which are technically are Unicode code units, i.e. [uint16] values (unsigned 16-bit integers) that situationally, but rarely, require a complementary, second code unit, with two such code units forming a so-called surrogate pair) will reflect the original byte values.

# Arbitrary binary data (array of bytes)
[byte[]] $bytes = 0..255

# Interpret the byte values as Unicode code units ([char]) and construct a .NET string
# from them (which is always UTF-16LE).
$str = [string] [char[]] $bytes
rhubarb-geek-nz commented 2 weeks ago

This is my solution, you need $path and $uri set up, My credentials are just as an example. This has no text or character sets. Everything is done using System.Net.Http objects to do the correct encoding.

Set up credentials

[string]$userName = 'guest'
[string]$userPassword = 'changeit'
[securestring]$secStringPassword = ConvertTo-SecureString $userPassword -AsPlainText -Force
[pscredential]$credObject = New-Object System.Management.Automation.PSCredential -ArgumentList $userName, $secStringPassword

Read file into memory and get name and content type

[byte[]]$bytes = Get-Content -LiteralPath $Path -AsByteStream -Raw
$name = (Get-Item -LiteralPath $Path).Name
[string]$contentType = 'application/octet-stream'

Create the multipart body

$byteArrayContent = New-Object -TypeName 'System.Net.Http.ByteArrayContent' -ArgumentList @(,$bytes)
$byteArrayContent.Headers.ContentType = New-Object -TypeName 'System.Net.Http.Headers.MediaTypeHeaderValue' -ArgumentList $contentType
$multipartFormDataContent = New-Object -TypeName 'System.Net.Http.MultipartFormDataContent'
$multipartFormDataContent.Add($byteArrayContent,'FILE',$name)

Get bytes and contentType from the multipart content

$memory = New-Object -TypeName 'System.IO.MemoryStream'
$multipartFormDataContent.ReadAsStream().CopyTo($memory)
$bytes = $memory.ToArray()
$contentType = $multipartFormDataContent.Headers.ContentType

Finally do the post with Invoke-WebRequest

Invoke-WebRequest -Credential $credObject-Uri $Uri -Method 'POST' -Body $bytes -ContentType $contentType

In my case I get a 201 created or a 204 no content as a successful response.

Alternatively use Invoke-RestMethod

Invoke-RestMethod -Credential $credObject-Uri $Uri -Method 'POST' -Body $bytes -ContentType $contentType
ray-wn commented 1 week ago

You guys are the best! It worked!

@mklement0 It worked as you mentioned. I had to save the bytes to a variable first. For some reason I couldn't get doing the GetBytes within the Invoke-RestMethod to work, and actually tried that before too.

@rhubarb-geek-nz I tried quite a while but couldn't fully get it to work, but super helpful for using the MultipartFormDataContent. I tried to use your example with the solution of getting the bytes from Latin1. I think the issue is that to pass the body I just need the MultipartFormDataContent Content and not the headers etc., so not sure getting the bytes returns other stuff too? I tried removing the headers too, but that didn't fully work. I guess I could use the enumerator and then parse the bytes, but for now I'm just happy I have a solution.

Thank you both very very much!

microsoft-github-policy-service[bot] commented 1 week ago

📣 Hey @ray-wn, how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://aka.ms/PSRepoFeedback

Microsoft Forms
mklement0 commented 1 week ago

Glad to hear it helped, @ray-wn (I've updated my previous comment to amend the background information about encodings).

You don't strictly a need an intermediate variable, though, so I suspect you merely had a syntax problem, which is presumably easily remedied by enclosing the [System.Text.Encoding]::Latin1.GetBytes() call in (...); a simplified example:

# Note the (...) around the [System.Text.Encoding]::Latin1.GetBytes() call.
Write-Host ([System.Text.Encoding]::Latin1.GetBytes('abc'))

In argument-parsing mode, [ is not a metacharacter; to force recognition of an expression such as a method call starting with [System.Text.Encoding] as such, enclosure in (...) is needed.

ray-wn commented 1 week ago

Thanks again @mklement0!!! You're absolutely right, I just retried it as I was certain that's what I did, but it worked. I have no idea what I must have done, as adding parentheses would be the first logical thing to do.

rhubarb-geek-nz commented 1 week ago

I tried to use your example with the solution of getting the bytes from Latin1.

I am glad you having something working and are happy with it.

I would still suggest that any solution that depends on encoding bytes with any character set and still expecting them to be treated as the correct bytes at the end of a chain is a 'hack'. Also the advantage of the MultipartFormDataContent is it handles all the boundaries.

For the record, this is my complete script which lets me transfer multiple files to my system in a single POST.

#!/usr/bin/env pwsh

param(
    [parameter(Mandatory=$true)]$Path,
    $Uri='http://wherever.it.goes'
)

$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'

trap
{
    throw $PSItem
}

[string]$userName = 'guest'
[string]$userPassword = 'changeit'
[securestring]$secStringPassword = ConvertTo-SecureString $userPassword -AsPlainText -Force
[pscredential]$credObject = New-Object System.Management.Automation.PSCredential -ArgumentList $userName, $secStringPassword

$multipartFormDataContent = New-Object -TypeName 'System.Net.Http.MultipartFormDataContent'

Get-Item -Path $path | ForEach-Object {
    if (Test-Path -Path $_ -Type Leaf)
    {
        $name =  $_.Name
        [byte[]]$bytes = Get-Content -LiteralPath $_ -AsByteStream -Raw
        $byteArrayContent = New-Object -TypeName 'System.Net.Http.ByteArrayContent' -ArgumentList @(,$bytes)
        $byteArrayContent.Headers.ContentType = New-Object -TypeName 'System.Net.Http.Headers.MediaTypeHeaderValue' -ArgumentList 'application/octet-stream'
        $multipartFormDataContent.Add($byteArrayContent,'FILE',$name)
    }
}

$memory = New-Object -TypeName 'System.IO.MemoryStream'
$multipartFormDataContent.ReadAsStream().CopyTo($memory)
$bytes = $memory.ToArray()
$contentType = $multipartFormDataContent.Headers.ContentType

Invoke-RestMethod -Credential $credObject -AllowUnencryptedAuthentication -Uri $Uri -Method 'POST' -Body $bytes -ContentType $contentType
ray-wn commented 1 week ago

@rhubarb-geek-nz You're totally right and I agree that this is the cleanest way. Right now I have other issues to resolve, but will definitely revisit it when I have a bit more time to look into it, but thanks a lot for the code and all your help!