search

binref / refinery

High Octane Triage Analysis
Other
618 stars 62 forks source link

The PowerShell Problem #5

Closed huettenhain closed 7 months ago

huettenhain commented 3 years ago

In fcc36d8aa9bebf779a07d7363359fa01dc3a684b, I added some extremely weak PowerShell support, this was improved slightly in 504f0155ad6f91536085e27f63f4d178af9591ab. The main issues are the following:

  1. https://github.com/PowerShell/PowerShell/issues/1908
  2. https://github.com/PowerShell/PowerShell/issues/559

Our current workaround is to:

  1. wrap all output as hex with a magic prefix when running under PowerShell
  2. unwrap all input when we are running under PowerShell and the magic prefix is detected
  3. simply accept the fact that PowerShell has no streaming stdin/stdout

If either of the above two PowerShell issues are ever resolved in a way that supports the binary refinery design, the workaround should be removed because it introduces an unnecessary encoding step.

Ghost-Terms commented 2 years ago

You could add support for Use-RawPipeline in the meantime.

jhhcs commented 2 years ago

Hey @ImportTaste, the way I understand Use-RawPipeline, it should already work. I have, however, not tried it out myself. If you already tested it, are there any issues to report?

cxiao commented 2 years ago

Doing some extremely simple tests with Use-RawPipeline, it does seem to work as expected in providing a pipeline for raw binary data.

Use-RawPipeline provides the following main cmdlets:

There are also cmdlets to read raw binary streams from file, and dump them again back to file.

PS> run emit M7EwMzVzBkI3IwNTczM3cyMg2wQA | run b64 | run zl | run hex | 2ps
Hello World

PS> run emit "Once upon a time, at the foot of a great mountain ..." | run aes "pbkdf2[32,s4lty]:swordfish" --iv md5:X -R | run ccp md5:X | run aes "pbkdf2[32,s4lty]:swordfish" --iv cut:0:16 | 2ps
Once upon a time, at the foot of a great mountain ...

PS> run emit .\CyberChef_v9.32.3.zip | run xtzip "["| run dump "cyberchef_extracted/{path}" "]" | 2ps
PS> Get-ChildItem .\cyberchef_extracted\ -Name
assets
images
modules
ChefWorker.js.LICENSE.txt
CyberChef_v9.32.3.html
DishWorker.js.LICENSE.txt
InputWorker.js.LICENSE.txt
LoaderWorker.js.LICENSE.txt
ZipWorker.js.LICENSE.txt

Of course, certain characters which Powershell uses as part of its shell language, such as [], {}, and ,, must be quoted.

BTW, I think that my tests above bypass your PowerShell check in is_powershell_process, as I am running PowerShell 7.1.3; all PowerShell versions after 6 have the process name pwsh.exe, rather than powershell.exe which is_powershell_process checks for. Therefore, I think the tests above are not using the JSON serialization support, and it is Use-RawPipeline actually doing the work of making the streams work properly. Trying to use refinery without Use-RawPipeline definitely doesn't work at least, for me:

> emit M7EwMzVzBkI3IwNTczM3cyMg2wQA | b64 | zl | hex
(00:18:30) failure in zl: exception of type ValueError; could not detect any zlib stream.
huettenhain commented 2 years ago

Alright. Still not sure how to handle this in the most graceful manner. I can fix is_powershell_process to include a check for pwsh.exe, but what I would like best would be to disable the horrible JSON encoding alltogether, it only works inside a frame anyway. Then, log a warning when running inside PowerShell, unless the user is using Use-RawPipeline. However, I have no way of knowing whether that is the case as far as I can see.

As a side note, when I work with all-text input/output, I don't always have to quote [ and ] and I don't think I have fully grokked when that is the case:

PS C:\> emit Foo Bar [| ccp x | sep ]
xFoo
xBar
PS C:\> emit Foo Bar [| ccp x ]]
usage: ccp [-h] [-L] [-Q] [-0] [-v] data
ccp: error: unrecognized arguments: ]
cxiao commented 2 years ago

Yes, I don't think there's any way for binref units to inspect their environment in such a way that they can tell when they are being run under Use-RawPipeline's Inspect-NativeCommand.

Another way to handle this would be to display a very prominent warning as a post-install step on Windows systems spelling out the issues with piping binary data in Powershell, and either listing the workarounds (use cmd.exe, use Use-RawPipeline, or accept the current JSON-serialization-based Powershell support), or just linking users to this issue. Is it possible to do this from setup.py?

cxiao commented 2 years ago

Also, Powershell versions after 6 are now available on Mac and Linux, and officially supported. I think the number of people who actually use it as their day-to-day interactive shell on those platforms is extremely small, though, and the people who are doing that likely know the limitations.

Ghost-Terms commented 2 years ago

Yes, I don't think there's any way for binref units to inspect their environment in such a way that they can tell when they are being run under Use-RawPipeline's Inspect-NativeCommand.

Another way to handle this would be to display a very prominent warning as a post-install step on Windows systems spelling out the issues with piping binary data in Powershell, and either listing the workarounds (use cmd.exe, use Use-RawPipeline, or accept the current JSON-serialization-based Powershell support), or just linking users to this issue. Is it possible to do this from setup.py?

It's easy enough to tell if the module is loaded through Get-Module. If it is loaded, it just needs to run different commands.

jhhcs commented 2 years ago

I don't think there is any way to run Get-Module from inside Python, and I don't think there is any way for us to find out whether Use-RawPipeline is being used or not. I don't think there is a lot of benefit in supporting it; PowerShell either supports binary pipelines, or it does not - currently, it does not.

Right now I am leaning heavily towards treating a PowerShell parent process as a critical error, i.e. abort execution with a short explanation of why that is currently not supported. There are just too many ways this can go wrong unless our band-aid is completely flawless, and I am not confident I can make it so.

jhhcs commented 2 years ago

Alright, I had another idea about a better band-aid yesterday, and implemented it in 03436e296366f5cfb5cd71dcde329535dd6fc6cb. This works a lot better now, and there is a warning displayed by default which can be disabled through an environment variable:

image

jhhcs commented 2 years ago

Of course, this now has the unfortunate side-effect that Use-RawPipeline does not work at all:

image

Ghost-Terms commented 2 years ago

Of course, this now has the unfortunate side-effect that Use-RawPipeline does not work at all:

image

How about an environment variable to use a Use-RawPipeline compatible method, then?

jhhcs commented 2 years ago

Oh yes, I like that! Not sure why I had not thought of that before.. 😅

jhhcs commented 2 years ago

This was implemented in 504f0155ad6f91536085e27f63f4d178af9591ab. I have changed the names of the variables a bit and added information to the warning message:

(venv) PS> run emit c0019718c4d4538452affb97c70d16b7af3e4816d059010c277c4e579075c944 `
>> |run perc SETTINGS  [ `
>> |    run put keylen cut::1 `
>> |    run rc4 cut::keylen `
>> |    run xtp socket ]`
>> |2ps
WARNING: PowerShell has no support for binary pipelines or streaming. Binary Refinery uses an unreliable and slow 
workaround: It is strongly recommended to use the command processor instead. Proceed at your own peril!
- To silence this warning: $env:REFINERY_SILENCE_PS1_WARNING=1
- To disable the band-aid: $env:REFINERY_DISABLE_PS1_BANDAID=1
- To get more information: https://github.com/binref/refinery/issues/5
[BRPS1]:72656D6D2E6475636B646E732E6F72673A37303037
(venv) PS> $env:REFINERY_DISABLE_PS1_BANDAID=1
(venv) PS> run emit c0019718c4d4538452affb97c70d16b7af3e4816d059010c277c4e579075c944 `
>> |run perc SETTINGS  [ `
>> |    run put keylen cut::1 `
>> |    run rc4 cut::keylen `
>> |    run xtp socket ]`
>> |2ps
remm.duckdns.org:7007
(venv) PS>

I will push out a new release now.

jhhcs commented 1 year ago

There is a PR open in PowerShell that would (hopefully) fix this: https://github.com/PowerShell/PowerShell/pull/17857

huettenhain commented 1 year ago

Alright! The PR that fixes this in PowerShell has been merged, the next PowerShell preview should support this. I am very excited to try that out.

jhhcs commented 1 year ago

Noting from https://github.com/PowerShell/PowerShell/issues/1908#issuecomment-1577142452 that the following command will be required to enable this experimental PowerShell feature:

Enable-ExperimentalFeature PSNativeCommandPreserveBytePipe
huettenhain commented 1 year ago

In PowerShell 7.4.0-preview.4, refinery now works as expected after running:

Since it will probably be quite a while before this is in an mainstream PowerShell release, I'm afraid we have to keep this issue open - but at least there is now a way to make it work inside some version of PowerShell.

image

huettenhain commented 7 months ago

Alright, this is now fully supported in PowerShell 7.4. 351e953215066400ed4de78c54d99fd59e342727 adds somewhat hacky code to support this, but I do consider the issue closed at this point since refinery can now work in PowerShell simply by installing the latest version.