search

Ne0nd0g / merlin

Merlin is a cross-platform post-exploitation HTTP/2 Command & Control server and agent written in golang.
GNU General Public License v3.0
5.03k stars 798 forks source link

Character code errors in multi-byte character environments #148

Closed n-etupirka closed 8 months ago

n-etupirka commented 9 months ago

In Japan, multi-byte characters are used. If the output of a command executed by the shell command contains multi-byte characters, an error will result.

Prerequisite

Environment Data

Expected Behavior

The output of the command is displayed on the server.

Actual Behavior

An error is displayed.

[!] 2023-11-30T07:31:11Z there was an error receiving a message from the server stream: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8

Steps to Reproduce Behavior

1. PS C:\> .\merlinAgent-Windows-x64-Debug.exe -proto http -psk ***** -sleep 5s -url http://*.*.*.*:80 -v
2. Merlin>> interact <AGENT GUID>
3. Merlin>> shell whoami //success (only in English)
4. Merlin>> shell whoami /priv //error (Japanese is included)

Misc Information

Ne0nd0g commented 9 months ago

@n-etupirka thanks for reporting this. I'll take a look and see what I need to do to resolve the problem.

Ne0nd0g commented 9 months ago

@n-etupirka Can you share what the output of whoami /priv looks like on the endpoint? I changed my Windows 11 VM's primary language to Japanese and see this:

[+] 2023-12-05T13:32:47Z Created C:\WINDOWS\system32\cmd.exe process with an ID of 2380

PRIVILEGES INFORMATION
----------------------

Privilege Name                Description                          State   
============================= ==================================== ========
SeShutdownPrivilege           Shut down the system                 Disabled
SeChangeNotifyPrivilege       Bypass traverse checking             Enabled 
SeUndockPrivilege             Remove computer from docking station Disabled
SeIncreaseWorkingSetPrivilege Increase a process working set       Disabled
SeTimeZonePrivilege           Change the time zone                 Disabled
Ne0nd0g commented 9 months ago

It looks like I'm able to send Japanese characters back and forth without problem

Merlin[agent][fd05e3d3-bbb3-4697-b766-66befed0825b]» ls C:\\Users\\Rick\\Downloads
Merlin[agent][fd05e3d3-bbb3-4697-b766-66befed0825b]»  
[-] 2023-12-05T13:38:42Z Created job fcDFUpydUL for agent fd05e3d3-bbb3-4697-b766-66befed0825b at 2023-12-05T13:38:42Z

[-] 2023-12-05T13:38:58Z Results of job fcDFUpydUL for agent fd05e3d3-bbb3-4697-b766-66befed0825b at 2023-12-05T13:38:58Z

[+] 2023-12-05T13:38:58Z Directory listing for: C:\Users\Rick\Downloads

-rw-rw-rw-      2023-12-01 09:23:24     282     desktop.ini
-rw-rw-rw-      2023-12-05 08:35:45     6       テスト.txt
n-etupirka commented 9 months ago

@Ne0nd0g The result of whoami /priv in the Japanese environment Windows 11 is output like this.

C:\Users\pentester>whoami /priv

PRIVILEGES INFORMATION
----------------------

特権名                        説明                                            状態
============================= =============================================== ====
SeShutdownPrivilege           システムのシャットダウン                        無効
SeChangeNotifyPrivilege       走査チェックのバイパス                          有効
SeUndockPrivilege             ドッキング ステーションからコンピューターを削除 無効
SeIncreaseWorkingSetPrivilege プロセス ワーキング セットの増加                無効
SeTimeZonePrivilege           タイム ゾーンの変更                             無効

This may be accomplished by setting not only the primary language to Japanese, but also the system locale to Japanese. (Settings -> Time & language -> Language & region -> Administrative language settings -> Change system locale... -> Japanese) スクリーンショット 2023-12-06 151052

n-etupirka commented 9 months ago

@Ne0nd0g The ls command, which is built into Merlin, did not cause the error. (The second image is Agent's debug log.) スクリーンショット 2023-12-06 152909

スクリーンショット 2023-12-06 153043

However, when I run dir from a shell command, I get an error. スクリーンショット 2023-12-06 153001

スクリーンショット 2023-12-06 153111

This is the correct output of the dir command at the endpoint.

C:\Users\pentester>dir C:\Users\pentester\Downloads
 ドライブ C のボリューム ラベルがありません。
 ボリューム シリアル番号は 1A28-6E18 です

 C:\Users\pentester\Downloads のディレクトリ

2023/12/06  15:27    <DIR>          .
2023/12/06  14:11    <DIR>          ..
2023/12/06  13:52                 9 テスト.txt
               1 個のファイル                   9 バイト
               2 個のディレクトリ  107,650,621,440 バイトの空き領域
Ne0nd0g commented 9 months ago

I just wanted to follow up and say that I'm able to duplicate the problem and I'm working on a fix.

Ne0nd0g commented 9 months ago

OK, I have a fix now. It's brittle, but it works. I check to make sure the output is UTF-8. If it isn't, I check the active code page. In this case, the code page is 932 for ShifJIS. I'm then able to decode the data to that encoding format. It is brittle because I've only handled decoding for UTF-8 and ShiftJIS. Anything else will just replace all invalid UTF-8 characters with this character: �. If others need their handling included for a different format, I will add it at that time.

The fix for handling ShiftJS is in the https://github.com/Ne0nd0g/merlin-agent repository. I still need to fix the fact that the gRPC stream broke and did not fix itself.

Ne0nd0g commented 9 months ago

@n-etupirka Are you comfortable building the Merlin Agent from source code using the dev branch to see if everything is working as expected? https://github.com/Ne0nd0g/merlin-agent/tree/dev

n-etupirka commented 9 months ago

@Ne0nd0g I have confirmed that everything is working as expected in the Japanese environment! スクリーンショット 2023-12-10 190035 Thank you for providing a great tool and great response.

Incidentally, are there any plans to fix the bug that breaks gRPC streams in languages other than UTF-8 and ShiftJIS?

Ne0nd0g commented 9 months ago

Yes, I do plan to fix the bug that broke gRPC. There are two different problems. One is that the gRPC stream received non UTF-8 text. On the Agent, I now detect non UTF-8 and convert it to UTF-8. The second is that the gRPC stream didn't recover. I'll work on that next.

Ne0nd0g commented 9 months ago

I'm going to use this issue to fix the gRPC error too.

Ne0nd0g commented 9 months ago

Updated merlin-cli to try and recover broken gRPC message stream identified in this issue https://github.com/Ne0nd0g/merlin-cli/commit/e4ef7b93fc754aeb8e0a23eafe62e51f11c18ed9

n-etupirka commented 8 months ago

I tried it in the following version, and confirmed that even in a non UTF-8 environment, no errors occurred and it was replaced with "�".

Merlin[agent][5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313]»
[-] 2023-12-15T07:58:13Z Created job uawOwEVDRd for agent 5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313 at 2023-12-15T07:58:13Z

[-] 2023-12-15T07:58:23Z Results of job uawOwEVDRd for agent 5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313 at 2023-12-15T07:58:23Z

[+] 2023-12-15T07:58:23Z Created C:\Windows\system32\cmd.exe process with an ID of 10716

***The output was not valid UTF-8 and there isn't a configured decoder for code page 936***

�Ȩ�Ϣ
----------------------

�Ȩ�                        �                 ״̬
============================= ==================== ======
SeShutdownPrivilege           �ر�ϵͳ             �ѽ�
SeChangeNotifyPrivilege       �ƹ�         �
SeUndockPrivilege             �չ�ȡ�¼� �ѽ�
SeIncreaseWorkingSetPrivilege �ӽ�̹�       �ѽ�
SeTimeZonePrivilege           �ʱ�             �ѽ�

Merlin[agent][5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313]»
Ne0nd0g commented 8 months ago

Awesome, thank you for trying it out. I added support for simplified and traditional Chinese as well as Korean in this commit if you want to try that out as well https://github.com/Ne0nd0g/merlin-agent/commit/4c60e45c18722b5dde357e68be492636df5ed972