Test whether appstraction and cyanoacrylate work on Windows (under WSL)

[baltpeter]

We are pretty sure that CA won't work on Windows due to the IPC mechanism used:

https://github.com/tweaselORG/cyanoacrylate/blob/b59010c79fa4939307b81bc58db0f52147e285fc/README.md?plain=1#L14

But we haven't actually verified that. And maybe it does work under WSL at least…

[baltpeter]

OK, I can confirm that an MRE for the IPC communication doesn't work on Windows.

index.js:

import { execa } from 'execa';

(async () => {
    const proc = execa('python', ['script.py'], { stdio: ['pipe', 'pipe', 'pipe', 'ipc'] });

    proc.on('message', (message) => {
        console.log(111, message);
    });

    const res = await proc;
    console.log(222, res);
})();

script.py:

import os
import json

ipcPipeFd = 3

res = os.write(ipcPipeFd, bytes(json.dumps("hi") + '\n', 'utf8'))

print(f"res: {res}")

This will never print 111 hi on Windows, even though it does on Linux.

[baltpeter]

Curiously, the Python process does print res: 5 to stdout though, which is the number of bytes written, indicating that it did write somewhere:

222 {
  command: 'python script.py',
  escapedCommand: 'python script.py',
  exitCode: 0,
  stdout: 'res: 5',
  stderr: '',
  all: undefined,
  failed: false,
  timedOut: false,
  isCanceled: false,
  killed: false
}

If I instead change ipcPipeFd to 4, the Python process errors with OSError: [Errno 9] Bad file descriptor, further suggesting that there is actually something listening at fd 3.

(Changing it to 1 or 2 prints to stdout or stderr, respectively.)

[baltpeter]

I got a little closer with https://www.npmjs.com/package/@kldzj/named-pipes, which is an abstraction over named pipes for both Linux and Windows. The script is mostly adapted from the README.

index-pipe.ts:

import { createNamedPipe } from '@kldzj/named-pipes';
import { execa } from 'execa';

const pipe = createNamedPipe();
console.log('Path to socket:', pipe.path);

const sender = pipe.createSender();
const receiver = pipe.createReceiver();

await sender.connect();
await receiver.connect();

receiver.on('data', (c) => console.log(c.toString()));
sender.once('connected', () => {
    execa('python', ['script-pipe.py', pipe.path]).then(() => pipe.destroy());
});

script-pipe.py:

import json
import sys

pipe_path = sys.argv[1]

with open(pipe_path, 'wb', buffering=0) as pipe:
    message = json.dumps("hi").encode('utf-8')
    pipe.write(message)

This still works fine on Linux and it correctly creates a pipe on Windows, but the Python code for writing to it is wrong.

[baltpeter]

A few things I tried, unsuccessfully:

• https://pypi.org/project/PyWPipe/ — I wasn't able to install that through pip
• https://gist.github.com/JonathonReinhart/bbfa618f9ad19e2ca48d5fd10914b069 / http://jonathonreinhart.blogspot.com/2012/12/named-pipes-between-c-and-python.html — that runs without errors but Node doesn't receive anything
• https://stackoverflow.com/a/50403960 using https://pypi.org/project/pywin32/ — That gives me an Access is denied. in CreateNamedPipe if I used the pipe created by Node. If I use a non-existent pipe, it hangs but doesn't error. I'm guessing it's trying to re-create the existing pipe, which doesn't work. I don't know how to tell it to open an existing pipe instead of creating one. The docs aren't exactly helpful.

[zner0L]

Seems like you did all the work I did again. When I tried this, I came to the conclusion, that the easiest and most elegant solution would be to just use libuv pipes in python as well, since this is what nodejs uses to create the pipes. This is also why you are able to write to the file descriptor. Somehow libuv creates a named pipe at this file descriptor in Windows, but I don't really understand how they write to it.

There is already python library for libuv: https://github.com/saghul/pyuv, but the maintainer doesn't really seem to care about it anymore https://github.com/saghul/pyuv/issues/276, so we can not really use it anymore sadly. But the part for the pipe is pretty simple and should be easy to reuse from pyuv.

[zner0L]

Are the pipes also failing on WSL, or did you only test them on pure Windows? I think they are not our only problem when porting to Windows. E.g. our paths are all formatted for UNIX operating systems.

[baltpeter]

I only tested Windows, not WSL.

E.g. our paths are all formatted for UNIX operating systems.

I don't think that should be a problem:

[baltpeter]

Also works fine in Python:

[zner0L]

OK, so I underestimated the amount of work necessary to get libuv binding working in python. Since I don't want to write another https://github.com/saghul/pyuv, we need to make a decision. pyuv fails to build against python 3.10, but that was fixed in https://github.com/saghul/pyuv/pull/275. However, this only fixes it for 3.10 for now, it breaks again on 3.11. The developer doesn't really seem so interested in it anymore, but I feel like the community is still behind using pyuv. So, we could just load the module from the master branch, as suggested here: https://github.com/saghul/pyuv/issues/276

However, we would depend on a fairly unstable module there, which I don't like, so I want your opinion, @baltpeter.

[zner0L]

The other option would be to implement some kind of complicated socket logic (Thank you for nothing, Windows. Why does everything need to be so complicated with you.).

[baltpeter]

Random thought: The reason we switched to the IPC mechanism in the first place was because we didn't receive the stdout from mitmproxy in time (maybe due to flushing problems).

Now that we use our own mitmproxy script anyway, wouldn't it make sense to check whether we can go back to stdout (with judicious flushing after every message) before implementing our own Windows layer thingy?

[zner0L]

We don't use our own mitmproxy script. I abandoned that because I realized it added no value. But I guess I could try that again, maybe that is the easiest way.

[baltpeter]

I meant the addon, not the script.

[baltpeter]

Should be possible to write to stdout from there as well, shouldn't it?

[zner0L]

I mean, I can just use os.write to another file descriptor.

[zner0L]

It seems like it works like that. We can just set --set ipcPipeFd=1 and the scripts send the messages via the stdout. which are flushed correctly. (It works on Linux and Windows, I just confirmed that.)

[baltpeter]

The structure of a venv on Windows and *nix are annoyingly quite different.

Linux:

├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── flask
│   ├── frida
│   ├── frida-apk
│   ├── frida-compile
│   ├── frida-create
│   ├── frida-discover
│   ├── frida-join
│   ├── frida-kill
│   ├── frida-ls
│   ├── frida-ls-devices
│   ├── frida-ps
│   ├── frida-pull
│   ├── frida-push
│   ├── frida-rm
│   ├── frida-trace
│   ├── litecli
│   ├── mitmdump
│   ├── mitmproxy
│   ├── mitmweb
│   ├── normalizer
│   ├── objection
│   ├── pip
│   ├── pip3
│   ├── pip3.10
│   ├── pygmentize
│   ├── pysemver
│   ├── python -> /usr/bin/python
│   ├── python3 -> python
│   ├── python3.10 -> python
│   ├── sqlformat
│   └── tabulate
├── include
├── lib
│   └── python3.10
│       └── site-packages
│           └── …
├── lib64 -> lib
└── pyvenv.cfg

Windows (no packages installed yet):

├───Include
├───Lib
│   └───site-packages
│       ├─── …
└───Scripts
│   └───activate
│   └───activate.bat
│   └───Activate.ps1
│   └───deactivate.bat
│   └───pip.exe
│   └───pip3.11.exe
│   └───pip3.exe
│   └───python.exe
│   └───pythonw.exe

[baltpeter]

Oops, accidentally posted these in the wrong issue:

For some reason, Parcel breaks os.platform (it removes the import and replaces it with an undefined identifier, very helpful). But process.platform works.

Windows just really wants to annoy you, doesn't it? Of course, PATH entries are delimited by ; instead of :. -.-

[baltpeter]

And here we have a commit just for Mr. "why don't you just grep?" @zner0L. :P

https://github.com/tweaselORG/appstraction/pull/61/commits/bed52e67e16266b4e153b945dcbafe624cc816e2

[baltpeter]

Oh joy. process.kill() "unconditionally terminates" a process on Windows (why can Node do that but taskmgr.exe can't? :P). That of course means that mitmdump can't write to the HAR file anymore.

[baltpeter]

Apparently, the solution is genuinely to call taskkill.exe manually. Wow.

There is a wrapper for that but I think we can manage on our own.

[baltpeter]

Are you kidding me, Windows? -.-

PS C:\Users\root> taskkill /pid 7956
ERROR: The process with PID 7956 could not be terminated.
Reason: This process can only be terminated forcefully (with /F option).

[baltpeter]

And, of course, using /f causes the same problem.

[baltpeter]

There is also a tskill, but that can only force-stop sigh:

PS C:\Users\root> tskill /?
Ends a process.

TSKILL processid | processname [/SERVER:servername] [/ID:sessionid | /A] [/V]

  processid           Process ID for the process to be terminated.
  processname         Process name to be terminated.
  /SERVER:servername  Server containing processID (default is current).
                         /ID or /A must be specified when using processname
                         and /SERVER
  /ID:sessionid       End process running under the specified session.
  /A                  End process running under ALL sessions.
  /V                  Display information about actions being performed.

[baltpeter]

Thank you kind strangers on the internet who and the same problem and wrote a literal friggin' native Rust module to send Ctrl + C to a process. <3

Thanks to you, it wööörks!

[baltpeter]

I am ever so slightly worried about the fact that no traffic was recorded for any of the apps I tried, though. .D

[baltpeter]

One problem: The first time I started mitmproxy, I dismissed the Windows Firewall dialog, assuming it would come back the next time. That was a mistake. It didn't and I subsequently forgot about it.

In "Windows Defender Firewall", I clicked "Allow an app or feature through Windows Defender Firewall", "Change Settings" and then set "Python" as allowed:

But that isn't enough. I think the HTTPS certificate is still broken.

[baltpeter]

Indeed HTTP traffic is recorded, but HTTPS traffic errors.

[baltpeter]

Amazingly enough, the path to the mitmproxy CA is the same on Windows. :D

[baltpeter]

sigh

There's a \r in the filename: adb push "C:\\Users\\root\\.mitmproxy\\mitmproxy-ca-cert.pem" "/system/etc/security/cacerts/c8750f0d\r.0"

Why? Because we are splitting on \n. :(

https://github.com/tweaselORG/appstraction/blob/c4502ba06d1600329241479400693dc7ac239981/src/android.ts#L170-L172

[baltpeter]

HTTPS traffic! \o/

[baltpeter]

I haven't tested the emulator, but between https://github.com/tweaselORG/cyanoacrylate/pull/19 and https://github.com/tweaselORG/appstraction/pull/61, Windows support should be mostly read. For Android at least. I haven't looked at iOS at all, yet.

[baltpeter]

I sometimes get this error:

undefined:2
{"status": "done"}
^

SyntaxError: Unexpected token { in JSON at position 75
    at JSON.parse (<anonymous>)
    at Socket.listener (file:///C:/Users/root/coding/tweasel/ca-test/node_modules/cyanoacrylate/dist/index.js:117:30)
    at Socket.emit (node:events:525:35)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)

Pretty sure that is the problem I was expecting here: https://github.com/tweaselORG/cyanoacrylate/pull/19#discussion_r1164275000

[baltpeter]

Here's the script I used for testing (based on the example):

import { readdir } from 'fs/promises';
import path from 'path';
import { pause, startAnalysis } from 'cyanoacrylate';

(async () => {
    const apkFolder = 'C:\\Users\\root\\Downloads\\single-apks';

    console.log('starting');
    const analysis = await startAnalysis({
        platform: 'android',
        runTarget: 'device',
        capabilities: ['frida', 'certificate-pinning-bypass'],
        targetOptions: {
        },
    });
    console.log('started');

    await analysis.ensureDevice();
    console.log('ensured.');

    const apks = await readdir(apkFolder);
    for (const apkFile of apks) {
        console.log(apkFile);
        const appAnalysis = await analysis.startAppAnalysis(path.join(apkFolder, apkFile));
        console.log('ana started');

        await appAnalysis.installApp();
        console.log('installed');
        await appAnalysis.setAppPermissions();
        console.log('perms set');
        await appAnalysis.startTrafficCollection();
        console.log('traffic started');
        await appAnalysis.startApp();
        console.log('app started');

        await pause(6_000);
        console.log('paused');

        await appAnalysis.stopTrafficCollection();
        console.log('traffic stopped');

        const result = await appAnalysis.stop();
        console.log('app stopped');
        await appAnalysis.uninstallApp();
        console.log('uninstalled');

        console.dir(result, { depth: null });
        console.log('\n\n');
    }

    await analysis.stop();
    console.log('done');
})();

[baltpeter]

And these were my setup steps:

• Install Node.js 18.15.0.
  • Mostly installer defaults (will already add to PATH).
  • Under "Tools for Native Modules", check "Automatically install the necessary tools."
• This will already install Python 3.11.2.
• Install Android platform tools and Android build tools (via), add to PATH. I needed to restart Windows Terminal for the changes to take effect. Opening a new terminal window wasn't enough.
• Connect phone, run adb devices to test. Trust computer on phone.
• I didn't install an OEM driver, but might be necessary.
• Install OpenSSL, add to PATH.

mkdir ca-test
cd ca-test
npm init --yes
npm i cyanoacrylate # I used a `yarn pack`ed `.tar.gz`.

• The first time mitmproxy is started, allow it to go through the firewall.

---

Honestly, that is all already covered in the README, you just need to figure out how to get those dependencies on Windows (but then, we don't provide instructions for that on Linux, either).

[baltpeter]

The ideviceinstaller command syntax is different between Windows and *nix. :(

PS C:\Users\root> ideviceinstaller
ERROR: Missing command.

Usage: C:\Users\root\Downloads\libimobile-suite-latest_x86_64-mingw64.tar\ideviceinstaller.exe OPTIONS

Manage apps on iOS devices.

COMMANDS:
  list                List installed apps
        -b, --bundle-identifier  Only query for given bundle identifier
            (can be passed multiple times)
        -o list_user      list user apps only (this is the default)
        -o list_system    list system apps only
        -o list_all       list all types of apps
        -o xml            print full output as xml plist
  install PATH        Install app from package file specified by PATH.
                      PATH can also be a .ipcc file for carrier bundles.
  uninstall BUNDLEID  Uninstall app specified by BUNDLEID.
  upgrade PATH        Upgrade app from package file specified by PATH.
  list-archives       List archived apps
        -o xml            print full output as xml plist
  archive BUNDLEID   Archive app specified by BUNDLEID, possible options:
        -o uninstall      uninstall the package after making an archive
        -o app_only       archive application data only
        -o docs_only      archive documents (user data) only
        -o copy=PATH      copy the app archive to directory PATH when done
        -o remove         only valid when copy=PATH is used: remove after copy
  restore BUNDLEID   Restore archived app specified by BUNDLEID
  remove-archive BUNDLEID    Remove app archive specified by BUNDLEID

OPTIONS:
  -u, --udid UDID     Target specific device by UDID
  -n, --network       Connect to network device
  -w, --notify-wait   Wait for app installed/uninstalled notification
                      to before reporting success of operation
  -h, --help          Print usage information
  -d, --debug         Enable communication debugging
  -v, --version       Print version information

Homepage:    <https://libimobiledevice.org>
Bug Reports: <https://github.com/libimobiledevice/ideviceinstaller/issues>

vs.

❯ ideviceinstaller
ERROR: No mode/command was supplied.
Usage: ideviceinstaller OPTIONS

Manage apps on iOS devices.

OPTIONS:
  -u, --udid UDID   Target specific device by UDID.
  -n, --network     Connect to network device.
  -l, --list-apps   List apps, possible options:
       -o list_user - list user apps only (this is the default)
       -o list_system   - list system apps only
       -o list_all  - list all types of apps
       -o xml       - print full output as xml plist
  -i, --install ARCHIVE Install app from package file specified by ARCHIVE.
                        ARCHIVE can also be a .ipcc file for carrier bundles.
  -U, --uninstall APPID Uninstall app specified by APPID.
  -g, --upgrade ARCHIVE Upgrade app from package file specified by ARCHIVE.
  -L, --list-archives   List archived applications, possible options:
       -o xml       - print full output as xml plist
  -a, --archive APPID   Archive app specified by APPID, possible options:
       -o uninstall - uninstall the package after making an archive
       -o app_only  - archive application data only
       -o docs_only - archive documents (user data) only
       -o copy=PATH - copy the app archive to directory PATH when done
       -o remove    - only valid when copy=PATH is used: remove after copy
  -r, --restore APPID   Restore archived app specified by APPID
  -R, --remove-archive APPID  Remove app archive specified by APPID
  -o, --options     Pass additional options to the specified command.
  -w, --notify-wait     Wait for app installed/uninstalled notification
                            to before reporting success of operation
  -h, --help        prints usage information
  -d, --debug       enable communication debugging
  -v, --version     print version information

Homepage:    <https://libimobiledevice.org>
Bug Reports: <https://github.com/libimobiledevice/ideviceinstaller/issues>

[baltpeter]

Looking good. That really wasn't too bad.

[baltpeter]

Here are my setup steps for iOS (again, nothing that isn't in the README already):

• Install Node.js 18.15.0.
  • Mostly installer defaults (will already add to PATH).
  • Under "Tools for Native Modules", check "Automatically install the necessary tools."
• This will already install Python 3.11.2.
• Download libimobiledevice build, add to PATH.
• Install iTunes to get the Apple Device Driver and the Apple Application Support service. Without these, I was getting ERROR: No device found!.
• Connect the iPhone and trust the computer.
• Run ideviceinfo to confirm it worked.

mkdir ca-test
cd ca-test
npm init --yes
npm i cyanoacrylate # I used a `yarn pack`ed `.tar.gz`.

• The first time mitmproxy is started, allow it to go through the firewall.

[baltpeter]

Actually, maybe we should mention that you need the iTunes drivers.