Create a CLI for cyanoacrylate

[baltpeter]

There are definitely common use cases for CA that shouldn't require you to write a custom JS script, e.g. capturing an app's traffic for a few seconds without interaction.

For those, it would be much nicer to have a CLI that you can just call. We have decided to make that a part of CA directly, instead of making it a separate package.

[baltpeter]

For now, I'll only implement a "record traffic" command that you give an app and a duration, and it will then run the app for that duration and spit out a HAR file of the recorded traffic at the end.

[baltpeter]

Which CLI library do we want to use? For thesis-mobile-consent-dialogs and backghup, I've used yargs because of its simplicity but I think it might not have all the features we'll need here. Also, I've encountered a few annoying rough edges (e.g. the help command output is not wrapper if you're using ESM).
For the dattel CLI, I used oclif.

I'll do a bit of research into what other options there are.

[baltpeter]

These are the options I considered (mostly compiled from https://openbase.com/categories/js/best-nodejs-cli-libraries?vs=commander%2Cyargs%2Cprompts):

• gluegun: The name is great of course. :D But it's not maintained by the original developer anymore.
• nash was last published five years ago. Doesn't have TypeScript support.
• caporal was last published three years ago.
• vorpal is unmaintained and was last published six years ago.
• sywac was last published three years ago and doesn't support TypeScript.
• commander is very mature, well maintained, downloaded 120M times a week. The interface looks nice and simple but still powerful. From a glance across the README, I haven't really seen any missing features.
• oclif is also mature, well maintained and popular. It has quite a few very nice features like the ability to generate standalone executables, autocomplete for bash and zsh, and automatic README generation. But that comes at a cost: it is much more complex than commander.

[baltpeter]

I started setting up oclif but already ran into two problems before I even really started:

• Their boilerplate generates a ./bin/run file that acts as the bin script in the package.json. That is however incompatible with type: module (which we are using). If you try to run ./bin/run, you get:

  node:internal/errors:490
    ErrorCaptureStackTrace(err);
    ^
  
  TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /home/benni/coding/JS/tweasel/cyanoacrylate/bin/run. Loading extensionless files is not supported inside of "type":"module" package.json contexts. The package.json file /home/benni/coding/JS/tweasel/cyanoacrylate/package.json caused this "type":"module" context. Try changing /home/benni/coding/JS/tweasel/cyanoacrylate/bin/run to have a file extension. Note the "bin" field of package.json can point to a file with an extension, for example {"type":"module","bin":{"run":"./bin/run.js"}}
    at new NodeError (node:internal/errors:399:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:79:11)
    at defaultGetFormat (node:internal/modules/esm/get_format:121:38)
    at defaultLoad (node:internal/modules/esm/load:81:20)
    at nextLoad (node:internal/modules/esm/loader:163:28)
    at ESMLoader.load (node:internal/modules/esm/loader:605:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:457:22)
    at new ModuleJob (node:internal/modules/esm/module_job:64:26)
    at #createModuleJob (node:internal/modules/esm/loader:480:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:434:34) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
  }

  An issue for this (https://github.com/oclif/oclif/issues/712) has been open since November 2021 but not gotten any response from maintainers.

  If I rename the file to run.cjs, I can run it with ./bin/run.cjs, but I am not sure whether that breaks anything else down the line.

• We have currently configured Parcel to only build the index.ts but oclif expects a folder in dist with a file for each command.

[baltpeter]

I'm tempted to just try commander instead. Losing the nice oclif features would be sad (e.g. commander has no plans to support shell autocomplete), but I'm not sure whether it's worth sinking too much more time into this. I guess, worst case, we can always migrate to oclif later if we do decide we need those features.

[baltpeter]

Well, unfortunately commander really is pretty limited. :/

• The TypeScript support is a bit of a lie. By default, the arguments aren't typed. There is a @commander-js/extra-typings package meant to solve this but that didn't work for me (the options were typed as {}).
• The conflict options are very limited (e.g. I can say that --opt1 conflicts with --opt2, but I can't say that --bypass-certificate-pinning conflicts with --platform ios).
• I haven't found a way to group multiple options together (I would like to have an "emulator options" group to make it clear that those options aren't relevant to physical devices).

[baltpeter]

Looks like oclif doesn't have these limitations: https://oclif.io/docs/flags

[baltpeter]

• We have currently configured Parcel to only build the index.ts but oclif expects a folder in dist with a file for each command.

Why does everything have to be complicated? Parcel supports globs on the command line, but not in package.json.

[baltpeter]

Trying to get the oclif hello world example to run, I was struggling with this error:

❯ yarn oclif manifest
yarn run v1.22.19
$ /home/benni/coding/JS/tweasel/cyanoacrylate/node_modules/.bin/oclif manifest
 ›   Error Plugin: cyanoacrylate: command hello not found

Now, I've found the problem. Turns out, Parcel completely strips the export default for some reason. o.O If I manually add that to the built files, it works.

[baltpeter]

It doesn't strip the export default if I add "isLibrary": true… That just cost me an hour. -.-

[baltpeter]

For the record (maybe this helps someone else), here's the oclif and Parcel configuration in package.json that seems to work so far:

{
    "oclif": {
        "bin": "cyanoacrylate",
        "dirname": "cyanoacrylate",
        "commands": "./dist/cli/commands",
        "plugins": [
            "@oclif/plugin-help",
            "@oclif/plugin-autocomplete"
        ]
    },
    "targets": {
        "library": {
            "distDir": "./dist",
            "source": "./src/index.ts",
            "context": "node",
            "isLibrary": true,
            "outputFormat": "esmodule",
            "optimize": false
        },
        "types": {
            "source": "./src/index.ts",
            "distDir": "./dist"
        },
        "cli": {
            "distDir": "./dist/cli",
            "source": [
                "./src/cli/index.ts",
                "./src/cli/commands/hello.ts"
            ],
            "context": "node",
            "isLibrary": true,
            "outputFormat": "esmodule",
            "optimize": false
        }
    }
}

Downside: You have to manually add each command to targets.cli.source.

Update: That wasn't quite it.

[baltpeter]

Their boilerplate generates a ./bin/run file that acts as the bin script in the package.json. That is however incompatible with type: module (which we are using).

I just noticed that there is official documentation on how to solve that, after all: https://oclif.io/docs/esm

[baltpeter]

Next problem: import { startAnalysis } from '../../index'; doesn't work. Parcel just doesn't include the code from that import in the bundle.

It replaces it with var $6NsKs = parcelRequire("6NsKs");, but that fails when running:

 ›   ModuleLoadError Plugin: cyanoacrylate: [MODULE_NOT_FOUND] import() failed to load 
 ›   /home/benni/coding/JS/tweasel/cyanoacrylate/dist/cli/commands/record-traffic.js: Cannot find module '6NsKs'
 ›   Code: MODULE_NOT_FOUND

For parcelRequire() to work, the module would have to be included in $parcel$modules but it isn't:

var $parcel$modules = {};
var $parcel$inits = {};

var parcelRequire = $parcel$global["parcelRequire2a89"];
if (parcelRequire == null) {
  parcelRequire = function(id) {
    if (id in $parcel$modules) {
      return $parcel$modules[id].exports;
    }
    if (id in $parcel$inits) {
      var init = $parcel$inits[id];
      delete $parcel$inits[id];
      var module = {id: id, exports: {}};
      $parcel$modules[id] = module;
      init.call(module.exports, module, module.exports);
      return module.exports;
    }
    var err = new Error("Cannot find module '" + id + "'");
    err.code = 'MODULE_NOT_FOUND';
    throw err;
  };

  parcelRequire.register = function register(id, init) {
    $parcel$inits[id] = init;
  };

  $parcel$global["parcelRequire2a89"] = parcelRequire;
}

[baltpeter]

This appears to be because Parcel thinks it has already built the main library. If I remove the library target, the bundle is produced correctly. But I do want both, of course. :D

The targets documentation doesn't have any info on how to solve that problem.

[baltpeter]

I think I finally have the magic combination of properties to get this to work (:|). You need to combine the library and cli targets, but still have a types target:

{
    "targets": {
        "types": {
            "source": "./src/index.ts",
            "distDir": "./dist"
        },
        "everything": {
            "distDir": "./dist",
            "source": [
                "./src/index.ts",
                "./src/cli/index.ts",
                "./src/cli/commands/record-traffic.ts"
            ],
            "context": "node",
            "isLibrary": true,
            "outputFormat": "esmodule",
            "optimize": false
        }
    }
}

Oh, and you can't name that combined target main. That will fail with Error: Bundles must have unique names.

[baltpeter]

Oh no, turns out including the entire library code in the CLI bundle doesn't work either. That breaks the path for the mitmproxy addons:

{
    spawnargs: [
      'mitmdump',
      '--mode',
      'wireguard',
      '-s',
      '/home/benni/coding/JS/tweasel/cyanoacrylate/dist/cli/mitmproxy-addons/ipcEventsAddon.py',
      '-s',
      '/home/benni/coding/JS/tweasel/cyanoacrylate/dist/cli/mitmproxy-addons/har_dump.py',
      '--set',
      'hardump=/tmp/f9144eb72e9ba6da19efa1530a12a672.har',
      '--set',
      'ipcPipeFd=3'
    ],

}

[baltpeter]

I've given up on including the CLI in here. Instead, I have created a new tweasel-cli package.

PR: https://github.com/tweaselORG/cli/pull/1