build for arm64 failed - Githubissues

I want build in arm64.but, when i run "npm run prepack",There are the following errors:

$ node -v v14.17.1 kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ uname -a Linux kylin-v10sp1-arm64-2107 5.4.18-35-generic #21-KYLINOS SMP Tue Jul 20 13:31:32 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ npm run prepack

httptoolkit-server@1.10.0 prepack /home/kylin/httptoolkit-server npm run build:src && oclif-dev manifest

httptoolkit-server@1.10.0 build:src /home/kylin/httptoolkit-server rm -rf lib && tsc -b --force

node_modules/portfinder/lib/portfinder.d.ts:39:1 - error TS1128: Declaration or statement expected.

39 export setBasePort(port: number);


node_modules/portfinder/lib/portfinder.d.ts:39:24 - error TS1005: ',' expected.

39 export setBasePort(port: number);
                          ~

node_modules/portfinder/lib/portfinder.d.ts:49:1 - error TS1128: Declaration or statement expected.

49 export setHighestPort(port: number);

node_modules/portfinder/lib/portfinder.d.ts:49:27 - error TS1005: ',' expected.

49 export setHighestPort(port: number); ~

Found 4 errors.

npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! httptoolkit-server@1.10.0 build:src: rm -rf lib && tsc -b --force npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the httptoolkit-server@1.10.0 build:src script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! /home/kylin/.npm/_logs/2022-08-12T05_50_15_861Z-debug.log npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! httptoolkit-server@1.10.0 prepack: npm run build:src && oclif-dev manifest npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the httptoolkit-server@1.10.0 prepack script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! /home/kylin/.npm/_logs/2022-08-12T05_50_15_891Z-debug.log

This isn't related to arm64 - this error is actually due to a bug in the latest release of portfinder: https://github.com/http-party/node-portfinder/issues/138.

That said, I'm surprised this is appearing for you here. This repo has a lockfile which still locks portfinder to 1.0.28 (which doesn't have this bug, and builds just fine): https://github.com/httptoolkit/httptoolkit-server/blob/3ce7fce83854685892209c2ecffcee5258dcf0c8/package-lock.json#L21337-L21345

That should stop portfinder updating unexpectedly like this. Any idea why that lockfile isn't being used in your case?

Regardless, as a quick workaround if you switch to using that portfinder version then everything should work correctly.

By the way, do let me know if you get this working, or if you run into any other issues. I don't have an arm64 test machine myself, but I'm definitely interested in supporting this in future, and I know others (like @shirshak55) are interested in this too. See https://github.com/httptoolkit/httptoolkit-desktop/issues/45 for some more context.

That said, I'm surprised this is appearing for you here

this is odd

这与 arm64 无关 - 此错误实际上是由于最新版本的 portfinder 中的错误：http-party/node-portfinder#138。

也就是说，我很惊讶这会出现在这里。这个 repo 有一个锁定文件，它仍然将 portfinder 锁定到 1.0.28（它没有这个错误，并且构建得很好）：

https://github.com/httptoolkit/httptoolkit-server/blob/3ce7fce83854685892209c2ecffcee5258dcf0c8/package-lock.json#L21337-L21345

这应该会阻止 portfinder 像这样意外更新。知道为什么在您的情况下没有使用该锁定文件吗？

无论如何，作为一种快速解决方法，如果您切换到使用该 portfinder 版本，那么一切都应该正常工作。

顺便说一句，如果你得到这个工作，或者如果你遇到任何其他问题，请告诉我。我自己没有 arm64 测试机，但我绝对有兴趣在未来支持它，而且我认识其他人（比如@shirshak55) 对此也很感兴趣。有关更多上下文，请参阅httptoolkit/httptoolkit-desktop#45 。

upgrade portfinder version to 1.0.30,the above error has been resolved.

but when I run "$ npm run build:release",it have new error. The following is the operation log: $ npm i npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142 npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated cli-ux@5.6.7: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. npm WARN deprecated har-validator@5.1.5: this library is no longer supported npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated npm WARN deprecated chokidar@2.1.8: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead. npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.

cpu-features@0.0.4 install /home/kylin/httptoolkit-server/node_modules/cpu-features node buildcheck.js > buildcheck.gypi && node-gyp rebuild

make: 进入目录“/home/kylin/httptoolkit-server/node_modules/cpu-features/build” CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_aarch64_linux_or_android.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_arm_linux_or_android.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_mips_linux_or_android.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_ppc_linux.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_x86_freebsd.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_x86_linux_or_android.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_x86_macos.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/impl_x86_windows.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/filesystem.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/stack_line_reader.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/string_view.o CC(target) Release/obj.target/cpu_features/deps/cpu_features/src/hwcaps.o AR(target) Release/obj.target/deps/cpu_features/cpu_features.a COPY Release/cpu_features.a CXX(target) Release/obj.target/cpufeatures/src/binding.o In file included from ../src/binding.cc:1: /home/kylin/.cache/node-gyp/14.17.1/include/node/node.h:758:43: warning: cast between incompatible function types from ‘void ()(Nan::ADDON_REGISTER_FUNCTION_ARGS_TYPE)’ {aka ‘void ()(v8::Local)’} to ‘node::addon_register_func’ {aka ‘void ()(v8::Local, v8::Local, void)’} [-Wcast-function-type] 758 | (node::addon_register_func) (regfunc), \ | ^ /home/kylin/.cache/node-gyp/14.17.1/include/node/node.h:792:3: note: in expansion of macro ‘NODE_MODULE_X’ 792 | NODE_MODULE_X(modname, regfunc, NULL, 0) // NOLINT (readability/null_usage) | ^~~~~ ../src/binding.cc:152:1: note: in expansion of macro ‘NODE_MODULE’ 152 | NODE_MODULE(cpufeatures, init) | ^~~ SOLINK_MODULE(target) Release/obj.target/cpufeatures.node COPY Release/cpufeatures.node make: 离开目录“/home/kylin/httptoolkit-server/node_modules/cpu-features/build”

node-datachannel@0.3.5 install /home/kylin/httptoolkit-server/node_modules/node-datachannel prebuild-install || (npm install --ignore-scripts && npm run _prebuild)

ssh2@1.11.0 install /home/kylin/httptoolkit-server/node_modules/ssh2 node install.js

make: 进入目录“/home/kylin/httptoolkit-server/node_modules/ssh2/lib/protocol/crypto/build” CXX(target) Release/obj.target/sshcrypto/src/binding.o SOLINK_MODULE(target) Release/obj.target/sshcrypto.node COPY Release/sshcrypto.node make: 离开目录“/home/kylin/httptoolkit-server/node_modules/ssh2/lib/protocol/crypto/build” Succeeded in building optional crypto binding

@sentry/cli@1.74.5 install /home/kylin/httptoolkit-server/node_modules/@sentry/cli node ./scripts/install.js

info sentry-cli Using cached binary: /home/kylin/.npm/sentry-cli/a85cf2-sentry-cli-Linux-aarch64

win-version-info@5.0.1 install /home/kylin/httptoolkit-server/node_modules/win-version-info node skip.js || node-gyp-build

registry-js@1.15.1 install /home/kylin/httptoolkit-server/node_modules/registry-js prebuild-install || node-gyp rebuild

prebuild-install WARN install No prebuilt binaries found (target=3 runtime=napi arch=arm64 libc= platform=linux) make: 进入目录“/home/kylin/httptoolkit-server/node_modules/registry-js/build” SOLINK_MODULE(target) Release/obj.target/registry.node COPY Release/registry.node make: 离开目录“/home/kylin/httptoolkit-server/node_modules/registry-js/build”

httptoolkit-server@1.10.0 prepare /home/kylin/httptoolkit-server ts-node ./prepare.ts

Installing override npm dependencies... npm WARN prepare removing existing node_modules/ before installation added 28 packages in 0.959s Override dependencies installed npm notice created a lockfile as package-lock.json. You should commit this file. npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.3.1 (node_modules/chokidar/node_modules/fsevents): npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"arm64"}) npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.7 (node_modules/watchpack-chokidar2/node_modules/chokidar/node_modules/fsevents): npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"arm64"})

added 1325 packages from 903 contributors and audited 1346 packages in 236.89s

64 packages are looking for funding run npm fund for details

found 9 vulnerabilities (2 low, 3 moderate, 4 high) run npm audit fix to fix them, or npm audit for details kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ npm run prepack

httptoolkit-server@1.10.0 prepack /home/kylin/httptoolkit-server npm run build:src && oclif-dev manifest

httptoolkit-server@1.10.0 build:src /home/kylin/httptoolkit-server rm -rf lib && tsc -b --force

wrote manifest to /home/kylin/httptoolkit-server/oclif.manifest.json kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ vim package.json kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ npm run build:release

httptoolkit-server@1.10.0 build:release /home/kylin/httptoolkit-server oclif-dev manifest && webpack && ts-node ./pack.ts

wrote manifest to /home/kylin/httptoolkit-server/oclif.manifest.json Sentry source map upload disabled - no token set Hash: 081b7aaddfb92a0e3d8f Version: webpack 4.46.0 Time: 162919ms Built at: 2022/08/15 上午8:47:37 43 assets Entrypoint index = index.js index.js.map Entrypoint error-tracking = error-tracking.js error-tracking.js.map [6] external "path" 42 bytes {0} {1} [built] [8] external "fs" 42 bytes {0} {1} [built] [52] external "child_process" 42 bytes {0} {1} [built] [53] external "./error-tracking" 42 bytes {1} [built] [59] ./src/util/promise.ts 1 KiB {1} [built] [78] ./src/util/fs.ts 3.35 KiB {1} [built] [100] ./src/constants.ts 1.99 KiB {0} {1} [built] [128] ./src/util/error.ts 389 bytes {1} [built] [183] ./src/shutdown.ts 1.11 KiB {1} [built] [188] ./src/interceptors/docker/docker-interception-services.ts 6.02 KiB {1} [built] [286] ./src/browsers.ts 3.69 KiB {1} [built] [328] ./src/certificates.ts 1.25 KiB {1} [built] [329] ./package.json 5.54 KiB {0} {1} [built] [616] ./src/index.ts 6.97 KiB {1} [built] [1250] ./src/error-tracking.ts 5.46 KiB {0} [built]

1357 hidden modules

WARNING in ./node_modules/express/lib/view.js 81:13-25 Critical dependency: the request of a dependency is an expression @ ./node_modules/express/lib/application.js @ ./node_modules/express/lib/express.js @ ./node_modules/express/index.js @ ./src/api-server.ts @ ./src/index.ts

WARNING in ./node_modules/node-fetch/lib/index.mjs Module not found: Error: Can't resolve 'encoding' in '/home/kylin/httptoolkit-server/node_modules/node-fetch/lib' @ ./node_modules/node-fetch/lib/index.mjs @ ./src/interceptors/android/fetch-apk.ts @ ./src/interceptors/android/android-adb-interceptor.ts @ ./src/interceptors/index.ts @ ./src/api-server.ts @ ./src/index.ts

WARNING in ./node_modules/ssh2/lib/protocol/crypto/build/Release/sshcrypto.node 1:0 Module parse failed: Unexpected character '' (1:0) You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders (Source code omitted for this binary file) @ ./node_modules/ssh2/lib/protocol/crypto.js 30:12-60 @ ./node_modules/ssh2/lib/protocol/keyParser.js @ ./node_modules/ssh2/lib/index.js @ ./node_modules/docker-modem/lib/ssh.js @ ./node_modules/docker-modem/lib/modem.js @ ./node_modules/dockerode/lib/docker.js @ ./src/interceptors/docker/docker-interception-services.ts @ ./src/index.ts

ERROR in ./node_modules/cpu-features/build/Release/cpufeatures.node 1:0 Module parse failed: Unexpected character '' (1:0) You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders (Source code omitted for this binary file) @ ./node_modules/cpu-features/lib/index.js 3:16-60 @ ./node_modules/ssh2/lib/protocol/constants.js @ ./node_modules/ssh2/lib/server.js @ ./node_modules/ssh2/lib/index.js @ ./node_modules/docker-modem/lib/ssh.js @ ./node_modules/docker-modem/lib/modem.js @ ./node_modules/dockerode/lib/docker.js @ ./src/interceptors/docker/docker-interception-services.ts @ ./src/index.ts

ERROR in ./node_modules/dns2/client/doh.js 26:36 Module parse failed: Unexpected token (26:36) You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders | @param {} param0 | */

const DOHClient = ({ dns, http, get = defaultGet } = {}) => { | return (name, type = 'A', cls = Packet.CLASS.IN, { clientIp, recursive = true } = {}) => { | const packet = new Packet(); @ ./node_modules/dns2/index.js 91:16-39 @ ./src/dns-server.ts @ ./src/api-server.ts @ ./src/index.ts npm ERR! code ELIFECYCLE npm ERR! errno 2 npm ERR! httptoolkit-server@1.10.0 build:release: oclif-dev manifest && webpack && ts-node ./pack.ts npm ERR! Exit status 2 npm ERR! npm ERR! Failed at the httptoolkit-server@1.10.0 build:release script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! /home/kylin/.npm/_logs/2022-08-15T00_47_39_848Z-debug.log

Interesting...

So, it looks like with this, installation and the basic build works OK, but building a release fails. That's a great start though.

If you run npm start, does the server seem to start OK?

If it does, and then you open https://app.httptoolkit.tech in Chrome, does the app work OK? You can launch some browsers etc to check that traffic is intercepted correctly (any browser is fine for this, except Safari, which unfortunately has some standards issues that make it unusable).

The error shown where when bundling the release code with webpack is related to the cpu-features package. That's an optional subdependency, and I think on my machine & in CI it actually normally fails to build, and so it's always skipped before this step! You should be able to fix the webpack error here though by adding that package to the externals list in the webpack config: https://github.com/httptoolkit/httptoolkit-server/blob/e1073277001d506d268f0ac5a4d9f3d4e17e7033/webpack.config.js#L61

That will exclude it from release bundling entirely, so it generally won't be available at runtime, but that's fine because we don't actually need it (and most builds don't include it anyway!). If that works for you, I'd happily accept a quick PR for this, I think we basically never want to include cpu-features in the bundle - it comes from a subdependency (ssh2) for SSH host support, used within dockerode, which helps power Docker interception. This is optional anyway, and I'm fairly sure we don't properly support even SSH in cases like that regardless, since I think Docker interception only works locally right now.

when i run npm start , server seem to start OK .

kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ npm run start

httptoolkit-server@1.10.0 start node-dev ./bin/run start

Config checked in 30 ms Certificates setup in 26 ms Standalone server started in 13 ms Server started in 35 ms Total startup took 104 ms Browser cache updated Websocket request for unrecognized mock session: e3672e2b-c5d5-4058-b387-29c680d833cd httptoolkit-server: Updating CLI... ⣯ fetching manifest

I have add cpu-features to webpack.config.js

externals: [
        '@oclif/plugin-update/lib/commands/update', // Lots of complicated dynamic requires in @oclif
        'registry-js', // Native module
        'win-version-info', // Native module
        'node-datachannel', // Native module
        'cpu-features',
        'vm2', // Does odd things with require, can't be webpack'd

One of the bugs has been resolved, the other one still exists

kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ npm run build:release

httptoolkit-server@1.10.0 build:release oclif-dev manifest && webpack && ts-node ./pack.ts

wrote manifest to /home/kylin/httptoolkit-server/oclif.manifest.json Sentry source map upload disabled - no token set Hash: 0ca12bbc35d6c72fc446 Version: webpack 4.46.0 Time: 73666ms Built at: 2022/08/16 下午3:00:58 43 assets Entrypoint index = index.js index.js.map Entrypoint error-tracking = error-tracking.js error-tracking.js.map [6] external "path" 42 bytes {0} {1} [built] [8] external "fs" 42 bytes {0} {1} [built] [52] external "child_process" 42 bytes {0} {1} [built] [53] external "./error-tracking" 42 bytes {1} [built] [59] ./src/util/promise.ts 1 KiB {1} [built] [78] ./src/util/fs.ts 3.35 KiB {1} [built] [100] ./src/constants.ts 1.99 KiB {0} {1} [built] [128] ./src/util/error.ts 389 bytes {1} [built] [183] ./src/shutdown.ts 1.11 KiB {1} [built] [188] ./src/interceptors/docker/docker-interception-services.ts 6.02 KiB {1} [built] [286] ./src/browsers.ts 3.69 KiB {1} [built] [328] ./src/certificates.ts 1.25 KiB {1} [built] [329] ./package.json 5.54 KiB {0} {1} [built] [616] ./src/index.ts 6.97 KiB {1} [built] [1249] ./src/error-tracking.ts 5.46 KiB {0} [built]

1356 hidden modules

const DOHClient = ({ dns, http, get = defaultGet } = {}) => { | return (name, type = 'A', cls = Packet.CLASS.IN, { clientIp, recursive = true } = {}) => { | const packet = new Packet(); @ ./node_modules/dns2/index.js 91:16-39 @ ./src/dns-server.ts @ ./src/api-server.ts @ ./src/index.ts

when i run npm start , server seem to start OK .

That looks very promising! Suggests that actually everything at runtime is set up correctly & working OK.

One of the bugs has been resolved, the other one still exists

Right, yes, so the error here is due to failing to parse dns2/client/doh.js. This is interesting though, because the unparseable code snippet shown:

const DOHClient = ({ dns, http, get = defaultGet } = {}) => {

is present in the latest version 2.0.5 but the package lock says (here) that you should be installing version 2.0.1, which doesn't include that defaultGet parameter.

This is relatively new JS syntax, though I'm not sure specifically why webpack doesn't handle it. That is almost certainly fixable, but I think it would be better to work out why you're not using the versions in the lockfile. In theory you can use any semver-compatible version for all dependencies if you like, but it's very likely that you'll run into awkward untested configurations with issues like this if you're not using the versions from the lockfile. If we avoid that, then we can focus on ARM-specific issues.

Which versions of node & npm are you using?

What happens if you run npm ci?

Thank you very much for your support, it can be packaged normally now The execution process is as follows

$ npm i dns2@2.0.1

changed 1 package, and audited 1326 packages in 10s

64 packages are looking for funding
  run `npm fund` for details

10 vulnerabilities (4 moderate, 6 high)

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
$ npm run build:release

> httptoolkit-server@1.10.0 build:release
> oclif-dev manifest && webpack && ts-node ./pack.ts

wrote manifest to /home/kylin/httptoolkit-server/oclif.manifest.json
Sentry source map upload disabled - no token set
Hash: 6c64904150ee98477b9d
Version: webpack 4.46.0
Time: 74068ms
Built at: 2022/08/17 上午9:29:10
                                                          Asset       Size  Chunks                   Chunk Names
                                         ../lib/api-server.d.ts  286 bytes          [emitted]        
                                           ../lib/browsers.d.ts  554 bytes          [emitted]        
                                  ../lib/cert-check-server.d.ts  348 bytes          [emitted]        
                                       ../lib/certificates.d.ts  364 bytes          [emitted]        
                                     ../lib/commands/start.d.ts  502 bytes          [emitted]        
                                          ../lib/constants.d.ts  307 bytes          [emitted]        
                                 ../lib/destroyable-server.d.ts  223 bytes          [emitted]        
                                         ../lib/dns-server.d.ts  561 bytes          [emitted]        
                                     ../lib/error-tracking.d.ts  275 bytes          [emitted]        
                                ../lib/hide-warning-server.d.ts  324 bytes          [emitted]        
                                              ../lib/index.d.ts  111 bytes          [emitted]        
                  ../lib/interceptors/android/adb-commands.d.ts   1.42 KiB          [emitted]        
       ../lib/interceptors/android/android-adb-interceptor.d.ts  726 bytes          [emitted]        
                     ../lib/interceptors/android/fetch-apk.d.ts  267 bytes          [emitted]        
           ../lib/interceptors/chromium-based-interceptors.d.ts   3.29 KiB          [emitted]        
         ../lib/interceptors/docker/docker-build-injection.d.ts   1.15 KiB          [emitted]        
                ../lib/interceptors/docker/docker-commands.d.ts   2.02 KiB          [emitted]        
                 ../lib/interceptors/docker/docker-compose.d.ts  350 bytes          [emitted]        
   ../lib/interceptors/docker/docker-interception-services.d.ts  717 bytes          [emitted]        
             ../lib/interceptors/docker/docker-interceptor.d.ts   1.06 KiB          [emitted]        
              ../lib/interceptors/docker/docker-networking.d.ts   2.21 KiB          [emitted]        
                   ../lib/interceptors/docker/docker-proxy.d.ts  329 bytes          [emitted]        
            ../lib/interceptors/docker/docker-tunnel-proxy.d.ts  559 bytes          [emitted]        
                              ../lib/interceptors/electron.d.ts  595 bytes          [emitted]        
                         ../lib/interceptors/fresh-firefox.d.ts  864 bytes          [emitted]        
                                 ../lib/interceptors/index.d.ts  694 bytes          [emitted]        
                                   ../lib/interceptors/jvm.d.ts  904 bytes          [emitted]        
../lib/interceptors/terminal/existing-terminal-interceptor.d.ts  678 bytes          [emitted]        
   ../lib/interceptors/terminal/fresh-terminal-interceptor.d.ts  473 bytes          [emitted]        
       ../lib/interceptors/terminal/terminal-env-overrides.d.ts  537 bytes          [emitted]        
             ../lib/interceptors/terminal/terminal-scripts.d.ts  603 bytes          [emitted]        
                                     ../lib/message-server.d.ts  355 bytes          [emitted]        
                                           ../lib/shutdown.d.ts  263 bytes          [emitted]        
                                         ../lib/util/error.d.ts  181 bytes          [emitted]        
                                            ../lib/util/fs.d.ts    1.4 KiB          [emitted]        
                                          ../lib/util/http.d.ts  122 bytes          [emitted]        
                            ../lib/util/process-management.d.ts  970 bytes          [emitted]        
                                       ../lib/util/promise.d.ts  388 bytes          [emitted]        
                                       ../lib/webextension.d.ts  266 bytes          [emitted]        
                                       bl-resources/Preferences  345 bytes          [emitted]        
                                    bl-resources/operaprefs.ini  739 bytes          [emitted]        
                                        bl-resources/phantom.js  882 bytes          [emitted]        
                                              error-tracking.js    150 KiB       0  [emitted]        error-tracking
                                          error-tracking.js.map    656 KiB       0  [emitted] [dev]  error-tracking
                                                       index.js   6.61 MiB       1  [emitted]        index
                                                   index.js.map   20.4 MiB       1  [emitted] [dev]  index
Entrypoint index = index.js index.js.map
Entrypoint error-tracking = error-tracking.js error-tracking.js.map
   [6] external "path" 42 bytes {0} {1} [built]
   [8] external "fs" 42 bytes {0} {1} [built]
  [52] external "child_process" 42 bytes {0} {1} [built]
  [53] external "./error-tracking" 42 bytes {1} [built]
  [59] ./src/util/promise.ts 1 KiB {1} [built]
  [78] ./src/util/fs.ts 3.35 KiB {1} [built]
 [100] ./src/constants.ts 1.99 KiB {0} {1} [built]
 [128] ./src/util/error.ts 389 bytes {1} [built]
 [183] ./src/shutdown.ts 1.11 KiB {1} [built]
 [188] ./src/interceptors/docker/docker-interception-services.ts 6.02 KiB {1} [built]
 [286] ./src/browsers.ts 3.69 KiB {1} [built]
 [328] ./src/certificates.ts 1.25 KiB {1} [built]
 [329] ./package.json 5.54 KiB {0} {1} [built]
 [616] ./src/index.ts 6.97 KiB {1} [built]
[1249] ./src/error-tracking.ts 5.46 KiB {0} [built]
    + 1356 hidden modules

WARNING in ./node_modules/express/lib/view.js 81:13-25
Critical dependency: the request of a dependency is an expression
 @ ./node_modules/express/lib/application.js
 @ ./node_modules/express/lib/express.js
 @ ./node_modules/express/index.js
 @ ./src/api-server.ts
 @ ./src/index.ts

WARNING in ./node_modules/node-fetch/lib/index.mjs
Module not found: Error: Can't resolve 'encoding' in '/home/kylin/httptoolkit-server/node_modules/node-fetch/lib'
 @ ./node_modules/node-fetch/lib/index.mjs
 @ ./src/interceptors/android/fetch-apk.ts
 @ ./src/interceptors/android/android-adb-interceptor.ts
 @ ./src/interceptors/index.ts
 @ ./src/api-server.ts
 @ ./src/index.ts

WARNING in ./node_modules/ssh2/lib/protocol/crypto/build/Release/sshcrypto.node 1:0
Module parse failed: Unexpected character '' (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)
 @ ./node_modules/ssh2/lib/protocol/crypto.js 30:12-60
 @ ./node_modules/ssh2/lib/protocol/keyParser.js
 @ ./node_modules/ssh2/lib/agent.js
 @ ./node_modules/ssh2/lib/index.js
 @ ./node_modules/docker-modem/lib/ssh.js
 @ ./node_modules/docker-modem/lib/modem.js
 @ ./node_modules/dockerode/lib/docker.js
 @ ./src/interceptors/docker/docker-interception-services.ts
 @ ./src/index.ts
Preparing packaging directory
Building for Linux
CONFIGURING FOR linux

BUILDING FOR linux

oclif-dev: gathering workspace for httptoolkit-server to /home/kylin/httptoolkit-server/build/tmp/httptoolkit-server
npm notice 
npm notice 📦  httptoolkit-server@1.10.0
npm notice === Tarball Contents === 
npm notice 92B     bin/run                                                                               
npm notice 31B     bin/run.cmd                                                                           
npm notice 739B    bundle/bl-resources/operaprefs.ini                                                    
npm notice 882B    bundle/bl-resources/phantom.js      
***
found 0 vulnerabilities
oclif-dev: packing tarball from ./tmp/httptoolkit-server to ./dist/v1.10.0/httptoolkit-server-v1.10.0.tar.gz
oclif-dev: building target httptoolkit-server-v1.10.0-linux-arm64.tar.gz
oclif-dev: downloading node-v16.16.0-linux-arm64
node-v16.16.0-linux-arm64.tar.xz: OK
oclif-dev: extracting node-v16.16.0-linux-arm64
oclif-dev: packing tarball from ./tmp/linux-arm64/httptoolkit-server to ./dist/v1.10.0/httptoolkit-server-v1.10.0-linux-arm64.tar.gz

BUILT

CHECKING FOR BAD CONFIG
EXPECTING: ELF
WHITELIST: registry-js|win-version-info/prebuilds
NATIVE BINS: ./tmp/linux-arm64/httptoolkit-server/node_modules/node-datachannel/build/Release/node_datachannel.node: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=13c67a6a0cafe542e8e6b6cbee06867490d9977b, not stripped
./tmp/linux-arm64/httptoolkit-server/node_modules/win-version-info/prebuilds/win32-ia32/node.napi.node: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows
./tmp/linux-arm64/httptoolkit-server/node_modules/win-version-info/prebuilds/win32-x64/node.napi.node: PE32+ executable (DLL) (GUI) x86-64, for MS Windows
./tmp/linux-arm64/httptoolkit-server/node_modules/registry-js/build/Release/registry.node: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb3bc1ff9f3699318bf9e355a5d85a16e597899a, not stripped
./tmp/linux-arm64/httptoolkit-server/node_modules/registry-js/build/Release/obj.target/registry.node: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb3bc1ff9f3699318bf9e355a5d85a16e597899a, not stripped
BUILD SUCCESSFUL
Building for Darwin
CONFIGURING FOR darwin

BUILDING FOR darwin
oclif-dev: gathering workspace for httptoolkit-server to /home/kylin/httptoolkit-server/build/tmp/httptoolkit-server
npm notice 
npm notice 📦  httptoolkit-server@1.10.0
npm notice === Tarball Contents === 
npm notice 92B     bin/run    
***
npm notice 3.3kB   package.json                                                                          
npm notice === Tarball Details === 
npm notice name:          httptoolkit-server                      
npm notice version:       1.10.0                                  
npm notice filename:      httptoolkit-server-1.10.0.tgz           
npm notice package size:  7.2 MB                                  
npm notice unpacked size: 18.7 MB                                 
npm notice shasum:        ead196b52ee40d609f6b6f6df04e2d0687d2328e
npm notice integrity:     sha512-+7cQmz6JXFkeb[...]/qouELxTOXQfQ==
npm notice total files:   1683                                    
npm notice 
npm WARN deprecated cli-ux@5.6.7: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm ERR! code 127
npm ERR! path /home/kylin/httptoolkit-server/build/tmp/httptoolkit-server/node_modules/node-datachannel
npm ERR! command failed
npm ERR! command sh -c prebuild-install || (npm install --ignore-scripts && npm run _prebuild)
npm ERR! added 1 package, removed 1 package, and audited 59 packages in 6m
npm ERR! 
npm ERR! 7 packages are looking for funding
npm ERR!   run `npm fund` for details
npm ERR! 
npm ERR! found 0 vulnerabilities
npm ERR! 
npm ERR! > node-datachannel@0.3.5 _prebuild
npm ERR! > prebuild --backend cmake-js
npm ERR! prebuild-install WARN install No prebuilt binaries found (target=16.13.2 runtime=node arch=arm64 libc= platform=darwin)
npm ERR! sh: 1: prebuild: not found

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kylin/.npm/_logs/2022-08-17T01_43_26_814Z-debug.log
    Error: Command failed: /bin/sh -c npm install --production
    Code: 127
Error: Spawn /home/kylin/httptoolkit-server/build/build-release.sh darwin exited with 1
    at ChildProcess.<anonymous> (/home/kylin/httptoolkit-server/pack.ts:16:25)
    at ChildProcess.emit (node:events:390:28)
    at ChildProcess.emit (node:domain:475:12)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ ls
bin               lib                  overrides          README.md             webpack.config.js
build             LICENSE              package.json       src
build-release.sh  node_modules         package-lock.json  test
bundle            nss                  pack.ts            tsconfig.json
custom-typings    oclif.manifest.json  prepare.ts         tsconfig.tsbuildinfo
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ cd build/
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server/build$ ls
bin               bundle  lib  oclif.manifest.json  package.json       prepare.ts
build-release.sh  dist    nss  overrides            package-lock.json  tmp
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server/build$ cd dist/
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server/build/dist$ ls
latest-manifest  v1.10.0
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server/build/dist$ cd v1.10.0/
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server/build/dist/v1.10.0$ ls
httptoolkit-server-v1.10.0-linux-arm64.tar.gz  httptoolkit-server-v1.10.0.tar.gz

This is the compiled tar package https://wwd.lanzoum.com/b03dacluf 密码:2zbm

Now, I want to know how to use local httptoolkit-desktop tarball to package httptoolkit-desktop.

$ npm run build

> httptoolkit-desktop@1.10.0 build
> npm run build:src && npm run build:electron:deb

> httptoolkit-desktop@1.10.0 build:src
> tsc

> httptoolkit-desktop@1.10.0 build:electron:deb
> ts-node ./setup-server.ts && electron-builder build -l deb

Downloading httptoolkit-server v1.10.0 for linux-arm64
Error: No server available matching /httptoolkit-server-v1.10.0-linux-arm64.tar.gz/
    at /home/kylin/httptoolkit-desktop/setup-server.ts:78:15
    at Generator.next (<anonymous>)
    at fulfilled (/home/kylin/httptoolkit-desktop/node_modules/tslib/tslib.js:112:62)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

I replaced /opt/HTTP Toolkit/resources/httptoolkit-server in the httptoolkit_1.10.0_arm64.deb on the x86 architecture with the content of the httptoolkit-server I compiled on the arm architecture, and now it can run normally

Yes, so the normal httptoolkit-desktop start & build process expects to pull a server from an official build. If you want to create a full desktop build without that, you can do so by manually extracting the build yourself into ./httptoolkit-server within the desktop repo checkout.

If you do so, then run npm run build, it should print Correct server already downloaded instead of Downloading .... After that it'll use your provided server version directly.

The only check it does is that a server exists and the version matches, i.e. the version field in ./httptoolkit-server/package.json matches the config->httptoolkit-server-version field in ./package.json.

@pimterry sorry was preoccupied with something. I ran npm install on this repo and I immediately get the following error.

npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated request-promise-native@1.0.5: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.3: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm notice
npm notice New minor version of npm available! 8.15.0 -> 8.18.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.18.0
npm notice Run npm install -g npm@8.18.0 to update!
npm notice
npm ERR! code 2
npm ERR! path /Users/quantum/Desktop/projects/learn/httptoolkit-server/node_modules/node-datachannel
npm ERR! command failed
npm ERR! command sh /var/folders/q7/qy7ng_2x7zsflljr40gfmjmm0000gn/T/install-6ada837e.sh
npm ERR! added 595 packages, removed 1 package, and audited 653 packages in 14s
npm ERR!
npm ERR! 66 packages are looking for funding
npm ERR!   run `npm fund` for details
npm ERR!
npm ERR! 4 high severity vulnerabilities
npm ERR!
npm ERR! To address all issues (including breaking changes), run:
npm ERR!   npm audit fix --force
npm ERR!
npm ERR! Run `npm audit` for details.
npm ERR!
npm ERR! > node-datachannel@0.3.4 _prebuild
npm ERR! > prebuild --backend cmake-js
npm ERR!
npm ERR! [
npm ERR!   '/Users/quantum/Library/Application Support/fnm/node-versions/v18.7.0/installation/bin/node',
npm ERR!   '/Users/quantum/Desktop/projects/learn/httptoolkit-server/node_modules/node-datachannel/node_modules/.bin/cmake-js',
npm ERR!   'rebuild',
npm ERR!   '--runtime-version=18.7.0',
npm ERR!   '--arch=arm64',
npm ERR!   '--runtime=node'
npm ERR! ]
npm ERR! prebuild-install WARN install No prebuilt binaries found (target=18.7.0 runtime=node arch=arm64 libc= platform=darwin)
npm ERR! npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm ERR! npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm ERR! npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! npm WARN deprecated tar@2.2.2: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! npm WARN deprecated tar@2.2.2: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! prebuild info begin Prebuild version 11.0.4
npm ERR! prebuild info build Preparing to prebuild node-datachannel@0.3.4 for node 18.7.0 on darwin-arm64 using cmake-js
npm ERR! info TOOL Using Ninja generator, because ninja is available.
npm ERR! ERR! OMG CMake executable is not found. Please use your system's package manager to install it, or you can get installers from there: http://cmake.org.
npm ERR! prebuild ERR! build Error: Failed to build cmake with exit code 1
npm ERR! prebuild ERR! build     at ChildProcess.<anonymous> (/Users/quantum/Desktop/projects/learn/httptoolkit-server/node_modules/node-datachannel/node_modules/prebuild/cmakebuild.js:32:19)
npm ERR! prebuild ERR! build     at ChildProcess.emit (node:events:513:28)
npm ERR! prebuild ERR! build     at ChildProcess._handle.onexit (node:internal/child_process:291:12)

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/quantum/.npm/_logs/2022-08-18T12_54_12_020Z-debug-0.log

I don't know which package is requiring this dependency. I have m1 laptop, so I can help here with no worries.

While we are on it, we can also switch other dependencies like request with the popular got package. But, that requires us to take radicle steps like changing npm project to esm only.

I don't know which package is requiring this dependency. I have m1 laptop, so I can help here with no worries.

Ah, ok, this is coming from node-datachannel, which is used in MockRTC, which powers some new WebRTC interception features.

They're actually working on M1 prebuilds to fix this, and but the main problem is that they're struggling to find somebody with an M1 to test the result. Could you help?

The issue is here: https://github.com/murat-dogan/node-datachannel/issues/120.

I'm not sure exactly what the test process should be for this, but I think you probably need to:

Check out that project
npm install (this will probably fail at the end, since there no prebuilds and you'll hit the same error as above, that's OK)
Download the .tar.gz M1 build from that issue
Extract it so you have the build/Release/node_datachannel.node file from that M1 build
Run the tests to check it works: npm test

If you run into any issues or you have any questions, ping in that issue and I'm sure they'll be very happy to help.

While we are on it, we can also switch other dependencies like request with the popular got package. But, that requires us to take radicle steps like changing npm project to esm only.

No need I think - we don't actually use request in the main codebase, it's just used as a test case in the tests, to make sure we can intercept it correctly if users use it. We actually already use Got in there too. That happens here tested here.

@pimterry where should I put node_datachannel.node file? Like I have node_modules folder and thousands of folders inside it. Where should I go from here.

(Replied on the other issue)

@shirshak55 now that that's fixed, can you try the server again here on your machine? I've just updated that subdependency to the new version with the prebuild for ARM64, so in theory if you pull that then everything should work. Maybe...

@pimterry hey, sorry for being late.

I think it works.

git clone --depth=1 https://github.com/httptoolkit/httptoolkit-server.git
Cloning into 'httptoolkit-server'...
remote: Enumerating objects: 220, done.
remote: Counting objects: 100% (220/220), done.
remote: Compressing objects: 100% (202/202), done.
remote: Total 220 (delta 11), reused 152 (delta 8), pack-reused 0
Receiving objects: 100% (220/220), 14.04 MiB | 12.41 MiB/s, done.
Resolving deltas: 100% (11/11), done.
cd httptoolkit-server/
npm i
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated request-promise-native@1.0.5: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.3: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

> httptoolkit-server@1.10.1 prepare
> ts-node ./prepare.ts

Installing override npm dependencies...
npm WARN config production Use `--omit=dev` instead.

added 28 packages, and audited 29 packages in 383ms

4 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
Override dependencies installed

added 1221 packages, and audited 1280 packages in 6s

54 packages are looking for funding
  run `npm fund` for details

11 vulnerabilities (2 moderate, 9 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
npm run
Lifecycle scripts included in httptoolkit-server@1.10.1:
  start
    node-dev ./bin/run start
  test
    cross-env TS_NODE_FILES=true mocha --exit -r ts-node/register 'test/**/*.spec.ts'

available via `npm run-script`:
  build:src
    rm -rf lib && tsc -b --force
  build:release
    oclif-dev manifest && webpack && ts-node ./pack.ts
  prepare
    ts-node ./prepare.ts
  prepack
    npm run build:src && oclif-dev manifest
  test:release
    cross-env TEST_BUILT_TARBALL=1 npm run test

npm run start

> httptoolkit-server@1.10.1 start
> node-dev ./bin/run start

(node:22756) [ENOENT] Error: spawn httptoolkit-server ENOENT
(Use `node --trace-warnings ...` to show where the warning was created)
Downloaded server directory is entirely outdated, deleting it
Config checked in 10 ms
Certificates setup in 21 ms
Standalone server started in 2 ms
Server started in 10 ms
Total startup took 43 ms

Superb! Thanks @shirshak55. I think we can close this issue now then, looks like low-level ARM64 support is all good in the server. That should help for people running with the server+browser setup manually in the short-term.

Full native support in the main HTTP Toolkit app is still going to be a work in progress for a while, since we'll need to start publishing server releases for ARM first, to fix https://github.com/httptoolkit/httptoolkit-desktop/issues/45, and then I think we'd need a way to build the desktop app for ARM within a GitHub Actions runner, which as far as I can tell isn't currently possible (but then might be some workarounds, or I wouldn't be surprised if GitHub added ARM runners in future which would make this much easier).

@pimterry what I can do is provide the HTTP toolkit server app manually. But yea there is going to be trust issues I guess. Or, I think you can purchase a mac mini and use it to publish the server which shouldn't cost that much for now?

what I can do is provide the HTTP toolkit server app manually.

We could do manual publishing to work around these limitations, you're totally right, but more than trust it's about the practical hassle of that. We run a fresh build for every single commit, plus a new deploy every few weeks or so. It's really nice that the app is built immediately for every target, automatically tested after every commit, and the release process is totally scripted so it's definitely exactly the same every time. That makes everything very reliable, and switching to a manual approach would have a lot of downsides for not much upside.

Fortunately, this is just a problem for the main desktop app. For the server we can cross-build for ARM64 from the normal Linux CI environment, because it's almost all JS, with just a couple of node modules that are (now) all prebuilt for ARM so just get downloaded. That's means no compilation involved, just downloading the correct prebuilds for each target.

We just need to configure the build process to build these new ARM targets, and then that'll be included in the next release. I've just done a test run for that here: https://github.com/httptoolkit/httptoolkit-server/actions/runs/2920847127. If you have a minute, could can you download & extract the distributables artifact from that build, then extract v1.10.1/httptoolkit-server-v1.10.1-darwin-arm64.tar.gz from in there, and in that folder run ./bin/httptoolkit-server start? If that works then I can merge it, and then we'll have full ARM support in the server (but not yet the desktop!) in the next release.

The desktop is harder I think, because I'm not sure Electron-Builder will happily build for the wrong architecture like this. We currently run a separate build on a different OS to generate each Electron app version. To be honest I've never tried though, there might be some good options there, once the server part is sorted out. GitHub doesn't yet have hosted runners for Mac M1 (https://github.com/actions/runner-images/issues/2187) but they did announce support a few weeks ago for self-hosting an M1 GitHub runner, so if it comes down to it I might buy that M1 mini after all... :smile:

httptoolkit / httptoolkit-server

build for arm64 failed #51