React Native and Mobile OS (iOS and Android) Compatibility

[joshuakarp]

Created by @CMCDragonkai

No js-polykey cannot be used in browsers. However for NS, we may need to polyfill alot of our dependencies.

The js-polykey doesn't need to do this. But in Polykey-NativeScript we are going to we webpack5 and have to resolve some packages.

Webpack 5 no longer polyfills node core modules, but we can do it manually:

• https://github.com/Richienb/node-polyfill-webpack-plugin/blob/master/package.json
• https://github.com/webpack/webpack/pull/8460/commits/a68426e9255edcce7822480b78416837617ab065

These are:

    assert: "assert",
    buffer: "buffer",
    console: "console-browserify",
    constants: "constants-browserify",
    crypto: "crypto-browserify",
    domain: "domain-browser",
    events: "events",
    http: "stream-http",
    https: "https-browserify",
    os: "os-browserify/browser",
    path: "path-browserify",
    punycode: "punycode",
    process: "process/browser",
    querystring: "querystring-es3",
    stream: "stream-browserify",
    _stream_duplex: "readable-stream/duplex",
    _stream_passthrough: "readable-stream/passthrough",
    _stream_readable: "readable-stream/readable",
    _stream_transform: "readable-stream/transform",
    _stream_writable: "readable-stream/writable",
    string_decoder: "string_decoder",
    sys: "util",
    timers: "timers-browserify",
    tty: "tty-browserify",
    url: "url",
    util: "util",
    vm: "vm-browserify",
    zlib: "browserify-zlib"

Some of these can be polyfilled for NS.

In other cases, we will need to write alternative implementations when they don't exist, like for example the usage of fs - https://docs.nativescript.org/ns-framework-modules/file-system

Related:

• https://github.com/matrixai/js-polykey/issues/173
• https://github.com/matrixai/js-polykey/issues/168

[CMCDragonkai]

In working on core activity 2 of our R&D report, I've discovered some prior context for this particular issue.

This comment https://github.com/MatrixAI/js-polykey/pull/43#issuecomment-644652639 explains how we came to be using node-forge as opposed to other cryptographic libraries.

I know that when Polykey project first started, we initially were thinking of using PGP and thus the kbpgp.js library (https://github.com/keybase/kbpgp). We ended up going away from PGP due to its limited usage in a number of scenarios that we want PK to deal with. Namely end to end encrypted network communication which is a TLS issue, that makes use of X.509 certificates rather than PGP certificates. Furthermore we also had symmetric encryption/decryption scenarios like js-encryptedfs that again would not make use of PGP standards. Therefore it just lacked interoperability with many other cryptographic scenarios, it seemed like its own island of standards, the library can still be brought back in in the future if we find usecases for PK using PGP.

However in choosing node-forge, we came across a few other problems. Mainly overall-cross platform compatibility planning for mobile devices. This is not just a problem with crypto, but also other libraries that are used in our networking domain such as utp-native.

Here are something I found that may be relevant to us proceeding here:

• The webcrypto standard (https://w3c.github.io/webcrypto/) is now the official standard for cryptographic operations in browser environments. It is now being supported by all major browsers and electron
• This webcrypto standard is now taking over other deployment platforms such as in Nodejs: https://nodejs.org/api/webcrypto.html & https://www.nearform.com/blog/implementing-the-web-cryptography-api-for-node-js-core/
• As a standardised API, switching over to this API gives us more cross-platform opportunity as the rest of the world's development ecosystem catches up and migrates over to webcrypto standard.
• In particular are nativescript and react-native ecosystems. Currently none of them have official crypto APIs, they expect the developer to use underlying native iOS or Android crypto APIs.
• However I found out that the GUN project https://github.com/amark/gun#additional-cryptography-libraries claims to be able to use webcrypto on react-native and other platforms.
• It does this through a "webview" trick. Basically this hooks into the browser runtimes that are on iOS and Android to perform the actual crypto and then return the results back to the main application. I have no idea about the performance characteristics of this trick. This trick is described here: https://gun.eco/docs/React-Native and https://github.com/webview-crypto/react-native-webview-crypto
• The usage of webcrypto libraries is pioneered by "PeculiarVentures" which has these main libraries:
  • https://github.com/PeculiarVentures/PKI.js - this could be an alternative to node-forge for all of our PKI/X.509 and TLS related functionality (previously it was claimed this was overly complex compared to node-forge)
  • https://github.com/PeculiarVentures/webcrypto - this is a generic polyfill for webcrypto, not entirely sure how it wraps the native webcrypto inside nodejs or if smooths over the differences, there is a discussion about this library in relation to other webcrypto polyfills targeting nodejs https://github.com/PeculiarVentures/webcrypto, because this is also mentioned with respect to the webview trick, there may also be a relationship between the webview trick and this polyfill
  • They have many other crypto related libraries that we should investigate

All of this will mean that we either replace node-forge, or end up creating a adapter pattern where we plugin different crypto implementions depending on our environment. At this point in time, the keys domain abstracts over most(all?) crypto operations for all other domains. Except in the case of EFS which is currently pinned to node-forge (it may be a good idea to abstract that and expect an interface of functions for EFS).

Cross platform compatibility here isn't just about the fundamental crypto library. It's also about other parts of PK. One closely related situation is the JOSE libraries. As they involve cryptographic operations, they currently seem to "fix" their underlying crypto library as well. It would be ideal that if we standardise on a crypto library for cross-platform deployment, that we can also ensure that our JOSE library is using the same crypto library to reduce our crypto attack surface. We are currently using https://github.com/panva/jose which uses native crypto depending on the platform including webcrypto. Contenders include https://github.com/cisco/node-jose (which fixes on node-forge) and https://github.com/square/js-jose.

Still to research:

• utp-native and networking libraries
• others...?

[CMCDragonkai]

Regarding the future of using web assembly, here are relevant resources:

• https://www.fastly.com/blog/standard-cryptography-api-webassembly
• https://github.com/WebAssembly/wasi-crypto
• https://stackoverflow.com/questions/48891593/webcryptoapi-vs-webassembly-encryption-module
• https://medium.com/nexenio/a-random-story-on-webcrypto-and-webassembly-6f9e00c73a

The last link is interesting as it describes how to use webcrypto together with webassembly when you need a custom crytographic functionality not supported inside webcrypto.

[CMCDragonkai]

Another example of wasm and webcrypto usage:

• https://medium.com/cubbit/how-to-build-a-crypto-isomorphic-library-with-javascript-and-webassembly-6fc7aa708437
• https://github.com/cubbit/enigma and https://github.com/cubbit/enigma/issues/16

[CMCDragonkai]

One note about WASM. There are 2 well supported runtimes for WASM:

• wasm3 - supports interpretation
• wasmer - supports JIT and AOT compilation

So that does mean wasmer can compile WASM code to native code. But wasm3 is for interpretation. Why use interpetation?

It appears that in some cases interpretation can be more widely deployed. There are examples of iOS apps using wasm3. https://github.com/kateinoigakukun/wasmic-ios

It's becoming fast a standard target for many languages. Even TypeScript when ported to AssemblyScript can be compiled to WASM.

Once it is WASM, the only thing missing is broad adoption of WASI. If WASI is broadly adopted like it is in nodejs (https://github.com/nodejs/uvwasi), then pretty much we have a universal portable binary capable of doing relevant system operations. WASI is like a universal standard of system calls. Like a whole new POSIX standard.

Then one would just use WASM and WASI for all platforms.

[CMCDragonkai]

Dynamic imports can be used to inject platform-specific implementation adapters.

async function test () {
  // dynamic imports work now!
  const p = await import('perf_hooks');
  console.log(p);
}

test();

So if we needed to supply different implementations for different platforms, we could do a feature/platform test, and then dynamically import the right module.

Important thing is that it is asynchronous, so it would have to be wrapped up in a class, or function.

TS compiler will translate the above into a promise that acquires modules using require in commonjs or other ways depending on the platform.

There is top level await in NodeJS as well, but I'm not sure it will work across all the platforms.

Could be important for all these areas:

• GRPC #235
• Cryptography (web crypto)
• Networking #234

Current attempt at using top level await to do this is located at the js-id work: https://github.com/MatrixAI/js-id/issues/1#issuecomment-936105951. Notice that there's quite a few changes required, and we would need to prototype a working solution in TypeScript-Demo-Lib. I reckon, this is something that will require TS 4.5 to be released completely, and ts-node support in https://github.com/TypeStrong/ts-node/issues/1007.

[CMCDragonkai]

Moved the discussions about crypto into its own issue #270. All issues that relate to the webcrypto integration/migration should point to that.

[CMCDragonkai]

This discussion at deno https://github.com/denoland/deno/issues/6607 is interesting.

• V8 has a jitless mode that can be run on ios
• NativeScript already moved to jitless mode for v8 https://www.infoq.com/news/2020/01/nativescripts-adopts-V8-ios/
• Both nodejs and deno uses v8

If you want to run deno on ios, you have to compile it on ios then:

You can already run Deno in V8 jitless mode: deno run --v8-flags="--jitless". It is just painfully slow. V8 lite mode is also supported via --v8-flags="--lite-mode". I am not sure everything works, but it is enough for initial testing.

The v8 runtime is the most important piece. Then higher level interfaces built around it like node or deno or nativescript. Which means as long as we stick to compatible Ecmascript, it should work, and for our native addons.

Note that deno will never support nodejs native addons. That's because nodejs native addons using things like n-api is based on the node wrapper around v8.

That doesn't mean C/C++ libraries won't work, it's just that the bindings would have to be written differently for deno vs for nodejs. Instead of n-api, you'd use https://deno.land/manual/runtime/ffi_api.

Anyway deno is still too early to be utilised, but it is very promising runtime for the future of TS/wasm applications.

[CMCDragonkai]

Other than reliance on nodejs APIs, these are the libraries that are "native" that we rely upon:

• fd-lock
• leveldown
• utp-native

The fd-lock is fairly easy to substitute, we are likely to replace it with @matrixai/js-file-locks.

The leveldown uses leveldb and we are replacing it with @matrixai/js-db, which itself may continue to use leveldb internally.

The utp-native is not easily substituted, and I find its code quite flaky, I'd prefer to use a QUIC implementation, and this may mean using something like MsQuic which is native code that can be compiled for different platforms.

In terms of porting for mobile, we must first figure out: Should we even try to get nodejs itself running on mobile? - The answer to this determines whether we can continue using native-addons for nodejs.

If nodejs worked on the mobile platforms, then any native addon should continue to work as long as the relevant platform-specific compilation is available, they would be compiled to a shared object node.napi.node file which will be dynamically loaded. This is the case for leveldown or classic-level, where there are prebuilt-binaries for android (not yet for ios). Because these addons all use the "napi" system, they are compatible from one node version to another. But also because they use the "napi" system, they are limited to working with nodejs runtime.

So how does one get the nodejs runtime on mobile? There are several options, and all are difficult because nodejs upstream does not maintain any releases to mobile, instead this has been done by third parties:

1. Compile node as a native library to be embedded in the mobile application, where the calling interface is a direct FFI from the host-app written in Java or Swift directly into nodejs APIs
   • node on android - https://github.com/node-on-mobile/node-on-android - an old project that hasn't been updated for a while
   • https://github.com/nodejs/node/issues/34115
2. Compile node to run as a separate executable thread, which then acts like a service/virtual-machine that's embedded inside the app
   • nodejs-mobile - https://github.com/JaneaSystems/nodejs-mobile - a fork of nodejs that matintains patches to compile for android and ios, this is the latest work however support has been discontinued, this is what would have been suitable for what we want, however if we are to use it, we would need to take over maintenance since the last release is v12 and we are on v16 lts (https://www.janeasystems.com/blog/significant-updates-in-the-new-release-of-node-js-mobile/)
   • termux - https://github.com/termux/termux-packages/tree/master/packages/nodejs - termux is a "shell" environment for android, and they maintain patches to nodejs source code that is required in order to compile for android, however there is no equivalent termux for ios, this is also the most up-to-date supporting v18 of nodejs, so you could just fork the source patches and rebuild nodejs there https://github.com/termux/termux-app/issues/1458
   • https://github.com/LiquidPlayer/LiquidCore - it's like running a "nodejs virtual machine", then the host-app can interact with the nodejs virtual machine over a network (but also other more "native FFIs" are available), it's quite an interesting architecture, because the runtime is virtualised, it even has its own virtual filesystem

Option 1. is basically the idea of "embedding nodejs as a library". This seems to be possible simply because of https://nodejs.org/api/embedding.html. However it seems a rare approach, and I guess many nodejs applications won't work due to expectations on the operating environment.

Option 2. is basically like Electron, where there is a "nodejs" backend process runtime is running completely independently, and then you just have the host-app frontend call the backend process using network calls (usually localhost TCP). Electron does the same thing with the nodejs backend process with some message passing APIs/rpc to the frontend (where the frontend is a "browser" chromium process).

Option 2 is popular because it's the same architecture applied to mobile settings, and it appears that's how one would integrate a nodejs backend to react-native. React-native just provides the ability to use JS as the host-app.

Similarly nativescript is just allowing you to use JS as the host-app.

Now why is option 1. quite rare? I suspect it has to do with the fact that node relied on v8, and until recently v8 was not runnable on ios due to its JIT usage. Only recently did its jitless mode become available, and so both NativeScript and nodejs-mobile swapped their JS runtime to v8 for both android and ios. Prior to this, both ecosystems had to maintain separate JS runtimes for android and ios. So now that the v8 is runnable on ios with jitless mode (https://v8.dev/docs/cross-compile-arm), it should mean that it would be easier to implement option 1 and option 2.

In terms of feasibility, running node as a backend process is what we are planning to do for electron (there are issues here to be solved too: #235).

As for mobile, the only ones that seem possible is liquidcore or using termux patches for android, or taking nodejs-mobile and patching it up to support v16.

This is all so that nodejs can be the one running. If we were able to substitute all the nodejs APIs for portable JS/WASM, then we are left with only the native addons that require consideration.

I believe all the native addons could be separately implemented, by directly compiling native code for android/ios.

For example, the @matrixai/js-file-locks could just use direct android and ios API calls.

The @matrixai/js-db which uses leveldb can just directly compile leveldb to ios/android and use a native FFI to call them, the JS code can still run, but it would need to abstracted over so instead of calling the C++ native addon using NAPI, they call code provided via native FFI to android/ios specific code.

As for the network, that's the only complexity. I haven't explored how msquic work.

See also:

• https://stackoverflow.com/questions/45459909/viable-options-for-running-nodejs-on-android-aug-2017

[CMCDragonkai]

We can try to explore option 2 to see if it is feasible. And if we hit some roadblocks, we may just forget about getting nodejs running on mobile, and instead substitute our native libraries with android/ios compatible shim code which would likely require utilising the nativescript framework.

Outside of native addons, all other JS should be portable JS and thus we must stop relying on nodejs APIs (except where it involves the OS side-effects), in those cases those things must be substituted as well.

[CMCDragonkai]

Another option is to target wasm from everything and then get a wasm interpreter that works on android/ios. That is WASM is then the portable intermediate code for everything. But this is all quite experimental. Right now there's no JS to WASM compiler, the closest is assemblyscript. Alternatively you compile the JS runtime to WASM, and then use that WASM program to run JS, like compiling v8 or quickjs to WASM.

Option 1 can be further explored, see if nodejs really can be compiled with jitless mode for android/ios. The only question is the calling interface would have to be mediated somehow.

[CMCDragonkai]

I've just surveyed the situation with React Native and NativeScript again, and it seems React Native is the better option now. NS support across all the frameworks seems breaking.

The ability to call into native code is improving with turbomodules.

Nodejs would still not work so native addons on nodejs would have to be converted to native code specific to Android/iOS.

Webassembly support is not available yet on react native (some work arounds: https://github.com/inokawa/react-native-react-bridge), it may be available in the future, as we are looking to use this to replace some of the crypto APIs that would be missing.

NodeJS APIs would need to be written with portable JS. Only crypto would be the issue.