Open joshuakarp opened 3 years ago
In working on core activity 2 of our R&D report, I've discovered some prior context for this particular issue.
This comment https://github.com/MatrixAI/js-polykey/pull/43#issuecomment-644652639 explains how we came to be using node-forge
as opposed to other cryptographic libraries.
I know that when Polykey project first started, we initially were thinking of using PGP and thus the kbpgp.js library (https://github.com/keybase/kbpgp). We ended up going away from PGP due to its limited usage in a number of scenarios that we want PK to deal with. Namely end to end encrypted network communication which is a TLS issue, that makes use of X.509 certificates rather than PGP certificates. Furthermore we also had symmetric encryption/decryption scenarios like js-encryptedfs that again would not make use of PGP standards. Therefore it just lacked interoperability with many other cryptographic scenarios, it seemed like its own island of standards, the library can still be brought back in in the future if we find usecases for PK using PGP.
However in choosing node-forge, we came across a few other problems. Mainly overall-cross platform compatibility planning for mobile devices. This is not just a problem with crypto, but also other libraries that are used in our networking
domain such as utp-native.
Here are something I found that may be relevant to us proceeding here:
node-forge
)All of this will mean that we either replace node-forge, or end up creating a adapter pattern where we plugin different crypto implementions depending on our environment. At this point in time, the keys
domain abstracts over most(all?) crypto operations for all other domains. Except in the case of EFS which is currently pinned to node-forge
(it may be a good idea to abstract that and expect an interface of functions for EFS).
Cross platform compatibility here isn't just about the fundamental crypto library. It's also about other parts of PK. One closely related situation is the JOSE libraries. As they involve cryptographic operations, they currently seem to "fix" their underlying crypto library as well. It would be ideal that if we standardise on a crypto library for cross-platform deployment, that we can also ensure that our JOSE library is using the same crypto library to reduce our crypto attack surface. We are currently using https://github.com/panva/jose which uses native crypto depending on the platform including webcrypto. Contenders include https://github.com/cisco/node-jose (which fixes on node-forge) and https://github.com/square/js-jose.
Still to research:
Regarding the future of using web assembly, here are relevant resources:
The last link is interesting as it describes how to use webcrypto together with webassembly when you need a custom crytographic functionality not supported inside webcrypto.
Another example of wasm and webcrypto usage:
One note about WASM. There are 2 well supported runtimes for WASM:
So that does mean wasmer can compile WASM code to native code. But wasm3 is for interpretation. Why use interpetation?
It appears that in some cases interpretation can be more widely deployed. There are examples of iOS apps using wasm3. https://github.com/kateinoigakukun/wasmic-ios
It's becoming fast a standard target for many languages. Even TypeScript when ported to AssemblyScript can be compiled to WASM.
Once it is WASM, the only thing missing is broad adoption of WASI. If WASI is broadly adopted like it is in nodejs (https://github.com/nodejs/uvwasi), then pretty much we have a universal portable binary capable of doing relevant system operations. WASI is like a universal standard of system calls. Like a whole new POSIX standard.
Then one would just use WASM and WASI for all platforms.
Dynamic imports can be used to inject platform-specific implementation adapters.
async function test () {
// dynamic imports work now!
const p = await import('perf_hooks');
console.log(p);
}
test();
So if we needed to supply different implementations for different platforms, we could do a feature/platform test, and then dynamically import the right module.
Important thing is that it is asynchronous, so it would have to be wrapped up in a class, or function.
TS compiler will translate the above into a promise that acquires modules using require
in commonjs or other ways depending on the platform.
There is top level await in NodeJS as well, but I'm not sure it will work across all the platforms.
Could be important for all these areas:
Current attempt at using top level await to do this is located at the js-id work: https://github.com/MatrixAI/js-id/issues/1#issuecomment-936105951. Notice that there's quite a few changes required, and we would need to prototype a working solution in TypeScript-Demo-Lib. I reckon, this is something that will require TS 4.5 to be released completely, and ts-node support in https://github.com/TypeStrong/ts-node/issues/1007.
Moved the discussions about crypto into its own issue #270. All issues that relate to the webcrypto integration/migration should point to that.
This discussion at deno https://github.com/denoland/deno/issues/6607 is interesting.
If you want to run deno on ios, you have to compile it on ios then:
You can already run Deno in V8 jitless mode: deno run --v8-flags="--jitless". It is just painfully slow. V8 lite mode is also supported via --v8-flags="--lite-mode". I am not sure everything works, but it is enough for initial testing.
The v8 runtime is the most important piece. Then higher level interfaces built around it like node or deno or nativescript. Which means as long as we stick to compatible Ecmascript, it should work, and for our native addons.
Note that deno will never support nodejs native addons. That's because nodejs native addons using things like n-api is based on the node wrapper around v8.
That doesn't mean C/C++ libraries won't work, it's just that the bindings would have to be written differently for deno vs for nodejs. Instead of n-api, you'd use https://deno.land/manual/runtime/ffi_api.
Anyway deno is still too early to be utilised, but it is very promising runtime for the future of TS/wasm applications.
Other than reliance on nodejs APIs, these are the libraries that are "native" that we rely upon:
fd-lock
leveldown
utp-native
The fd-lock
is fairly easy to substitute, we are likely to replace it with @matrixai/js-file-locks
.
The leveldown
uses leveldb
and we are replacing it with @matrixai/js-db
, which itself may continue to use leveldb internally.
The utp-native
is not easily substituted, and I find its code quite flaky, I'd prefer to use a QUIC implementation, and this may mean using something like MsQuic which is native code that can be compiled for different platforms.
In terms of porting for mobile, we must first figure out: Should we even try to get nodejs itself running on mobile? - The answer to this determines whether we can continue using native-addons for nodejs.
If nodejs worked on the mobile platforms, then any native addon should continue to work as long as the relevant platform-specific compilation is available, they would be compiled to a shared object node.napi.node
file which will be dynamically loaded. This is the case for leveldown
or classic-level
, where there are prebuilt-binaries for android (not yet for ios). Because these addons all use the "napi" system, they are compatible from one node version to another. But also because they use the "napi" system, they are limited to working with nodejs runtime.
So how does one get the nodejs runtime on mobile? There are several options, and all are difficult because nodejs upstream does not maintain any releases to mobile, instead this has been done by third parties:
Option 1. is basically the idea of "embedding nodejs as a library". This seems to be possible simply because of https://nodejs.org/api/embedding.html. However it seems a rare approach, and I guess many nodejs applications won't work due to expectations on the operating environment.
Option 2. is basically like Electron, where there is a "nodejs" backend process runtime is running completely independently, and then you just have the host-app frontend call the backend process using network calls (usually localhost TCP). Electron does the same thing with the nodejs backend process with some message passing APIs/rpc to the frontend (where the frontend is a "browser" chromium process).
Option 2 is popular because it's the same architecture applied to mobile settings, and it appears that's how one would integrate a nodejs backend to react-native. React-native just provides the ability to use JS as the host-app.
Similarly nativescript is just allowing you to use JS as the host-app.
Now why is option 1. quite rare? I suspect it has to do with the fact that node relied on v8, and until recently v8 was not runnable on ios due to its JIT usage. Only recently did its jitless mode become available, and so both NativeScript and nodejs-mobile swapped their JS runtime to v8 for both android and ios. Prior to this, both ecosystems had to maintain separate JS runtimes for android and ios. So now that the v8 is runnable on ios with jitless mode (https://v8.dev/docs/cross-compile-arm), it should mean that it would be easier to implement option 1 and option 2.
In terms of feasibility, running node as a backend process is what we are planning to do for electron (there are issues here to be solved too: #235).
As for mobile, the only ones that seem possible is liquidcore or using termux patches for android, or taking nodejs-mobile and patching it up to support v16.
This is all so that nodejs
can be the one running. If we were able to substitute all the nodejs APIs for portable JS/WASM, then we are left with only the native addons that require consideration.
I believe all the native addons could be separately implemented, by directly compiling native code for android/ios.
For example, the @matrixai/js-file-locks
could just use direct android and ios API calls.
The @matrixai/js-db
which uses leveldb can just directly compile leveldb to ios/android and use a native FFI to call them, the JS code can still run, but it would need to abstracted over so instead of calling the C++ native addon using NAPI, they call code provided via native FFI to android/ios specific code.
As for the network, that's the only complexity. I haven't explored how msquic work.
See also:
We can try to explore option 2 to see if it is feasible. And if we hit some roadblocks, we may just forget about getting nodejs running on mobile, and instead substitute our native libraries with android/ios compatible shim code which would likely require utilising the nativescript framework.
Outside of native addons, all other JS should be portable JS and thus we must stop relying on nodejs APIs (except where it involves the OS side-effects), in those cases those things must be substituted as well.
Another option is to target wasm from everything and then get a wasm interpreter that works on android/ios. That is WASM is then the portable intermediate code for everything. But this is all quite experimental. Right now there's no JS to WASM compiler, the closest is assemblyscript. Alternatively you compile the JS runtime to WASM, and then use that WASM program to run JS, like compiling v8 or quickjs to WASM.
Option 1 can be further explored, see if nodejs really can be compiled with jitless mode for android/ios. The only question is the calling interface would have to be mediated somehow.
I've just surveyed the situation with React Native and NativeScript again, and it seems React Native is the better option now. NS support across all the frameworks seems breaking.
The ability to call into native code is improving with turbomodules.
Nodejs would still not work so native addons on nodejs would have to be converted to native code specific to Android/iOS.
Webassembly support is not available yet on react native (some work arounds: https://github.com/inokawa/react-native-react-bridge), it may be available in the future, as we are looking to use this to replace some of the crypto APIs that would be missing.
NodeJS APIs would need to be written with portable JS. Only crypto would be the issue.
Created by @CMCDragonkai
No js-polykey cannot be used in browsers. However for NS, we may need to polyfill alot of our dependencies.
The js-polykey doesn't need to do this. But in Polykey-NativeScript we are going to we webpack5 and have to resolve some packages.
Webpack 5 no longer polyfills node core modules, but we can do it manually:
These are:
Some of these can be polyfilled for NS.
In other cases, we will need to write alternative implementations when they don't exist, like for example the usage of
fs
- https://docs.nativescript.org/ns-framework-modules/file-systemRelated: