Support Node.js and browser specific packages in NativeScript application

[rosen-vladimirov]

Description

There are more than 700 000 packages in npm. Most of them cannot be used in NativeScript application as they have some browser or Node.js specific calls, which cannot work in the NativeScript runtime. It will be great if there's an easy way to use all of these plugins in the application. This could be done in several ways:

• Provide polyfills for all Node.js and browser specific APIs - this way the packages will work. However, this requires implementing a huge amount of APIs in the NativeScript framework itself. When new API is added or change in Node.js for example, we will have to assert it behaves in the same way in NativeScript.
• Replace specific dependencies during project build with their nativescript equivalents. For example, in case you have a project that is shared for web and mobile use, you may have a single package.json file. When building the NativeScript application, you may want to replace some of the dependencies with other ones, that provide the same API. This way you'll not have to change the code, but during build, the correct NativeScript dependency will be used.
• Replace specific files of the dependency when building the project with NativeScript specific ones - in many cases only specific files of the package should be replaced in order to make it work in NativeScript. The replacements could be defined and package level or application level.

Suggestion

The current suggestion to implement this feature is to make CLI replace specific files and packages during build of the NativeScript application. How can we achieve this:

1. Allow declaring replacement details in the nativescript key inside package.json
2. The replacement data should allow replacements of dependencies and specific files.
3. Replacement could be defined in the plugin and in the project itself.
4. Project replacements should be executed last, this will allow "fixing" a plugin that has incorrect replacement data (for example forgotten to replace some files).
5. Replacement should allow excluding some files from the plugins source, in case they are not compatible with the NativeScript project
6. We may add a different entry point to the plugin, when it is used inside nativescript environment.

Example of plugin's package.json:

{
    "name": "nativescript-plugin",
    "version": "1.0.0",
    "main": "index.js",
    "nativescript": {
        "platforms": {
            "android": "3.0.0",
            "ios": "3.0.0"
        },
        "replacementData": {
            "main": "./lib/nativescript/index.js",
            "dependencies": {
                "firebase": {
                    "nativescript-plugin-firebase": "2.0.0"
                }
            },
            "files": {
                "./lib/service.js": "./lib/nativescript/service.js"
            },
            "exclude": [
                "./server/**/*",
                "./lib/**/*.client.js"
            ]
        }
    },
    "dependencies": {
        "firebase": "*"
    },
    "author": "rosen-vladimirov",
    "license": "Apache-2.0"
}

When this nativescript-plugin is added to the application and CLI builds the NativeScript project, the firebase dependency should be replaced with nativescript-plugin-firebase. However, this leads to problem in the plugin's code when you use require("firebase"). CLI will have to replace the require("firebase") with require("nativescript-plugin-firebase") in the application. We'll have to modify the .ts, .js files, Angular specific files, etc. During building the application, CLI should move all plugin files to the <project dir>/platforms/<platform>/.../tns_modules/nativescript-plugin/ directory. At this point CLI should move only files, which are not included in exclude patterns, i.e. all files under server directory should not be included. Also all files under lib directory, which are named <smth>.client.js should not be moved to tns_modules. After that CLI should replace the content of lib/service.js file with the content of lib/nativescript/service.js file (directly in the tns_modules/nativescript-plugin directory. The last part is the replacement of the entry point. CLI can directly replace the content of the index.js with the content of lib/nativescript/index.js file. This approach will allow the plugin authors to allow using their plugin in many environments: server side, browser, nativescript, etc.

Same rules can be applied based on project's package.json. For example, it may contain information how to replace specific files from the project, for some plugins, etc.

CLI should support with and without using webpack for building the application.

NOTE: Part of this approach is already implemented by nativescript-nodeify plugin.

[NathanaelA]

I tend to think that Polyfill would be the safest; this replacement system seems like it is fraught with lots of weird gotcha's, that can then be broken by changes in other tooling outside of your control. As it is changes in tooling outside seems to frequently cause CLI issues. ;-(

I actually started a polyfill (https://github.com/NathanaelA/nativescript-node-compat) a couple years ago; but no one helped, I think I polyfilled most of the "fs".

[sean-perkins]

I agree with Nathanael, polyfills do seem like the safest overall option. It still is limiting and could cause a lot of package pipeline issues downstream or under the hood, but providing limited support is still useful.

I am curious of the actual benefits of supporting more NPM packages. Shouldn't we be relying more on the underlying bridge to native libraries, instead of allocating more of the JS runtime? Some packages I see immediate benefit that have strong node.js libraries, but a lot of those vendors supply native sdks as well to tap into.

[sitefinitysteve]

+1 for being nervous about magical replacing of code. Man like something gets replaced in an npm package, breaks the core plugin somehow, then that plugin author starts getting issues submitted due to this, etc...

I would rather see HMR, Killer debugging, Swift support (w/out Objc Headers) before spending time on this.

[EddyVerbruggen]

There's no doubt in my mind fi. HMR is more important, but that's a different discussion.

The problem this issue tries to fix, is currently replacing node modules is magic (when using the Nodeify plugin, which I don't encourage anyone to do), and the aim is to make it not magic - only the dependencies in the problematic modules should be replaced, not affecting others.

As an example, I recently wanted to control a Particle Photon with NativeScript. These babies are controlled via a REST API so it should be easy enough. However, it's 2018 and nobody writes their own REST API wrappers anymore, and conveniently, Particle has a JavaScript SDK wrapping their API for Node and the browser, as well as iOS and Android SDKs doing the same.

So the first thing I tried was using their JS SDK because that would mean I needn't create (and maintain!) a plugin wrapping their native SDKs. Almost immediately I hit a showstopper though: the JS SDK requires the native (built-in) Node crypto module because of a dependency on "superagent" which in turn depends on "formidable", which.. you guessed it: requires crypto.

Developers of Node libraries that also need to run in the browser have fixed this ages ago by providing a browser node in package.json which is used by fi. Browserify to replace specific modules, as well as built-in Node modules (like the aforementioned crypto) - that's why their JS SDK works in the browser as well.

Note that React Native takes the same approach, as shown in this case, for AWS.

If the NativeScript CLI would be able to parse a "nativescript" node in package.json, we could send a PR to Particle's JS SDK, adding (among other things):

"nativescript": {
  "crypto": "nativescript-crypto"
}

Where "nativescript-crypto" is a stub, very much like @NathanaelA's nativescript-fs module. Or we could probably even use this readily available browser-compatible version that I've used before in a NativeScript app:

"nativescript": {
  "crypto": "crypto-browserify"
}

This will have the NativeScript CLI swap out any requires of crypto by a NativeScript-runtime compatible version, but only local to this package - not affecting other modules in your app's bundle.

The big win here is once merged to the Particle JS SDK repo, any future updates to their SDK also work with NativeScript (unless new incompatible dependencies are added, of course), and NativeScript users will always be immediately able to use the latest bits. As opposed to having to wait for my lazy a$$ to update the nativescript-particle plugin and bump (and test) the new iOS and Android SDKs. It's a win-win-win IMO.