Idea: auto-patch native addon repos

[staltz]

Compiling native addons is the biggest hurdle in using nodejs-mobile.

I had an idea which should improve developer experience. In the ideal scenario, you don't need to do anything about native addons, they will compile correctly no matter what. How can we achieve that ideal scenario? If all native addon dependencies would support nodejs-mobile in their binding.gyp or whatever. But that's not realistically happening, not in the present and not in the future. So what some people do is fork those dependencies, fix their binding configurations, and then brute-force replace the original dependency with the forked dependency.

What if nodejs-mobile took ownership of such "forks" such that we begin accumulating a set of patches for well-known native addons? In practice, we could have a new repo nodejs-mobile/nodejs-mobile-addons with patch files for various well known npm libraries (and their respective versions), and then using patch-package during nodejs-mobile-react-native build scripts we apply patches to that library prior to starting compilation.

Then, whenever a native addon doesn't compile for nodejs-mobile, we treat that as an issue and "fix" it by creating a patch file in nodejs-mobile-addons. Of course, these patches can be promoted to PRs in the upstream repositories, and hopefully some PRs will be merged, but we can't count on all dependencies accepting such PRs.

[achou11]

think this is a nice idea! not sure if I'm adamant about having nodejs-mobile taking on the maintenance burden for these patches by hosting a repo for them, but if you think that's reasonable then not going to discourage it 😄

based on my experience, what would generally be helpful is for some documentation/guidance about what kind of patches are needed and how to go about that, so that end-users can apply these based on their own needs/constraints.

[staltz]

We can do both (this issue's idea, and better docs how to handle native addons).

Ideally native addons is not a concern that nodejs-mobile (as a project) should be responsible for, but the patches would exist on a "temporary" basis because they should be upstreamed as PRs.

[staltz]

Documenting here in public, mostly for my own sake as I'm pausing work on this and want to resume easily in the future.

I started implementing this in nmrn and the basic idea works: identify all the patch files we have in a directory structure patches/${moduleName}/${versionGlob}.patch, scan through existing node_modules looking for moduleName matches, then parse their package.json version and see if it matches versionGlob, then apply ${versionGlob}.patch file.

One important problem is how to support noderify or esbuild. nmrn Android script differs from the iOS script. The iOS script copies everything from nodejs-assets and then compiles native modules. The Android script has two separate copy tasks, one copy task dedicated for compiling native modules then plucking specific .node files out of that. This means that in projects like Manyverse and others, for Android you should (1) compile native modules, (2) apply noderify, (3) present a nodejs-assets folder that has no node_modules, (4) let nmrn take over the rest. But for iOS you should (1) apply noderify, (2) present a nodejs-assets folder that HAS node_modules, (4) let nmrn do the rest including compile native modules.

This presents a challenge for ${moduleName}/${versionGlob}.patch patches that would have to modify js files which need to ultimately be noderified.

I have two ideas:

Make iOS and Android scripts consistent: to improve dev experience (e.g. in projects structured like Manyverse), have the same order of tasks in scripts for both platforms.

Include esbuild in nmrn scripts: currently application of bundlers (esbuild, noderify) is left to nmrn users, but we could internalize this. We might have to support tweaking/configuring esbuild, so maybe that could be done in the project's package.json under some new field "nodejs-mobile-react-native". Example:

{
  ...
  "dependencies": {
    ...
  },
  "nodejs-mobile-react-native": {
    "esbuild": {
      "enabled": true,
      "entryPoint": "main.js"
      "outfile": "bundle.js",
      "aliases": [
        "node-extend=xtend",
        "chloride=sodium-chloride-native-nodejs-mobile"
      ],
      "external": [
        "bl",
        "supports-color"
      ]
    }
  }
}

For reference, sequence of tasks in nmrn Android scripts versus iOS scripts:

Android

graph TB;

CopyNodeProjectAssetsFolder
GenerateNodeProjectAssetsLists
preBuild

subgraph if [if shouldRebuildNativeModules]
  direction TB
  ApplyPatchScriptToModules
  CopyNodeProjectAssetsARCH
  DeleteIncorrectPrebuildsARCH
  DetectCorrectPrebuildsARCH
  BuildNpmModulesARCH
  CopyBuiltNpmAssetsARCH
  GenerateNodeNativeAssetsListsARCH
end

CopyNodeProjectAssetsFolder-->GenerateNodeProjectAssetsLists
GenerateNodeProjectAssetsLists-->preBuild
CopyNodeProjectAssetsFolder-->ApplyPatchScriptToModules
ApplyPatchScriptToModules-->GenerateNodeProjectAssetsLists

CopyNodeProjectAssetsARCH-->DeleteIncorrectPrebuildsARCH-->DetectCorrectPrebuildsARCH-->BuildNpmModulesARCH-->CopyBuiltNpmAssetsARCH-->GenerateNodeNativeAssetsListsARCH-->preBuild

iOS

graph TB;

ios-copy-nodejs-project.sh
ios-build-native-modules.sh[ios-build-native-modules.sh\n\nIf should rebuild native modules\nDelete incorrect prebuilds\nDetect correct prebuilds\nApply patch script to modules\nBuild npm native modules]
ios-sign-native-modules.sh
ios-remove-framework-simulator-strips.sh

ios-copy-nodejs-project.sh-->ios-build-native-modules.sh-->ios-sign-native-modules.sh-->ios-remove-framework-simulator-strips.sh