Improving 3rd Party Module Compatibility

[TheSavior]

I've heard from people that the compatibility between RN and 3rd party modules is a big pain point with React Native. I'd like to start a conversation about that and see if there is something we can do to improve the situation.

Background

Here is an example to demonstrate my understanding of the issue:

React Native 0.60 introduced a breaking change to AndroidX. This meant that 3rd party libraries needed to make native changes to support AndroidX and thus 0.60. These were breaking changes because AndroidX modules weren’t compatible with 0.59.

So module authors have to make a new release of their module that only supports 0.60. This is required for anyone to be able to upgrade, which also means that most people aren’t using 0.60 yet. That means that when people on 0.59 install that module for the first time, they will get a version that doesn’t work on their installation and they need to go track down why and realize they need to install a different version.

Question: I know this was a big issue for 0.60, but I feel like I’ve heard this being a similar issue for other releases as well. Is it?

If this is a big problem for other releases than just 0.60, is there something we can do to make this better?

One potential proposal

@acoates-ms proposed maybe having react-native add instead of yarn add that would check the repo of the package, in some root config file or something that would let the package author specify the supported compatibility of their library with different versions of RN.

I mostly just want to get the conversation started on 3rd party module compatibility. It is possible that I’m not capturing the extent of the problem or the different things that need to be solved here, so hopefully people can chime in with additional thoughts.

[kelset]

I think that what @acoates-ms suggests is a good step in helping people being aware of packages/3rd party libs not being "new-version-proofed". And the approach of passing via a new CLI command may be the best way. But, for example, it wouldn't help with the issue you mention above of upgrading (since the pack was already added).

I fear that this is not something that can fully solved (for everyone/for all libs) - for the example you mention, 0.60 and AndroidX, we did what we could as "React Native Community" IMHO: we opened an issue (#129) to discuss how to handle it, tried to raise awareness through social media, and @mikehardy & others prepared a tool to "auto-fix" the packages that couldn't be compatible (jetifier).

But the core issue is, basically, that "the third party lib you use is not ready for you when you want to upgrade / when you want to use the latest version of React Native".

So the question should be, why is that? From my experience, it basically boils down to the amount of overhead/work that provide support for latest requires on one hand, and not knowing that the lib actually requires changes on the other.

For the first part (doing the actual work) there isn't much we can do to have every package work with every version: it's up to the maintainer to still maintain said lib, or to at least merge a PR upgrading (which, as for all the things OSS, it's not a given). The best that can be done IMHO on this is, again, raise awareness and provide a place to discuss a "plan" so that then maintainers can use that as a source of inspiration (or even for small changes a copy/paste of code). For a few precise scenarios there is also the jetifier "post-install-script" option, but I'm not sure how easy it would be to have that in place for every breaking change for every RN version.

The second part (being aware of changes incoming/my lib is not ready for RN ex. 0.62) is more actionable, in a way: I discussed a while ago with @bartolkaruza and we think that if there was an easy "RNTester-like" folder that you can drop in into your lib, that uses master has target + with a GitHub Action you could basically test that your lib + master is still working as expected, or not.

Anyway, I think that this is an issue that affects every OSS ecosystem and it would be great to see the RN world be able to come up with some viable solutions 🤗 so thanks for opening this!

[mikehardy]

Interesting - just got tagged in here. It's a divergent ecosystem run by volunteers maintaining libs so the libs will never be in sync with the core releases as a binary / 100% state. That is intractable.

However, this immediately made me think of the way react handled the deprecation of componentWillMount etc by releasing a codemod at the same time.

Interestingly they used (and linked to) an article by @cpojer which I thought was fun in a small-world way: https://medium.com/@cpojer/effective-javascript-codemods-5a6686bb46fb

So perhaps if someone is motivated for this, whatever library things must be changed in breaking ways to handle a new release can be accompanied by a codemod in the blog post, with encouragement for users to fork the libs they need, apply the codemod and PR the result (while in the meantime using patch-package to move forward). That's my basic workflow except without the automated codemod, anyway.

[mikehardy]

As a follow-on, in general I find the proliferation of config files to be painful, it's a tremendous cognitive load for me to know where the config file for every little ecosystem component is and whether it's JSON, JS, or whatever, and what stage of their config file deprecation lifecycle they are in (rnpm? in the package.json? react-native-config.json? :man_shrugging: ) so I have developed a strong preference for pushing signal through the existing channels if possible.

To me that means package.json should have peerDependencies set correctly on modules for the react-native versions they support. Should obviate need for react-native add and some other manifest file with metadata etc.

Final nit - for AndroidX at least in the jetifier world, the AndroidX transition was backwards and forwards compatible so it resulted in major version bumps for modules but did not actually strand people. That was a major feature goal for the tool, and if I can connect it with the idea of codemods, if they can be backwards and forwards-compatible and maybe dynamic (for most cases, where reasonable, etc) then the entire upgrade process becomes a lot less painful in normal scenarios.

That was nice for AndroidX but I don't think there was any viable solution for the switch pods, so sometimes the ecosystem just breaks and we all move on across the breaking change

[brentvatne]

@acoates-ms proposed maybe having react-native add instead of yarn add that would check the repo of the package, in some root config file or something that would let the package author specify the supported compatibility of their library with different versions of RN.

we do this with expo currently, you can run expo install and it will install a compatible version of supported native packages as determined by a simple json file. it'd be great if we could use react-native-cli for this instead so we just have one set of instructions across the ecosystem, but we'd need it to work with expo SDK versions and not just a given react-native version.

my vision for native.directory (which is now going to be collapsed into react-native-website) was that we could use it for storing version compat information in addition to all sorts of other data about the packages, and this would then become a source of truth for a command like expo install

[brentvatne]

To me that means package.json should have peerDependencies set correctly on modules for the react-native versions they support. Should obviate need for react-native add and some other manifest file with metadata etc.

this is problematic for a few reasons: 1) it forces library authors to update their peer dependencies on every release even if there were no changes. in the ideal world this is nice and acknowledges that your library is compatible, but realistically what you end up with is endless peer dependency warnings. 2) npm has weird rules for peer dependencies and prereleases that lead to again more logspam for warnings. 3) people don't pay attention to peer dependency warnings anymore because there is so much noise and so little of it is actionable on their part, and also 1 & 2 will contribute to that continuing to be the case.

[brentvatne]

It's a divergent ecosystem run by volunteers maintaining libs so the libs will never be in sync with the core releases as a binary / 100% state. That is intractable.

i agree it's intractable for a divergent ecosystem run by volunteers. i think one possible alternative to this is for the ecosystem to depend more on modules from the expo sdk. expo is a company and part of what we do is build an extended standard library for react-native. we update all libraries in the expo sdk to support the react-native version that we support, and although we don't always track the latest version of react-native we are open to prs to make them compatible.

generally speaking i also typically recommend that people don't try to track the latest react-native version in their apps immediately on release. third library compatibility is one big issue, but each release also comes with its own new set of bugs that are slowly resolved over a month or two in ~10ish patch releases.

maybe part of better third party library compatibility is to keep the new major version releases in beta or some other state that discourages widespread adoption until more of these issues have been resolved and more libraries have been updated.

[mikehardy]

@brentvatne a fair point on peerDependencies warnings - I'm just so tired of YASJF (Yet Another Simple JSON File) :sweat_smile: - but if it's the only clean solution, then that's it.

The problem I have with your proposal on depending on more centralization is that the tracking speed is exactly a problem. If I understand correctly for instance right now react-native-firebase-current-release and Expo-current-release don't get along because of an irritating AdMob semver break under the covers, so I would need to use older versions of react-native-firebase and google ios pods, but...they just finished removing UIWebView references in google ios pods 6.8.1 so I need to get current for the iOS 13 release tomorrow (if our guesses are correct) and so the otherwise-intractable army of uncoordinated volunteers is messy but high velocity. In the end I prefer it personally but I understand it's a matter of taste.

[brentvatne]

If I understand correctly for instance right now react-native-firebase-current-release and Expo-current-release don't get along because of an irritating AdMob semver break under the covers, so I would need to use older versions of react-native-firebase and google ios pods

this applies to ExpoKit which we are in the process of deprecating because it's too monolithic and rigid, not to the process described in this post which is basically just the same as installing any given library in a react-native project.

and so the otherwise-intractable army of uncoordinated volunteers is messy but high velocity. In the end I prefer it personally but I understand it's a matter of taste.

agree to disagree. i think the benefits of this apparent velocity are largely if not entirely lost on the effort that it takes to compensate for the messiness. but indeed to each their own.

[mikehardy]

I think a middle ground - in between the ExpoKit and the full-on messiness happening right now would be welcome actually. I'm keenly aware of the cost of compensation. Just be clear that I'm not anti just that the ExpoKit rigidity has actually bitten me in a support capacity so I couldn't support that, but if there is a less rigid future that is also less messy I'll probably be all over it.

[edited to add - you sent me some good unimodule info recently and I am interested but have not looked yet, so I'm actually also excited about it as a possible middle ground, but currently ignorant]

[acoates-ms]

I was thinking of adding something to package.json:

"react-native": {
  "matchingVersions": {
    "android": {
      "^0.61.0": "^1.2",
      "^0.60.0": "^1.1",
    },
    "ios": {
      "^0.61.0": "^1.2",
      "^0.60.0": "^1",
    },
    "windows": {
      "^0.62.0": "^1.2"
      "^0.61.0": "^1.2",
      "^0.60.0": "^1",
    }
  }
}

This should provide enough information not only to satisfy what react-native add would need. But also provide information to react-native upgrade which could then warn the user about which packages might not work for the platforms the app is using. I put this in package.json because it allows the value to be easily queried without downloading the package using yarn info.

I imagine the workflow on react-native add would either be to install the matching version that satisfies all the platforms the user needs. Or if there is no such version to provide interactive options such as:

No version of Foo is marked as supporting all your platforms.
Please select from these available options:
latest (4.0): - Marked as supporting iOS for your version.
3.2: - Marked as supporting Windows and iOS but not Android
3.1: - Marked as supporting Android and iOS but not Windows

Similarly react-native upgrade would be able to provide information about what packages you are using that might not work after upgrade:

Determining application compatibility with react-native@0.64.0....
The following react-native extensions may not work with the latest version of react-native.
react-native-foo [Max supported version 0.63]
react-native-bar [Max supported version 0.63]
react-native-something  [Max supported version 0.60]
Are you sure you wish to continue? [yN]: 

[brentvatne]

@acoates-ms - that looks nice, i like the usage of yarn info as well here.

an overall concern i have with this approach is the same as 1) from my peer dependencies comment above:

it forces library authors to update their peer dependencies on every release even if there were no changes. in the ideal world this is nice and acknowledges that your library is compatible, but realistically what you end up with is endless peer dependency warnings.

if i am the author of react-native-linear-gradient and nothing changes from 0.60 -> 0.70, i still need to update my package.json and republish for each release. the alternative is that i let the semver matcher be more flexible and the compatibility information is less useful because it may not be accurate.

fwiw i don't think this is a bad idea i just want to lay out some of the limitations.

[brentvatne]

The second part (being aware of changes incoming/my lib is not ready for RN ex. 0.62) is more actionable, in a way: I discussed a while ago with @bartolkaruza and we think that if there was an easy "RNTester-like" folder that you can drop in into your lib, that uses master has target + with a GitHub Action you could basically test that your lib + master is still working as expected, or not.

I like this idea. I think that possibly in addition to this we could have tests for the most popular third party libraries built into core itself, so we know immediately when a particular commit breaks a crucial community library (eg: react-native-gesture-handler) and can address it before doing a release.

[henrymoulton]

Just a question for expo'ers/facebookers, based on what @brentvatne has said briefly about Unimodules.

• What's the roadmap for Unimodules?
• How does it fit in with Expo's strategy for professionals
• Will it be likely that react-native-community members deprecate or merge common mobile "standard library" repositories in favour of one
• Facebook still recommends expo-cli for beginners, will it weigh in on recommending the adoption of Unimodules on it's Getting Started page

Context:

Before unimodules, I believe some of the Expo modules were extracted into their own react-native-community repository folders, I think react-native-camera is an example of this.

But now with unimodules, which I believe Expo / @ide is planning on turning into a specification (https://github.com/unimodules/unimodules.org) , the community doesn't have to be tied to Expo to make use of unimodules.

I believe that for most people, they will prioritise stability and performance of cross-platform mobile APIs over whether it's a 3rd party or Expo module. However developers can still be scared of lock-in which I believe is the biggest thing holding back React Native developers adopting Expo (source: talking to other developers at the React Native London meetup).

Example: react-native-touch-id is a 3rd party module that is no longer actively developed. expo-local-authentication is something we're looking at adopting through using Unimodules. Early results are promising, @pvinis has also mentioned he's had good results.

3rd party module compatibility is a pain, we're holding back on upgrading to 0.60/0.61 because we have to ensure that our 3rd party dependencies have done a stable release for compatibility with 0.60 or look for an alternative library, or write our own.

[ide]

@henrymoulton To answer your questions, the plan for Unimodules is to eventually migrate the underlying infrastructure to use Turbomodules when they are ready. Unimodules provide several things such as a shared native module registry for when modules need to depend on each other, and tooling for making modules work universally across Android, iOS, web, and other future platforms. We also want the performance overhead of Unimodules to be near-zero especially if you are not using certain features of them ("pay only for what you use").

Anyone can write a Unimodule and over time I wouldn't be surprised if most Unimodules were authored by Expo developers not on the Expo team. Currently there is very little documentation on how to write a Unimodule and we're also refining things (ex: Turbomodules, some base data types), so nearly all the Unimodules today are written by the Expo team. We've dubbed this set of Unimodules the "Foundation Unimodules" just to give them a name.

Unimodules also generally will work with apps using bare React Native. Because Unimodules run on Expo, which includes Android, iOS, and web support -- and perhaps more in the future -- the Android and iOS implementations of a Unimodule will work with React Native as well.

2\ Regarding professionals, we've found that a very large number of apps' needs are mostly met by the base APIs that Expo provides (most of the Foundation Unimodules) in the simplified, managed client, but that they also need just one or two custom modules. We've been working on a way for developers to write custom modules (Unimodules) with a way to compile apps with those modules on their own computers (bare workflow). Longer term, we'll also look at offering services that can compile the apps for you (think AWS for compiling and hosting universal apps).

We have also been investing in test infrastructure and libraries for more comprehensive, universal testing. The JS tests for the Foundation Unimodules run for the platforms that Expo currently supports (Android, iOS, web, SSR): https://blog.expo.io/testing-universal-react-native-apps-with-jest-and-expo-113b4bf9cc44. We're also improving our end-to-end test infrastructure with Detox. There's a lot more work to be done here and these are the most recent steps we've taken to make Expo run universally and stay working universally.

3\ Whether a maintainer of another library would like to continue maintaining that library or suggest that developers use a library authored by Expo is up to them. I imagine in some cases maintainers will also have their own needs (ex: the maintainer of an audio module could be working on a podcast app that needs a very specific audio API that might not make sense for other audio libraries to spend time maintaining), and in other cases the right decision for them may be to spend time elsewhere and ramp down maintenance for their module.

4\ I can't speak to Facebook's choices directly but in general we are working to make Unimodules use Turbomodules and not introduce performance overhead especially for parts you don't use. Unimodules are not an alternative specification to Turbomodules -- writing a Turbomodule will be part of writing a Unimodule. The other parts that Unimodules introduce are universal compatibility (web implementation) and the ability to depend on other modules' native code in a way that works for an open-source ecosystem.

Hope that covers what you were looking to learn, at least from the perspective I can give.

[henrymoulton]

Thanks so much for the in-depth answer @ide!