AndroidX Migration Plan

matt-oakes commented 5 years ago

Introduction

React Native 0.60 will move from using the Support Library to AndroidX. These are mostly the same library, however the artifacts names and package names have all changed. Google has stopped supporting the support library (ha) and will only be releasing updates to AndroidX.

Details from Google.

With this change, React Native apps will need to begin using AndroidX themselves. They cannot be used side-by-side in one app, so all of the app code and dependency code needs to be using one or the other.

A normal native app can use the Jetifier tool to migrate. This goes through all of their dependencies and changes any references to the support library to AndroidX.

Jetifier will not work for React Native projects as it only works on "packaged artifacts". Almost all React Native dependencies come in the form of a reference to the library source code. These are not migrated by Jetifier.

The Core of It

We are going to have lots of reported issues where developers Android apps will not build because some of their dependencies are using AndroidX and some are using the support library.

The "best" solution is for all libraries to release a version which has migrated to AndroidX and the app developer can then upgrade all of their dependencies at once. This would mean, however, that users who want to get bug fixes for that library would need to migrate their app to AndroidX. This, in turn, means that if just one of their dependencies is not migrated to AndroidX, they are locked into using <=RN 0.59 and cannot use new versions of libraries which have migrated.

Most React Native libraries do not need to use the support library, however, there are many that do and not all are actively maintained.

Discussion points

How can we help developers with this problem?

mikehardy commented 5 years ago

Thanks Matt!

the ideas I have seen so far are:

convert module to AndroidX, then use some tool (bob?) to generate AAR for distribution and run reverse jetifier
maintain module with support libraries and distribute as AAR so consumers may use Jetifier have a postinstall script that does a translation using something like this https://developer.android.com/topic/libraries/support-library/downloads/androidx-class-mapping.csv
maintain separate branches (tagged @AndroidX or @support or similar)

I'm interested in the AAR ideas but I'm unsure of the exact mechanics to generate an AAR, distribute it, and how to document AAR integration into a project.

matt-oakes commented 5 years ago

convert module to AndroidX, then use some tool (bob?) to generate AAR for distribution and run reverse jetifier

Would this require changes to the way libraries are referenced in the build.gradle files? If so, this could be confusing to developers where Gradle is already pretty opaque and hard to understand.

Another idea I was to have a postinstall script which "migrates" dependency source using sed and a CSV mapping file which Google provides.

This could work, but also has the possibility to just break more things than it fixes.

mikehardy commented 5 years ago

I believe this would require different dependency referencing yes, but this is also something that CLI could likely handle in an unlink/link phase, reducing the problem set to consuming projects performing an upgrade (or can CLI run scripts on upgrade?)

If signaled by a semver major with instructions for projects that could not migrate, it might at least work. It's my understanding that we don't have any solution candidates that handle all cases, so even having this idea sussed-out to a "could at least work" status would be a positive.

yeswanth commented 5 years ago

The above solutions might be tough for someone not familiar with Android. I would also like to see a list showing which libraries are compatible, so that users can make a conscious call on whether they can safely upgrade to 0.60 or not. Maintaining this list might be difficult, but it could be a good official approach (not just for this particular use case, but can be useful for others too).

mikehardy commented 5 years ago

For unrelated reasons I just ended up chewing on a library dependency that had updated to AndroidX and it was unusable in a react-native 0.59 context. That library emitted an AAR so it was possible for me to reverse-jetify it (if we had the infrastructure for that to be easy for people) and that would have possibly solved my problem. However I had to fork another library dependency and update it then point package.json to my github fork and that made me think - even if we build an AAR reverse-jetify infrastructure (which I would argue would solve a lot of problems) people that have one foot across AndroidX (by using RN0.60+some reverse jetifier AAR magic) but one foot not across it (by relying on libraries that don't support it and don't provide any workarounds) will not be able to quickly fork+modify libraries that have gone AndroidX because any AAR+reverse-jetifier infra will necessarily be some post-processing that doesn't work in a package.json+git-reference context.

I consider this an edge case, but it is one that affects me so I thought it was worth mentioning. There may be nothing that can be done, some breaking changes are like that.

I'm still thinking AAR distributions and some reverse jetifier magic will help the largest number of people but I haven't proven that out yet as feasible or acceptable

matt-oakes commented 5 years ago

Just to note that this is the sort of issue which will be opened if we don't find a way to make this migration work:

https://github.com/react-native-community/react-native-netinfo/issues/107

mikehardy commented 5 years ago

In the linked issue from react-native-firebase above, a user commented that this allowed them to use react-native-firebase in an AndroidX app, I have not tested it but it might work?

UPDATE: this was tested (see below) and DOES NOT WORK. I wish it did, but it does not. I am leaving it here so we know what doesn't work, but don't waste your time (sadly)

Edit: For anyone else, the workaround is this:
Add gradle.properties into the android folder of each incompatible module, with this content:

android.useAndroidX=false
android.enableJetifier=false

EskelCz commented 5 years ago

@mikehardy it got me through react-native-firebase, react-native-google-signin and react-native-maps errors. For react-native-navigation it didn't work and I'm stuck, thinking about reverting the whole 'upgrade'. Only difference I see is that react-native-navigation has gradle wrapper inside it and disabled android.enableAapt2 because of some building issues.

I'm new to android so this whole thing drives me crazy. Seems like a move to promote flutter on part of google.

mikehardy commented 5 years ago

Shame it didn't work all the way - I wouldn't be so cynical though, AndroidX the layer beneath flutter I believe, and it's one of those painful ecosystem-wide migrations that need to happen once in a while. I'm not aware of anyone on the pure-Android side that has problems with it because in general jetifier works great, using dependencies by source inclusion is extraordinarily rare in the Android world...except react-native :-)

matt-oakes commented 5 years ago

I’m going to look into this as it’s clear this is going to be a pretty big pain point for the 0.60 release. I’ll report back with any updates I have on progress.

matt-oakes commented 5 years ago

@mikehardy Just to note, that workaround does not work. I've just tested on my machine with the NetInfo module and I get the same results as without it.

mikehardy commented 5 years ago

Dang - thanks for checking @matt-oakes - not happy to hear it didn't work but always happy to hear concrete reproducible results - I will edit the comment so as not to steer anyone wrong

rozPierog commented 5 years ago

Hi guys,

the jetifier only works on packaged artifacts

So no way that it gonna change source code of our apps, and React-Native modules/packages are treated in Android as part of a source code, not as libraries. Only way to move RN package to AndroidX is to update imports, and gradle to new androidx.* format. But this will break compatibility with older versions of RN (<0.60) so I would recommend creating new major version of library with AndroidX support and annotate it with Breaking Changes explaining why it wouldn't work on RN <0.60. I was researching jump to AndroidX for the last few month so if you have any questions I'll try to answer them

mikehardy commented 5 years ago

Only way to move RN package to AndroidX is to update imports, and gradle to new androidx.* format. But this will break compatibility with older versions of RN (<0.60)

I was the under the impression compiling libraries pre-distribution into AARs, and then distributing them would allow the jetifier to work while maintaining backwards compatibility from a single (not branched) code line (though it would be a breaking change as people updated how they included the library).

I'm not aware of concrete results showing whether we could migrate a library to AndroidX then compile an AAR and reverse-jetify it and include that in a non-AndroidX/pre-RN0.60 app, but in theory it should work and just needs proving (then if proved, tooling)

My purpose in making this point is to see if there is some way to have a single non-branched library codebase service both sides of the RN0.60/AndroidX divide

rozPierog commented 5 years ago

I was the under the impression compiling libraries pre-distribution into AARs, and then distributing them would allow the jetifier to work while maintaining backwards compatibility from a single (not branched) code line

That… should actually work. Let me test this right now

mikehardy commented 5 years ago

@rozPierog I am really excited to hear results, either way. Here is the inspiration for the "reverse jetify" idea, in case a library moves to AndroidX but wants to produce an artifact consumable in non-AndroidX apps: https://ncorti.com/blog/jetifier-reverse

rozPierog commented 5 years ago

It seems to work! I've tested it with app on RN 0.59 and react-native-netinfo migrated to AndroidX and witf RN 0.60-rc.0 and react-native-netinfo migrated to AndroidX. It was pretty straightforward process. Only downside that I see is that it requires additional tooling, jetifier-standalone, and something to create build.gradle with

configurations.maybeCreate("default")
artifacts.add("default", file('react-native-netinfo.aar'))

but other than that I think it might work.

What I've done:

Open react-native-netinfo in Android Studio
Refactor > Migrate to AndroidX
Gradle sync
./gradlew assemble in react-native-netinfo android project folder
download jetifier-standalone
extract it to convenient place

./jetifier-standalone/bin/jetifier-standalone -r -i react-native-netinfo/android/build/outputs/aar/android.aar -o react-native-netinfo.aar

In a folder where now react-native-netinfo.aar is located create build.gradle with

configurations.maybeCreate("default")
artifacts.add("default", file('react-native-netinfo.aar'))

open setting.gradle of your RN app and add

include ':react-native-netinfo'
project(':react-native-netinfo').projectDir =
    new File(rootProject.projectDir, '../../../RNN')

where '../../../RNN is relative path to your .arr + build.gradle folder

In RN apps app/build.gradle add

dependencies {
implementation project(':react-native-netinfo')
...
}

Done!

It seems that it can be fully automated with simple script

Would love if someone repeated those steps and reported if it's working for his project as well

matt-oakes commented 5 years ago

@rozPierog I experimented with that as well. If a library included a file like react-native-netinfo.aar in the android folder of their NPM package, would the linking code need to be updated at all to pick it up? From your message above steps 9 and 10 suggest it will work with the same code as before but will pick up the car if available. Is that correct?

If so, then we can just ask that libraries add the code to generate a reverse jetified AAR file in their NPM package to support both AndroidX and the support library. This won't be needed for all libraries as only some currently make use of the support library.

rozPierog commented 5 years ago

@matt-oakes I have a feeling that everything is dependent on build.gradle that is inside projectDir if we move .aar to root of android project folder there will be conflict of build.gradles, maybe, maybe, adding those 2 lines to main build.gradle would work, but I don't really know. It seems that the best option is to create something like library-name/android/aar where you can store .aar with build.gradle. But that's just a speculation. I cannot check if thats true right now.

mikehardy commented 5 years ago

Sounds like there are some subtleties to work out, but is it possible to summarize this as "if a library, using AndroidX or support libraries, builds an AAR in the right way, a react-native app using either AndroidX or support libraries can use it". i.e., this can work for all cases?

If so, this is a tremendous result and would motivate the tooling if any was needed. Excellent @rozPierog and @matt-oakes

step 5 and 6 - "download jetifier-standalone and extract it etc"

I just did this to make it easier, I hope? https://www.npmjs.com/package/jetifier (after npm i jetifier, npx jetifier-standalone should be available). So hopefully playing around with this is easier for people

I haven't tried the steps 7-10 yet but I will say that as a library consumer, if I convert my project to AndroidX I am most likely already following a set of instructions. If there was something I needed to alter with my existing dependencies I would not be surprised at all. If there was a script to do it, better of course, but not vital so long as it is possible

As a library maintainer, if there was a package I could use that automated AAR construction (optionally with de-jetification, or maybe always both with standard suffixes?) that verified I had the proper stuff I would be thrilled. Seems like bob should do it (or similar)

rozPierog commented 5 years ago

Thanks for uploading jetifier to npm! I've created pure gradle script that automates whole thing can't really tell if this will be good approach or not.

mikehardy commented 5 years ago

@rozPierog ! working code is always the winning approach :-). And that code looks good to me

Might be best to package this up as well, maybe we could add it to the jetifier thing I just put together (I can give you all the github and npm perms there - update: I just went and sent you the invites, I hereby trust you haha) and we can document that people should include the reference and just call the method in their gradle scripts vs copy?

That would let the community collaborate on the implementation. This could become important as the blog post on reverse jetifier I linked mentions that sometimes the reverse process requires disambiguation, so maybe there will need to be some config mechanism or something. Or maybe not, but if it was distributed as part of an npm package then everyone could benefit.

mikehardy commented 5 years ago

I would just change the jetifier package version scheme to use real semver so we had space to work, vs my current / first-try style where it's just matching upstream now

rozPierog commented 5 years ago

Thanks for the invites! Sadly that's all the time I have today for open source work so this is my closing statement:

Some good news some bad news:

Good: I've just tested de-jetified aar on clean RN 0.60 project and it seems to be working (previous test was on updated project) update: or not? I have some errors regarding AppState, don't know if this is related
Bad: Shiny and new Autolinker need to be modified to accommodate libraries with aars. Therefore we need to standardize localization of aar within android project folder.

I feel like we should promote this discussion on https://github.com/react-native-community/releases/issues/116 so that more people would help/look at this problem/solution. I'll have very limited time in the next two days so I would appreciate if someone would help with this (move gradle script (or rewrite it to something else, like pure shell or js) to jetifier repo, made documentation on how and when to use it)

mikehardy commented 5 years ago

anyone that wants to collaborate - but no pressure, and I think we have time - a couple weeks? - just let me know and I can send invites to the jetifier repo. Or do your own package or whatever works. I also have limited time, but I'm obviously interested. I'll cross-link to the releases issue and I know @kelset is busy herding cats there already, and is here. Tremendous work @rozPierog

sibelius commented 5 years ago

this is causing some problem for us here https://github.com/react-native-community/react-native-camera/pull/2068

I think we should move forward when v60 lands

matt-oakes commented 5 years ago

Thanks for your work @rozPierog! I am planning to look at making some changes to the autolinking to make this work in the next couple of days.

@mikehardy I'm not sure if I'll need it, but feel free to send over invites for the Jetifier project and I'll help if I can.

mikehardy commented 5 years ago

@matt-oakes done! may not be much to do or may be depending on where we want to take it, but either way we've increased bus factor by 1 now, so cool. Feel free to do whatever in general there. If you're in autolinking it'll probably be best to put stuff in CLI, but if it belongs near the jetifier have at it

matt-oakes commented 5 years ago

I have opened a couple of new PRs which make some progress towards making the developer experience of migrating to AndroidX a little nicer.

I have created a new "target" in the bob build tool which supports using AAR files and also supports running a reverse jetify process on it to create a version which uses the support library. The PR for this is available here:

https://github.com/react-native-community/bob/pull/4

I have temporarily published this as @mattoakes/bob to allow me to open a PR for NetInfo which makes use of this new target with as little faffing as possible. That PR is here:

https://github.com/react-native-community/react-native-netinfo/pull/115

With this PR the library is migrated to AndroidX, however, the NPM package now includes two AAR files inside lib/aar and lib/aar/support. If the user is using React Native >= 0.60 or they are using another library which uses AndroidX, then they can use autolinking as-is or the previous manual linking method still works.

If, however, they are using an older version of React Native and want to continue using the support library version, you just need to update the reference to the module in their android/settings.gradle file like this:

include ':react-native-community-netinfo'
- project(':react-native-community-netinfo').projectDir = new File(rootProject.projectDir, '../node_modules/@react-native-community/netinfo/android')
+ project(':react-native-community-netinfo').projectDir = new File(rootProject.projectDir, '../node_modules/@react-native-community/netinfo/lib/aar/support')

To make testing this easier, I have release the NetInfo package under the next tag to it can be installed with one of these commands:

yarn add @react-native-community/netinfo@next

npm install --save @react-native-community/netinfo@next

Let me know how all of this sounds to you @mikehardy and @rozPierog.

mikehardy commented 5 years ago

This was exactly what I was hoping, but wasn't sure was possible. Both PRs look like fine work on visual review. How are you testing this? Do you have an app successfully working on RN0.60 yet? Just curious as I had some fails while playing with it so I'm not sure I can be a set of testing eyes. This looks like it fixes things for active library maintainers. That's huge.

Thinking ahead for users:

TL;DR: If I am right about the default case - user wants to convert to AndroidX and use RN0.60, then we'll need something that "auto fixes" un-converted libraries. And we may increase developer pain a lot if RN can't provide an AAR itself for users that can't convert

Thinking through the 2x2 of support vs AndroidX and RN<0.60 RN>=0.60 now, to arrive at that TLDR

1 - app is on support + RN0.59, then as libraries convert to AndroidX, I'll update my settings.gradle to support/android.aar. Not bad. Maybe even paper over it with CLI or postinstall

2 - app is on support + RN0.60 - not possible unless RN0.60 provides a support AAR? Is that possible? If so then it reduces to 1 above. Can happen if project experiences AndroidX bug or API change for an app while RN0.60 tests okay. I have sympathy for these users, well-intentioned but left behind. Investigating an RN support AAR might keep them with us

3 - app on AndroidX + RN0.59 - requires react-native AAR similar to 2 above, as a RN0.59 patch release, plus all dependencies to convert to AndroidX or some sort of CLI tool that took any library that didn't provide an AAR and auto-did it postinstall. This quadrant doesn't seem likely to me but the following case is similar and interesting:

4 - app on AndroidX + RN0.60 - our goal - but then I think we have a problem. Even with timely library maintainers, AndroidX conversion can't be coordinated. That makes me think pursuing an "auto-AAR" constructor in CLI for dependencies that aren't AndroidX in an app that is important.

I'm just trying to think through it, please tear it up and let me know where I'm wrong!

rozPierog commented 5 years ago

2 - app is on support + RN0.60

This can be a pain if you are using some old unsupported react native library, and would like to be on the latest RN version. We can ease those people be providing easy to do steps on how to fork and create aar.

@matt-oakes great work! I have similar questions to Mike. Have you tried this on Rn 0.59 app and 0.60 app? Would love to hear that it's working

mikehardy commented 5 years ago

This can be a pain if you are using some old unsupported react native library, and would like to be on the latest RN version. We can ease those people be providing easy to do steps on how to fork and create aar.

In my experience supporting a few repos, this will be a huge hurdle, so I still have fingers crossed there is some CLI magic available (or some package that could provide a post-install hook like patch-package) where you could generate an AAR for any package that didn't have one

matt-oakes commented 5 years ago

In theory you can run gradle assemble on any RB library and it will give you an AAR. Maybe we can change the auto linking code to do this for libraries that don't have an AAR included?

One thing I'm not clear on is if there is any limitations with using AARs. Do you have any insights into that @mikehardy

mikehardy commented 5 years ago

The only limitation I can think of is that you can't patch-package an included java file.

However, if this AAR construction became standard I would imagine patch-package itself would extend to call the gradle assemble (esp if it were in some standard part of the machinery like in CLI so AAR construction didn't skew w/copy-paste implementations)

Other than that I'm not aware of anything. Maybe the MultiDex case is worth testing but I already have some AARs in my project, I have to MultiDex (for API<21 support) and it's fine. AARs are a fundamentally supported Android thing, should just work as long as we generate them correctly

mikehardy commented 5 years ago

In separate discussion, it was mentioned that some libraries have project-dependent build-time steps / do fancy things in gradle, which is a fair point.

If I'm correct in summarizing the current AAR testing as "we know the user will need jetifier or reverse jetifier even on the source code of modules, so let's do it for them regardless of other settings", and we were testing this as static production pre-distribution for the modules with linking for apps, then maybe we could go one step further and implement the jetify/de-jetify at build-time with a simple gradle hook to a package or CLI or something

Investigation showed that there are a bunch of implementations of jetify (one in Python even, so maybe quickly javascript portable) so implementing it should be manual labor but not research, and there is apparently a way to detect if you are using support libraries or not like so https://github.com/plnice/can-i-drop-jetifier

We've shown that jetifier in general can do it as a process, this would just alter when it happens, and then hopefully handle all native integrations even with fancy builds

Any reason it wouldn't work?

mikehardy commented 5 years ago

Here is quick shell that uses sed to recursively edit inside your project to implement the CSV map published by google: https://gist.github.com/janicduplessis/df9b5e3c2b2e23bbae713255bdb99f3c#file-androidx-sh

Is it possible to address the point above where modules can do things at build time with a script like this run before the gradle invocation? If so, then this approach could work?

Then "polish" could be that it was cross-platform, was packaged as a CLI plugin, and downloaded the CSV itself if needed?

mikehardy commented 5 years ago

Maybe worth nothing that Play Services and Firebase (to big chunks of ecosystem) just released major/breaking versions, using AndroidX https://developers.google.com/android/guides/releases

benoitdion commented 5 years ago

Do we have any insight on when 0.60 will get released? While a fully automated approach would be nice, with the time we have left we might be better off trying to get most popular RN libs androidx ready (focus on 4 https://github.com/react-native-community/discussions-and-proposals/issues/129#issuecomment-498921324).

For the less maintained libs, react-native-community could maintain a list of patch-package patches.

mikehardy commented 5 years ago

week or two I think? and in the meantime the "minimum viable product" is a link to that script, such that you can install RN0.60, use that script, and things will actually work..

matt-oakes commented 5 years ago

...in the meantime the "minimum viable product" is a link to that script, such that you can install RN0.60, use that script, and things will actually work..

Hopefully work 😉

mikehardy commented 5 years ago

Okay you laughing people, it works! I think? Please test!

https://github.com/mikehardy/jetifier#usage-for-source-files

The only laughable thing now is performance, but proof-of-idea first

If this tests out for others (it worked for me) then our status from the above table is:

1 + 2: working with reverse jetifier if libraries provide an AAR. Maybe also workable if people take the npx jetify command I just implemented and do implement the reverse 3 + 4: handled with this npx jetify script

This whole thing might work

mikehardy commented 5 years ago

Has anyone else tested jetify and has results yet? I had flawless victory and I have started recommending it but would love feedback. It is still thick and fast with bugs in various repos right since Google released AndroidX versions of their APIs yesterday - it's time to package this up and have canonical documentation etc

laurent22 commented 5 years ago

Has anyone else tested jetify and has results yet?

I did try it and it didn't solve my issues. Modules that were not compatible with AndroidX were still not. I would recommend to avoid any AndroidX module for now.

mikehardy commented 5 years ago

@laurent22 can you elaborate? Which modules did you have linked in, I would like to test them. I did an app with react-native-device-info, react-native-share, rn-fetch-blob, react-native-geolocation-android and react-native-firebase (among others) and all of those in the list use the support libraries. All of them converted fine. So that was 5 for 5 and made me think it could work. If there is a counter-example I really want to investigate. Thanks!

laurent22 commented 5 years ago

@mikehardy, I'm a bit stuck on the issue and it's possible I'm missing something. The list of packages is there. Without jetify, I'm getting the error I described here, and with it, I'm getting errors on @react-native-community/slider for example, which cannot find the right libraries.

Now that I think about it I didn't try removing the slider module to see if it would make a difference as I assumed that old (rarely updated) modules like rn-fetch-blob wouldn't work, but maybe I need to give it another try.

mikehardy commented 5 years ago

@laurent22 I attempted to test as many of your native dependencies as possible and it builds for me. I did not attempt to exercise them (though I do so in my production app for many of the same dependencies). You can try cloning this repo and running the script to see https://github.com/mikehardy/rn-androidx-demo ?

If you want to exercise them in some simple ways just modify the App.js in the repo root to do so - it will be copied in before it does run-android. Feel free to PR any extra stuff you add in to demonstrate success or failure

laurent22 commented 5 years ago

Thanks @mikehardy, the test project worked, and running npm i jetifier && npx jetify on my project made it work too. I've tested a few things and as far as I can tell there's no issue.

I'm curious though, why is it necessary to run jetifier when the project doesn't (as far as I understand) use AndroidX? In particular RN has not been upgraded to AndroidX and none of the modules either. Or if they have, is there any way to find out which one? Ideally I'd just downgrade those modules for now if that's possible.

I think during the transition it would be useful to tell developers how to pinpoint these modules, so that they can choose if they either downgrade or use jefity.

mikehardy commented 5 years ago

Yesssss - very happy that worked. Except I am confused, as if your project was not converted then you don't need this, you need the second part of your question, what's using androidx:

cd android && ./gradlew :app:dependencies will give you a list of your deps, pipe it through grep androidx and you can find them

I did find one case so far where the library would not auto-transform but it was fixable, a totally busted ancient .aar file that jetifier (the official tool) barfed on. So it's like 95% success rate now, and 100% after fixing 2nd-order dependency bug. No problems with the tool + idea so far that I can see

NuclearKev commented 5 years ago

@mikehardy jetify worked on my project!

mikehardy commented 5 years ago

Excellent @NuclearKev ! thanks for the report. We just got one failure report which is significant, it doesn't work on windows (which makes sense, I built jetify on mac and linux). If anyone with windows skills could either provide instructions on how to install things (it needs find, sed and sort) maybe via chocolatey? or cygwin? I have no experience - that would help

react-native-community / discussions-and-proposals