Prebuilding react-native pods

andreialecu commented 4 years ago

I've been trying to see if this works with react-native, and one of the first issues I ran into is related to the Podfilebeing moved to the Frameworks directory.

The standard Podfile in react-native looks like:

platform :ios, '10.0'

inhibit_all_warnings!

require_relative '../node_modules/react-native/scripts/react_native_pods'
require_relative '../node_modules/@react-native-community/cli-platform-ios/native_modules'

target 'myproject' do

All paths are wrong, and because of the intermediary directory, they would need to be resolved to '../../node_modules instead.

This seems an easy enough change to do manually, but unfortunately it is not enough. Other pods within node_modules are used by react native's auto-linking system and the paths there are hardcoded and cannot be changed.

I wonder if there is any solution for this?

tcamin commented 4 years ago

Could you provide an example project that I could play with?

andreialecu commented 4 years ago

@tcamin sure! I think you could use this one to test: https://github.com/DylanVann/react-native-fast-image https://github.com/DylanVann/react-native-fast-image/tree/master/ReactNativeFastImageExample

Make sure you have a recent version of node installed. Then run yarn install in the ...Example directory. The XCode project files are in the ios directory. The podfile is at https://github.com/DylanVann/react-native-fast-image/blob/master/ReactNativeFastImageExample/ios/Podfile

If you cd ios && pod install then run the project in xcode it should work.

Now if you try to use PodBuilder on it, you will run into the issue with paths I originally described.

andreialecu commented 4 years ago

By the way, there is currently no information on being able to pre-build pods for react native projects anywhere that I could find online. I've been looking at cocoapods-binary as well but ran into issues with it as well. What drew me to this project was the json config in which you could exclude certain pods.

If we'd manage to get this working, I think the project could see incredible notoriety :)

React Native xcode projects are generally mainly pods. The app logic is separate, in JavaScript/TypeScript, and doesn't contribute much to build time.

Compiling all the pods takes anywhere from 5-10 minutes and this is especially costly and inefficient on CI systems, considering that the pods almost never change (unless you upgrade react-native or some dependency).

I assume you do not use react native yourself, so if you need any further information, let me know.

andreialecu commented 4 years ago

🚀 I noticed some commits referencing React Native in the v2 branch: https://github.com/Subito-it/PodBuilder/commit/67516d1fad7162e0dca842c59b73ba25d0a28b85

If I can help in testing it, let me know. If this works it would really be a killer feature for React Native builds.

For example, one of the projects I have CI set-up for has 3 steps:

build for E2E testing - takes 10-12 minutes - mostly Pod compilation
run E2E testing - around 8 minutes
build for release - takes 10-12 minutes - again, Pod compilation

Having the pods prebuilt should slash the times for step 1 and 3 to probably 5 minutes total. For a total run-time of less than half. Also good for the environment, less wasted CPU cycles. 😆

andreialecu commented 4 years ago

One thing to be aware of is that the Podfile recommendation has changed in the latest RN version, and the example project I linked above isn't updated.

Search for Podfile: https://react-native-community.github.io/upgrade-helper/?from=0.62.2&to=0.63.2

They mainly just moved here: https://github.com/facebook/react-native/blob/master/scripts/react_native_pods.rb

tcamin commented 4 years ago

Is there any updated example project you can point me to? To be fair the build chain of the RN project is pretty convoluted so I'm unsure if this is actually feasible with a reasonable number of changes.

andreialecu commented 4 years ago

I have set up a barebones RN project which also has a dependency on some third party native pods (via the same react-native-fast-image as above):

https://github.com/andreialecu/rn63-newproj

After cloning, run yarn install then cd ios && pod install && cd .., and yarn ios to run it on an emulator (or use xcode to open the ios directory and run from there.

PS: One workaround to make the relative paths work and keep the same directory structure would be to add a symlink from proj/ios/node_modules -> proj/node_modules - but there are still some errors in the ruby script with this approach.

andreialecu commented 4 years ago

One note about the RN project above:

https://github.com/andreialecu/rn63-newproj/blob/master/ios/Podfile#L16-L24

Flipper may not work with use_frameworks! or generate other problems by itself while compiling. So it is safe to comment it out from the Podfile if it creates too many problems, or cannot be made to work. Not being able to use Flipper would be an acceptable trade-off, RN has other powerful debugging options available.

sunnylqm commented 4 years ago

I am also very interested in prebuilt pods for rn, not only because it's slow but more importantly, it's extremely difficult to download pod sources in China. Hope this could be a savior 🙏

tcamin commented 4 years ago

I've been giving this a look in my spare time but unfortunately supporting rn goes beyond fixing relative paths and requires a series of non trivial changes mainly because by default AFAIU the Podfile does not include the use_framework! entry which makes things tricky when dealing with Swift pods.

andreialecu commented 4 years ago

@tcamin it's fine to suggest using use_frameworks! if there are no other side effects. The Podfile itself is user modifiable.

andreialecu commented 4 years ago

The default Podfile even mentions that you should disable Flipper if you need to use frameworks. See:

https://github.com/andreialecu/rn63-newproj/blob/master/ios/Podfile#L16-L24

tcamin commented 4 years ago

You might give the latest beta a try then, let me know how it goes.

sunnylqm commented 4 years ago

tried beta.25 and got an error like this:

Found multiple xcworkspaces:
........../node_modules/@react-native-community/geolocation/ios/RNCGeolocation.xcworkspace
........../ios/superlab.xcworkspace
/Library/Ruby/Gems/2.6.0/gems/pod-builder-2.0.0.beta.25/lib/pod_builder/core.rb:152:in `find_xcodeworkspace'
    /Library/Ruby/Gems/2.6.0/gems/pod-builder-2.0.0.beta.25/lib/pod_builder/core.rb:159:in `prepare_basepath'
    /Library/Ruby/Gems/2.6.0/gems/pod-builder-2.0.0.beta.25/lib/pod_builder/command/build_all.rb:8:in `call'
    /Library/Ruby/Gems/2.6.0/gems/pod-builder-2.0.0.beta.25/exe/pod_builder:381:in `block in parse_commandline'
    /Library/Ruby/Gems/2.6.0/gems/pod-builder-2.0.0.beta.25/exe/pod_builder:380:in `each'
    /Library/Ruby/Gems/2.6.0/gems/pod-builder-2.0.0.beta.25/exe/pod_builder:380:in `parse_commandline'
    /Library/Ruby/Gems/2.6.0/gems/pod-builder-2.0.0.beta.25/exe/pod_builder:396:in `<top (required)>'
    /usr/local/bin/pod_builder:23:in `load'
    /usr/local/bin/pod_builder:23:in `<main>'

Command failed!

tcamin commented 3 years ago

Beta 34 brings initial support to prebuilding vanilla react-native projects as the one you provided (including Flipper support).

git clone https://github.com/andreialecu/rn63-newproj.git
cd rn63-newproj
yarn install
pod_builder init
pod_builder build_all

Finally you'll need to add export CONFIGURATION=Release in the Bundle React Native code and images build phase as shown below

Since I'm not a RN expert it would be nice if you could provide some feedback it this indeed works as expected

andreialecu commented 3 years ago

Going to try this soon! Can you clarify how to install beta 34? I'm not very familiar with Ruby.

tcamin commented 3 years ago

gem install pod-builder --pre

tcamin commented 3 years ago

Note that prebuilding (at least on my mac) takes quite some time. The good is that once this is done, app starts really quickly :-)

andreialecu commented 3 years ago

I have just tried it on a real-world project and I'm getting:

The following pods are in development mode: `BVLinearGradient, BugsnagReactNative, BugsnagReactNative/Core, FBLazyVector, FBReactNativeSpec, Permission-LocationWhenInUse, RCTRequired, RCTTypeSafety, RNAWSCognito, RNBootSplash, RNCAsyncStorage, RNCClipboard, RNCMaskedView, RNDateTimePicker, RNFastImage, RNGestureHandler, RNImageCropPicker, RNImageCropPicker/QBImagePickerController, RNLocalize, RNPermissions, RNReanimated, RNSVG, RNScreens, RNVectorIcons, React, React-Core, React-Core/Default, React-Core/DevSupport, React-Core/RCTWebSocket, React-CoreModules, React-RCTActionSheet, React-RCTAnimation, React-RCTBlob, React-RCTImage, React-RCTLinking, React-RCTNetwork, React-RCTSettings, React-RCTText, React-RCTVibration, React-callinvoker, React-cxxreact, React-jsi, React-jsi/Default, React-jsiexecutor, React-jsinspector, ReactCommon/turbomodule/core, ReactNativeAutogrowTextinput, Yoga, react-native-config, react-native-contacts, react-native-date-picker, react-native-geolocation-service, react-native-get-random-values, react-native-maps, react-native-netinfo, react-native-notifications, react-native-safe-area-context, react-native-selectable-text, toolbar-android`, won't proceed building.

You can ignore this error by passing the `--allow-warnings` flag to the build command

I did add export CONFIGURATION=Release.

Seems that pod_builder build_all --allow-warnings got past it, but not sure if that's expected.

tcamin commented 3 years ago

but not sure if that's expected

yes, I forgot that allow-warnings needs to be passed to properly build. Will make it default for RN projects.

If compilation fails it would help to have a sample project with your setup.

sunnylqm commented 3 years ago

When init I got prompt to input admin password again and again, is that expected?

sunnylqm commented 3 years ago

Got another error with a fresh rn63 project

sunnylqm commented 3 years ago

My system info:

andreialecu commented 3 years ago

@sunnylqm I had the same error related to use_frameworks!. Initially I tried adding it to the ios/Podfile but it didn't help. Then I noticed there's a copy of it in ios/PodBuilder/Podfile and added it there also (manually), and got past the error.

The build itself failed I think though.

There was a Building log line that appeared and stayed for a long time, without any progress logged. Is this normal?

After a while it printed Oh no! Something went wrong during prebuild phase! Do you want to open the prebuild project....

Would've been helpful to see what the actual error was. 🤔

I will fiddle with it some more, and will create a repro soon.

andreialecu commented 3 years ago

@tcamin I added some extra dependencies to the starter project, which should show the build failing. I still haven't confirmed what the error is.

Switch to the more-deps branch on https://github.com/andreialecu/rn63-newproj/tree/more-deps

Remember to run yarn install after switching, so that the deps are installed.

andreialecu commented 3 years ago

Alright, I notice there's a log file created, it seems to contain this as the source of the error:

/Users/andreialecu/Work/rn63-newproj/node_modules/react-native/ReactCommon/yoga/yoga/YGEnums.h:10:10: error: double-quoted include "YGMacros.h" in framework header, expected angle-bracketed instead [-Werror,-Wquoted-include-in-framework-header]
#include "YGMacros.h"
         ^~~~~~~~~~~~
         <YGMacros.h>
1 error generated.

One thing that is weird, is that VSCode, which is the editor of choice for the ecosystem, shows a huge number of git changes after running pod_builder build_all:

Not sure why they're being picked up, seems weird.

tcamin commented 3 years ago

shows a huge number of git changes

You opened the temporary project that PodBuilder uses to prebuild libs (see the path that is /private/tmp/pod_builder). Those are not changes to your project and can be safely ignored.

tcamin commented 3 years ago

@andreialecu beta 35 fixes the issue you reported (sudo gem install pod-builder --pre to update). You need to remove the use_frameworks! you added in the ios/PodBuilder/Podfile as only static library packaging will be supported for RN projects.

tcamin commented 3 years ago

Would've been helpful to see what the actual error was. 🤔

The project that opens when you answer 'Y' to the error message allows you to build the prebuilding project which will show the error in Xcode. Compilation error messages can be very complex and seeing them directly in Xcode is the most effective solution

tcamin commented 3 years ago

@sunnylqm to report an issue please provide a sample project that shows the problem as @andreialecu did.

tcamin commented 3 years ago

@sunnylqm the packaging warning on a fresh project should be fixed in beta 36

andreialecu commented 3 years ago

shows a huge number of git changes

You opened the temporary project that PodBuilder uses to prebuild libs (see the path that is /private/tmp/pod_builder). Those are not changes to your project and can be safely ignored.

FWIW, I did not open that project. I wouldn't even know to find it. Somehow it appeared there automatically. Weird :)

I just tried beta 36 and it seems to be printing this

Building |====================================================================>|
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |
Building |====================>                                                |

It keeps adding new identical lines, it's unclear whether it's doing anything. I can't notice any significant activity in Activity Monitor.

Edit: I'm rerunning it now, on the first progress bar it's doing something, I can notice a lot of CPU usage.

sunnylqm commented 3 years ago

Yeah the same as above, it seems never ending (more than 1 hour on an i7-10700)

tcamin commented 3 years ago

I just tried beta 36 and it seems to be printing this

is this on a local machine or on CI?

@andreialecu the sample project you provided took a very long time to build, around 30minutes.

andreialecu commented 3 years ago

@tcamin local machine, but real project. It's actually using the same list of dependencies as the sample project I provided. So it should work the same. 🤔

I guess I'll try to build the sample and see what happens.

BTW: I think I waited more than 30 minutes. I'm on a 2017 Macbook Pro with 2.5GHz Dual Core i7 and 16 GB of RAM. During the first progress bar it had loud fan noise, but afterwards there was no activity on all the subsequent bars.

tcamin commented 3 years ago

You may want to start trying on your master (instead of the more-deps branch) as there are less dependencies.

How long does it take to build the app for release?

andreialecu commented 3 years ago

Building for release takes significantly less than 30 minutes. On CircleCI it is around 8 minutes. I think locally it's about the same as well.

Just double checked (from CircleCI):

andreialecu commented 3 years ago

Attaching a GIF (progress on first bar seems to have jumped from around the middle, to the end, then it started the loop):

Oct-14-2020 17-58-38

This is with the main project. I'm testing the samples now.

andreialecu commented 3 years ago

@tcamin I just tried on the sample project's master and the same thing happens. First progress bar shows 100%, then it starts printing neverending smaller bars.

tcamin commented 3 years ago

Will check

tcamin commented 3 years ago

I cannot reproduce this the initial jump of the progress bar. Can you try again running pod_builder build_all -u -f?

andreialecu commented 3 years ago

@tcamin It's running now.

Btw, not sure if it's possible or not, but on CI I would still prefer if the full compilation output is shown, in order to be able to debug things if necessary. Perhaps it could run through xcpretty by default, but allow disabling that.

tcamin commented 3 years ago

To debug compilation issues you should definitely run the prebuild project that pod-builder automatically opens when an error occurs (the one that got opened here) . Finding problems from the debug console isn't trivial.

Wrt to the multiple lines that get printed to console what version of ruby-progressbar do you have currently installed?

andreialecu commented 3 years ago

ruby-progressbar 1.10.1

To debug compilation issues you should definitely run the prebuild project that pod-builder automatically opens when an error occurs (the one that got opened here) . Finding problems from the debug console isn't trivial.

Yes, but I cannot debug on CI (should an error only occur there). So if PodBuilder has access to the stdout, it could expose it somehow (maybe --verbose cli flag)

andreialecu commented 3 years ago

pod_builder build_all -u -f just resulted in the same progress bar infinite loop issue. Keep in mind that once that starts happening, there is no more CPU Activity anyway, so I don't think the progress bar is to blame.

I didn't notice any other kind of different output either. Also, I cannot open the prebuild project because I have to ctrl+c out of PodBuilder so I don't get the option.

tcamin commented 3 years ago

ok, I found the issue.

tcamin commented 3 years ago

Give beta 37 a try 🤞

andreialecu commented 3 years ago

🚀 seems like it compiled everything successfully on the master of the sample project. I'll try it out on the real project tomorrow and report back. If this works it will be big! 🤩

andreialecu commented 3 years ago

Seems to have compiled everything successfully on the main project as well! The app also seems to run on the simulator, with some issues:

1) https://github.com/oblador/react-native-vector-icons the vector icons that are part of this package don't seem to render, instead they appear as the unicode missing char symbol [?]. I'll try to figure it out first, and if I can't find anything I'll update the sample project to show a repro.

2) While building for the app store using Fastlane (xcode archive), it seems that the linker complains about:

ld: '/Volumes/Work/Projects/projectsport/ios/PodBuilder/Prebuilt/BVLinearGradient/BVLinearGradient/libBVLinearGradient.a(BVLinearGradientManager.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. file '/Volumes/Work/Projects/projectsport/ios/PodBuilder/Prebuilt/BVLinearGradient/BVLinearGradient/libBVLinearGradient.a' for architecture armv7

And various other pods, and other architectures.

3) Not that I'm complaining, since this seems to reduce like 95% of build time after the initial compilation. But I'm curious why it takes much longer to precompile the pods than it does to fully compile the app, with the pods in it?

Thanks!

tcamin commented 3 years ago

This might be an issue with the RNVectorIcons pod which I remember to have workarounded since it was failing copying some of its resources (it may be a problem with the podspec). If you pinpoint the problem it would make fixing the issue faster.

2.

By default libraries are prebuilt with bitcode disabled (since it is more convenient in most cases). In the sample project you provided you have bitcode enabled in release mode. Just switch it off and errors should go away. If you really need bitcode support you have to enable it also in development and rebuild everything enabling bitcode in PodBuilder. See https://github.com/Subito-it/PodBuilder/tree/deterministic_build#build_settings

3.

The fact is that libraries are built for several architectures. For example your example project deploys back to iOS 10 which means you still support 32 bit devices. This means that PodBuilder has to build 4 architectures (i386, x86_64 simulators, armv7 and arm64 for devices). Bumping to iOS11 should cut the prebuilt times by half since 32 bit devices are no loner supported. The advantage to prebuild all architectures is that you have everything ready both for simulators and devices.

When you normally compile the app you build just one architecture at the time. Even when you archive you build just 2 architectures instead of 4.

Subito-it / PodBuilder

Prebuilding react-native pods #10