Caching Vendors

[mdere-unbound]

Describe the solution you'd like From what I can tell there is no way a node_module can be cached. Would be awesome if we had that option so we can speed up our build.

Describe alternatives you've considered Can commit my whole node_module to repo, but wont.

Additional context Is it a huge LOE?

[nilofer]

Hi @mdere-unbound thanks for creating this feature request. In terms of your question, it is a non-trivial change and we are constantly evaluating what the next features we should work on are. We'll keep this open to track further interest in this!

[mdere-unbound]

@nrajpurkar - I appreciate that! Thanks!

[GrillAndChill]

My team can't use App center due to the fact that the builds are timing out.

We are using React Native. Which requires NPM packages and Cocoapods for iOS.

NPM itself takes about 7-10min, and Cocoapods goes even longer.

[lechuckcaptain]

Same issue here, it would be nice to have the possibility to cache Cocoapods (and similar) dependencies

[kincade-dev]

seconded- the speed benefits of tools Carthage are wasted on build solutions that don't support caching.

[danilowoz]

@nilofer I would really like this issue to move forward. It seems more an essential feature in terms of builds tool than an improvement. My team needs to spend about 15 minutes to know that a problem occurred in the build step, or even when it has multiples build in the queue, it takes too much time to do the same step in every single build.

All our projects use Appcenter - which is a fantastic tool - and we are willing to continue using it, since Appcenter improves these issues that for us is crucial.

[AndresTapiero72]

@nilofer Any progress with this problem? I also use npm and Cocoapods for iOS and my builds are doing timeout sometimes

[akiwarheit]

bumping this - we're timing out on iOS due to build times. pod install making up a quarter of it.

 Initialize job --- took: 0.046816666666666666 minutes
 Install Apple certificate --- took: 0.03546666666666667 minutes
 Install Apple provisioning profile --- took: 0.012466666666666666 minutes
 Select Node.js Version v2 --- took: 0.1 minutes
 Install build scripts --- took: 0.0538 minutes
 Pre Build Script --- took: 0.012066666666666667 minutes
 Generate source map --- took: 3.1489333333333334 minutes
 Copy source map file to staging --- took: 0.005066666666666666 minutes
 Tag build --- took: 0.011166666666666667 minutes
 Set Xcode path --- took: 0.0047666666666666664 minutes
 Determine signing style and pods update --- took: 0.050883333333333336 minutes
 Pod install --- took: 16.725683333333333 minutes
 Set Bundle version --- took: 0.03266666666666666 minutes
 Update Xcode build settings --- took: 0.018583333333333334 minutes
 Xcode build (signed) --- took: 0.0038666666666666667 minutes
 Install Apple provisioning profile --- took: 0.0048 minutes
 Install Apple certificate --- took: 0.0054666666666666665 minutes
 Checkout app@release/platform to s --- took: 0.0075 minutes
 Finalize Job --- took: 0.004216666666666666 minutes
 Determine project file path --- took: 0.003683333333333333 minutes
 yarn/npm install --- took: 3.0921833333333333 minutes
 Checkout app@release/platform to s --- took: 2.0602666666666667 minutes
 Xcode build (signed) --- took: 34.56445 minutes

UPDATE: If anyones having trouble with build times on iOS - this is how I optimized ours:

1. Make sure it's using cocoapods CDN (source 'https://cdn.cocoapods.org/' inside Podfile)
2. Get rid of Flipper (this gets built in production but not linked)
3. Using legacy build system
4. Disabling bitcode
5. SWIFT_COMPILATION_MODE -> wholemodule

we were timing out before but now we're at 45mins.

 Initialize job --- took: 0.349 minutes
 Install Apple certificate --- took: 0.04156666666666667 minutes
 Install Apple provisioning profile --- took: 0.02385 minutes
 Install build scripts --- took: 0.05736666666666667 minutes
 Select Node.js Version v2 --- took: 0.14723333333333333 minutes
 Pre Build Script --- took: 0.013466666666666667 minutes
 Generate source map --- took: 3.20485 minutes
 Copy source map file to staging --- took: 0.007233333333333333 minutes
 Tag build --- took: 0.0145 minutes
 Set Xcode path --- took: 0.029666666666666668 minutes
 Pod install --- took: 6.499983333333334 minutes
 Determine project file path --- took: 0.0042 minutes
 Determine signing style and pods update --- took: 0.054433333333333334 minutes
 Set Bundle version --- took: 0.031516666666666665 minutes
 Update Xcode build settings --- took: 0.0142 minutes
 Copy build files to staging --- took: 0.012616666666666667 minutes
 Publish build --- took: 0.23125 minutes
 Copy symbol files to staging --- took: 0.020666666666666667 minutes
 Xcode build (signed) --- took: 0.0039 minutes
 Publish symbols --- took: 0.2991333333333333 minutes
 Install Apple provisioning profile --- took: 0.00385 minutes
 Install Apple certificate --- took: 0.004633333333333333 minutes
 Checkout app@release/platform to s --- took: 0.0089 minutes
 Finalize Job --- took: 0.004433333333333333 minutes
 Checkout app@release/platform to s --- took: 2.1555333333333335 minutes
 yarn/npm install --- took: 3.3934166666666665 minutes
 Xcode build (signed) --- took: 29.821766666666665 minutes

[RobertSasak]

@akiwarheit Can you please explain in more detail how to get rid of Flipper in production build? I would like to keep flipper for debugging purpose only. There is not need to add source 'https://cdn.cocoapods.org/' into Podfile as it is used by pod 1.8.0 or higher. Which is also case of Appcenter.

[dylan-westbury]

We have a very simple app that is just a base project for authentication, no more than 10 screens, only a a few SDKs used including ms app centre and a pre-build script that lasts 3 mins.

Our build was timing out at 30mins, we disabled flipper and got it to 22 minutes. Just seems still like a long time to increase.

I was able to disable flipper by commenting it out 2 lines out in the podfile:

  # Enables Flipper.
  #
  # Note that if you have use_frameworks! enabled, Flipper will not work and
  # you should disable these next few lines.
  # use_flipper!
  post_install do |installer|
    # flipper_post_install(installer)
    end
  end

Still seems like a long time for a simple app, probably some more optimisations that can be made, caching of vendor assets would help greatly.

[andrew-skrypnyk]

Is there any way to completely remove those limits? Our big iOS project takes more than 1.5 hours to build and run all the necessary tests and we already optimize everything we could.

[romanonthego]

For react-native app it can be helped a little:

1. Add CI=true in environment variables
2. Modify Podfile (Podfile is just ruby so you actually can do whatever you want)

   
   IS_CI = ENV['CI']
   # ...

if not IS_CI use_flipper! end

...

post_install do |installer| if not IS_CI use_flipper! end end

3. Disabled bitcode in Xcode.

Shaved good 10 minutes of my builds (35m vs 24m).

[kyle-ssg]

Ah thanks so much for that. I've made a slight adjustment on above, you can just use APPCENTER_BUILD_ID rather than manually adding a var.

Here's what mine looks like:

require_relative '../node_modules/react-native/scripts/react_native_pods'
require_relative '../node_modules/@react-native-community/cli-platform-ios/native_modules'
IS_CI = ENV['APPCENTER_BUILD_ID']
platform :ios, '12.0'

def shared_pods
  config = use_native_modules!
  use_react_native!(:path => config["reactNativePath"])
  if not IS_CI
    use_flipper!({ 'Flipper' => '0.74.0' })
  end
end

target 'mobile' do
    shared_pods
end

target 'appcenter-production' do
    shared_pods
end

post_install do |installer|
    if not IS_CI
        flipper_post_install(installer)
    end
end

Before optimisations: 25m After Podfile change: 18m

Disabling bitcode had no effect for me.

[mifi]

@romanonthego's solution also helped me shave 10 min off the build to be able to build it in 25min (using the free AppCenter 30min limit) 🎉 bitcode still enabled

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[zanechua]

I would say that there is no official AppCenter way of caching however, there is definitely a way to manually cache the necessary node_modules folder and improve build times.

I'll probably write up a post on it, but suggestions would be to use Azure Storage Containers that are located in the US since all the machines are in the US. Otherwise you're going to get abysmal speeds when uploading/downloading the cache.

[zanechua]

Hey folks,

Here's how we can do caching for AppCenter and how I am doing it for my app.

https://zanechua.com/blog/appcenter-caching-vendor-libraries

[micnguyen]

Hey folks,

Here's how we can do caching for AppCenter and how I am doing it for my app.

https://zanechua.com/blog/appcenter-caching-vendor-libraries

Thanks for your article, however the problem isn't the downloading of the dependencies. A large amount of the time is the building of the pods

[zanechua]

Hey folks, Here's how we can do caching for AppCenter and how I am doing it for my app. https://zanechua.com/blog/appcenter-caching-vendor-libraries

Thanks for your article, however the problem isn't the downloading of the dependencies. A large amount of the time is the building of the pods

I have another article coming out which addresses that specifically.

If I remember correctly, it's the "xcodebuild" command that cleans the "DerivedData" directory therefore even if you cache that it's pretty much useless.

But you might wanna take a look at https://github.com/mbitsnbites/buildcache if you wanna try implementing it, it's pretty simple. I'll post my article going through that soon.

Also have another trick up my sleeve to cut build times on AppCenter. 😊 I'll also be sharing that in a separate article.

If you do implement it, it's pretty much the same steps as caching node_modules but now it's for buildcache

[zanechua]

@micnguyen

Since you were specifically asking for it.

Here's the article about reducing build times for iOS:

https://zanechua.com/blog/reduce-react-native-xcode-build-time

Even though it says react-native, this will work with any iOS project since buildcache is meant for C and C++ build products

[micnguyen]

Thanks @zanechua - I am also doing a react-native project on MS AzureDevOps and will report back in a few weeks when I can implement this.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[bpofficial]

Bumping to keep feature request alive

[bobanminic96]

I'm trying to build almost freshly created project and release it to Test Flight, few packages included, and it gets canceled after 30minutes without any reason there. I guess it is related to timeout.

[iamandiradu]

@bobanminic96, the free tier has a 30m time limit.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[zanechua]

obligatory bump

[tmadej]

Huge issue in our workflow. Please prioritize, MS.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[tmadej]

Bump. This is still an issue.

[VitalikKizlov]

still issues

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[zanechua]

obligatory bump

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ThomasReyskens]

yup, still an issue

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[zanechua]

obligatory bump

[Nolascoin]

Happy New Year 2023, this issue still exists btw.

Update Feb 2023 Ended up moving to Microsoft's Github Actions with fastlane. Now I get pod/npm caching, 60+ minutes, upload builds to AppCenter for distribution, and it just works (React Native 0.70) both iOS and Android.

[jpangan]

This is still an issue until now on Valentine's day 2023

[jeemercado]

Bump

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[nriccar]

Bump

[michaelachrisco]

This is causing issues with an app we are building as well.

[jtvargas]

bump

[ilirhushi]

bump

[msannan]

still an issue. How to fix it? IOS build is taking 28 minutes

[mefjuu]

still an issue. How to fix it? (...)

Consider migrating to Bitrise and take a breath.