search

jlengstorf / netlify-plugin-gatsby-cache

185 stars 21 forks source link

Cache always empty on Deploy Preview builds #22

Closed ffdead closed 3 years ago

ffdead commented 4 years ago

Hi! Big thanks for this plugin - it's helping to bring down the build time on our develop branch significantly.

However, on Deploy Preview builds the cache seems to always be empty - even on repeat builds. Is this a know limitation of the plugin?

We were able to remove the re-install overhead of the plugin by following the steps in https://github.com/netlify/build/issues/1134 - but the yarn command still takes ~1 minute on the Deploy Preview builds for some reason - compared to ~1 second on the develop branch. (related?)

Deploy Preview log on a repeat build:

Deploy Preview #103: feature/footer-drawing@80783d2.

1:14:00 AM: Build ready to start
1:14:02 AM: build-image version: 2dbd444fcdce00cf06325060a8238d5ae3e86774
1:14:02 AM: build-image tag: v3.3.7
1:14:02 AM: buildbot version: ed44b74dccd74a30dec84aacc6e49a3511498577
1:14:02 AM: Fetching cached dependencies
1:14:02 AM: Starting to download cache of 207.6MB
1:14:10 AM: Finished downloading cache in 7.504368281s
1:14:10 AM: Starting to extract cache
1:14:18 AM: Finished extracting cache in 8.584933729s
1:14:18 AM: Finished fetching cache in 16.152955811s
1:14:18 AM: Starting to prepare the repo for build
1:14:19 AM: Preparing Git Reference pull/103/head
1:14:21 AM: Starting build script
1:14:21 AM: Installing dependencies
1:14:22 AM: Started restoring cached node version
1:14:26 AM: Finished restoring cached node version
1:14:26 AM: Attempting node version 'v10.15.1' from .nvmrc
1:14:27 AM: v10.15.1 is already installed.
1:14:27 AM: Now using node v10.15.1 (npm v6.4.1)
1:14:28 AM: Attempting ruby version 2.6.2, read from environment
1:14:29 AM: Using ruby version 2.6.2
1:14:29 AM: Using PHP version 5.6
1:14:29 AM: Started restoring cached node modules
1:14:29 AM: Finished restoring cached node modules
1:14:29 AM: Started restoring cached yarn cache
1:14:29 AM: Finished restoring cached yarn cache
1:14:30 AM: Installing NPM modules using Yarn version 1.13.0
1:14:31 AM: yarn install v1.13.0
1:14:31 AM: [1/4] Resolving packages...
1:14:32 AM: [2/4] Fetching packages...
1:15:03 AM: info fsevents@2.1.2: The platform "linux" is incompatible with this module.
1:15:03 AM: info "fsevents@2.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
1:15:03 AM: info fsevents@1.2.12: The platform "linux" is incompatible with this module.
1:15:03 AM: info "fsevents@1.2.12" is an optional dependency and failed compatibility check. Excluding it from installation.
1:15:03 AM: [3/4] Linking dependencies...
1:15:03 AM: warning " > eslint-loader@3.0.3" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
1:15:03 AM: warning "gatsby > react-hot-loader@4.12.20" has unmet peer dependency "@types/react@^15.0.0 || ^16.0.0".
1:15:03 AM: warning "gatsby > @typescript-eslint/eslint-plugin > tsutils@3.17.1" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
1:15:03 AM: warning " > gatsby-plugin-eslint@2.0.8" has incorrect peer dependency "eslint@^6.6.0".
1:15:03 AM: warning "gatsby-plugin-netlify > webpack-assets-manifest@3.1.1" has unmet peer dependency "webpack@>=4.4.0".
1:15:03 AM: warning "gatsby-plugin-transition-link > babel-preset-gatsby-package > @babel/preset-flow@7.9.0" has unmet peer dependency "@babel/core@^7.0.0-0".
1:15:03 AM: warning "gatsby-plugin-transition-link > babel-preset-gatsby-package > @babel/preset-flow > @babel/plugin-transform-flow-strip-types@7.9.0" has unmet peer dependency "@babel/core@^7.0.0-0".
1:15:03 AM: warning "gatsby-plugin-transition-link > babel-preset-gatsby-package > @babel/preset-flow > @babel/plugin-transform-flow-strip-types > @babel/plugin-syntax-flow@7.8.3" has unmet peer dependency "@babel/core@^7.0.0-0".
1:15:03 AM: warning " > react-three-fiber@4.0.23" has incorrect peer dependency "three@>=0.114".
1:15:35 AM: [4/4] Building fresh packages...
1:15:40 AM: Done in 69.20s.
1:15:40 AM: NPM modules installed using Yarn
1:15:40 AM: Started restoring cached go cache
1:15:40 AM: Finished restoring cached go cache
1:15:40 AM: unset GOOS;
1:15:40 AM: unset GOARCH;
1:15:40 AM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
1:15:40 AM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
1:15:40 AM: go version >&2;
1:15:40 AM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
1:15:40 AM: go version go1.12 linux/amd64
1:15:40 AM: Installing missing commands
1:15:40 AM: Verify run directory
1:15:42 AM: ​
1:15:42 AM: ┌─────────────────────────────┐
1:15:42 AM: │        Netlify Build        │
1:15:42 AM: └─────────────────────────────┘
1:15:42 AM: ​
1:15:42 AM: ❯ Version
1:15:42 AM:   @netlify/build 0.4.3
1:15:42 AM: ​
1:15:42 AM: ❯ Flags
1:15:42 AM:   mode: buildbot
1:15:42 AM: ​
1:15:42 AM: ❯ Current directory
1:15:42 AM:   /opt/build/repo
1:15:42 AM: ​
1:15:42 AM: ❯ Config file
1:15:42 AM:   /opt/build/repo/netlify.toml
1:15:42 AM: ​
1:15:42 AM: ❯ Context
1:15:42 AM:   deploy-preview
1:15:44 AM: ​
1:15:44 AM: ❯ Loading plugins
1:15:44 AM:    - netlify-plugin-gatsby-cache@0.2.1
1:15:44 AM: ​
1:15:44 AM: ┌────────────────────────────────────────────────────────┐
1:15:44 AM: │ 1. onPreBuild command from netlify-plugin-gatsby-cache │
1:15:44 AM: └────────────────────────────────────────────────────────┘
1:15:44 AM: ​
1:15:44 AM: No Gatsby cache found. Building fresh.
1:15:44 AM: ​
1:15:44 AM: (netlify-plugin-gatsby-cache onPreBuild completed in 8ms)
1:15:44 AM: ​
1:15:44 AM: ┌────────────────────────────────────┐
1:15:44 AM: │ 2. build.command from netlify.toml │
1:15:44 AM: └────────────────────────────────────┘
1:15:44 AM: ​
1:15:44 AM: $ yarn build
1:23:34 AM: yarn run v1.13.0
1:23:34 AM: $ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
1:23:34 AM: success open and validate gatsby-configs - 0.113s
1:23:34 AM: success load plugins - 1.494s
1:23:34 AM: success onPreInit - 0.012s
1:23:34 AM: success initialize cache - 0.011s
1:23:34 AM: success copy gatsby files - 0.083s
1:23:34 AM: success onPreBootstrap - 0.010s
1:23:34 AM: success createSchemaCustomization - 0.006s
1:23:34 AM: success source and transform nodes - 0.696s
1:23:34 AM: success building schema - 0.471s
1:23:34 AM: success createPages - 0.222s
1:23:34 AM: success createPagesStatefully - 0.141s
1:23:34 AM: success onPreExtractQueries - 0.000s
1:23:34 AM: success update schema - 0.102s
1:23:34 AM: success extract queries from components - 0.834s
1:23:34 AM: success write out requires - 0.005s
1:23:34 AM: success write out redirect data - 0.001s
1:23:34 AM: success Build manifest and related icons - 0.446s
1:23:34 AM: success onPostBootstrap - 0.448s
1:23:34 AM: ⠀
1:23:34 AM: info bootstrap finished - 7.460 s
1:23:34 AM: ⠀
1:23:34 AM: Webpack Bundle Analyzer is started at http://127.0.0.1:3001
1:23:34 AM: Use Ctrl+C to close it
1:23:34 AM: success Building production JavaScript and CSS bundles - 83.182s
1:23:34 AM: success Rewriting compilation hashes - 0.001s
1:23:34 AM: success Downloading remote files - 13.993s - 67/67 4.79/s
1:23:34 AM: success run queries - 99.006s - 19/19 0.19/s
1:23:34 AM: (node:1633) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
1:23:34 AM: success Building static HTML for pages - 21.481s - 16/16 0.74/s
1:23:34 AM: success Delete previous page data - 0.000s
1:23:34 AM: success Generating image thumbnails - 377.603s - 468/468 1.24/s
1:23:34 AM: warning app-c4b73d405234476ae39c.js.map is 4.16 MB, and won't be precached. Configure maximumFileSizeToCacheInBytes to change this limit.
1:23:34 AM: warning static/ce6ef49ec41fd5ce26895a8ddbd70a87/b0260/b9f5c0ab-9347-462d-8d26-2d13c4233e11_14-case-antler-header02.png is 2.97 MB, and won't be precached. Configure maximumFileSizeToCacheInBytes to change this limit.
1:23:34 AM: info Generated public/sw.js, which will precache 565 files, totaling 51355309 bytes.
1:23:34 AM: The following pages will be precached:
1:23:34 AM: /offline-plugin-app-shell-fallback/index.html
1:23:34 AM: success onPostBuild - 0.452s
1:23:34 AM: info Done building in 469.796 sec
1:23:34 AM: info Built pages:
1:23:34 AM: Updated page: /preview
1:23:34 AM: Updated page: /project
1:23:34 AM: Updated page: /santa-tracker
1:23:34 AM: Updated page: /interland
1:23:34 AM: Updated page: /antler
1:23:34 AM: Updated page: /just-food-for-dogs
1:23:34 AM: Updated page: /vardagskampen
1:23:34 AM: Updated page: /proxy
1:23:34 AM: Updated page: /offline-plugin-app-shell-fallback/
1:23:34 AM: Updated page: /404/
1:23:34 AM: Updated page: /about/
1:23:34 AM: Updated page: /contact/
1:23:34 AM: Updated page: /
1:23:34 AM: Updated page: /lab/
1:23:34 AM: Updated page: /scroll-test/
1:23:34 AM: Updated page: /404.html
1:23:34 AM: Done in 470.19s.
1:23:34 AM: ​
1:23:34 AM: (build.command completed in 7m 50.5s)
1:23:34 AM: ​
1:23:34 AM: ┌─────────────────────────────────────────────────────────┐
1:23:34 AM: │ 3. onPostBuild command from netlify-plugin-gatsby-cache │
1:23:34 AM: └─────────────────────────────────────────────────────────┘
1:23:34 AM: ​
1:23:35 AM: Stored the Gatsby cache to speed up future builds.
1:23:35 AM: ​
1:23:35 AM: (netlify-plugin-gatsby-cache onPostBuild completed in 1s)
1:23:35 AM: ​
1:23:35 AM: ┌─────────────────────────────┐
1:23:35 AM: │   Netlify Build Complete    │
1:23:35 AM: └─────────────────────────────┘
1:23:35 AM: ​
1:23:35 AM: (Netlify Build completed in 7m 53.1s)
1:23:36 AM: Caching artifacts
1:23:36 AM: Started saving node modules
1:23:36 AM: Finished saving node modules
1:23:36 AM: Started saving yarn cache
1:23:36 AM: Finished saving yarn cache
1:23:36 AM: Started saving pip cache
1:23:36 AM: Finished saving pip cache
1:23:36 AM: Started saving emacs cask dependencies
1:23:36 AM: Finished saving emacs cask dependencies
1:23:36 AM: Started saving maven dependencies
1:23:36 AM: Finished saving maven dependencies
1:23:36 AM: Started saving boot dependencies
1:23:37 AM: Finished saving boot dependencies
1:23:37 AM: Started saving go dependencies
1:23:37 AM: Finished saving go dependencies
1:23:37 AM: Build script success
1:23:37 AM: Starting to deploy site from 'public'
1:23:37 AM: Creating deploy tree asynchronously
1:23:37 AM: Creating deploy upload records
1:23:39 AM: 23 new files to upload
1:23:39 AM: 0 new functions to upload
1:23:41 AM: Starting post processing
1:23:48 AM: Post processing done
1:23:48 AM: Site is live
1:24:26 AM: Finished processing build request in 10m24.269252737s
jlengstorf commented 4 years ago

I'm not 100% sure; I would expect that a deploy preview would use the cache, but there's a possibility that this is done to prevent conflicts since the branches won't match

cc @ehmicky — is this intended, or should I file a buildbot bug?

ehmicky commented 4 years ago

I think I've heard there were some reasons why deploy preview caches are sometimes empty, partially intended, but also partially due to existing issues. I would suggest asking the buildbot team.

ffdead commented 4 years ago

Related? https://github.com/netlify/build/issues/1225

jlengstorf commented 4 years ago

yeah, that's related. I was able to duplicate on a non-Gatsby site as well, so something’s going on for sure

keeko20 commented 4 years ago

Hi, This is also happening to me it keeps showing on Netlify build console "No Gatsby cache found. Building fresh."

the build times pretty much the same with or without build plugin.

Any advice? thank you

ffdead commented 4 years ago

~FYI The cache seems to work on Deploy previews for us as of yesterday. no change on our end.~ Deploy Previews were blazing fast during Monday 18th of May - now they are slow again.

kaganjd commented 3 years ago

Exciting update: this was an upstream Netlify issue that has been fixed 🎉

Check out our Community post for more details: https://community.netlify.com/t/build-cache-is-now-updated-in-deploy-preview-builds/26344

jlengstorf commented 3 years ago

thanks so much for the update @kaganjd!