[gatsby-source-contentful] What's the cache directory location for this plugin?

[tmilewski]

Summary

I'm using the gatsby-plugin-netlify-cache plugin which, by default, caches the .cache directory on Netlify. I've confirmed that it's successfully caching the appropriate data/directories, but gatsby-source-contentful is still re-requesting all data from Contentful on every build.

I noticed locally that I don't see any data from gatsby-source-contentful in .cache, despite the data not being fully re-requested on every update/build.

So, my simple question is, where is this data being stored, if not in .cache?

Relevant information

Environment

$ gatsby info

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (8) x64 Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
    Shell: 5.6.2 - /usr/local/bin/zsh
  Binaries:
    Node: 12.6.0 - /var/folders/_x/sz_v8_d550v91rpxrz2txstr0000gq/T/yarn--1565711575669-0.9816823406056241/node
    Yarn: 1.17.3 - /var/folders/_x/sz_v8_d550v91rpxrz2txstr0000gq/T/yarn--1565711575669-0.9816823406056241/yarn
    npm: 6.9.0 - /usr/local/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 75.0.3770.100
    Firefox: 68.0.1
    Safari: 12.1.1
  npmPackages:
    gatsby: ^2.13.31 => 2.13.31
    gatsby-image: ^2.2.6 => 2.2.6
    gatsby-plugin-google-fonts: ^1.0.1 => 1.0.1
    gatsby-plugin-google-tagmanager: ^2.1.4 => 2.1.4
    gatsby-plugin-manifest: ^2.2.3 => 2.2.3
    gatsby-plugin-offline: ^2.2.4 => 2.2.4
    gatsby-plugin-purgecss: ^4.0.0 => 4.0.0
    gatsby-plugin-react-helmet: ^3.1.2 => 3.1.2
    gatsby-plugin-sass: ^2.1.3 => 2.1.3
    gatsby-plugin-sharp: ^2.2.8 => 2.2.8
    gatsby-plugin-sitemap: ^2.2.3 => 2.2.3
    gatsby-source-contentful: ^2.1.19 => 2.1.19
    gatsby-source-filesystem: ^2.1.5 => 2.1.5
    gatsby-transformer-remark: ^2.6.7 => 2.6.7
    gatsby-transformer-sharp: ^2.2.4 => 2.2.4

File contents

gatsby-config.js:

/* eslint-disable max-len */
const activeEnv =
  process.env.GATSBY_ACTIVE_ENV || process.env.NODE_ENV || 'development'

require('dotenv').config({
  path: `.env.${activeEnv}`
})

module.exports = {
  siteMetadata: {
    title: `...`,
    description: `...`,
    author: `@...`,
    siteUrl: `https://www.abc.com`,
  },
  plugins: [
    {
      resolve: 'gatsby-plugin-netlify-cache',
      options: {
        cachePublic: true
      }
    },
    {
      resolve: `gatsby-source-contentful`,
      options: {
        spaceId: process.env.CONTENTFUL_SPACE_ID,
        accessToken: process.env.CONTENTFUL_ACCESS_TOKEN,
        host: (process.env.CONTENTFUL_HOST || 'cdn.contentful.com'),
        environment: (process.env.CONTENTFUL_ENV || 'master'),
        downloadLocal: (!!process.env.CONTENTFUL_DOWNLOAD_LOCAL)
      },
    },
    {
      resolve: `gatsby-transformer-remark`
    },
    {
      resolve: `gatsby-plugin-sass`,
      options: {
        implementation: require('sass'),
        postCssPlugins: [
          require('postcss-flexbugs-fixes'),
          require('autoprefixer')
        ],
        cssLoaderOptions: {
          localIdentName: activeEnv === 'development' ?
            '[name]--[local]-[hash:base64:5]' :
            '[local]-[hash:base64:6]'
        }
      },
    },
    `gatsby-plugin-react-helmet`,
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `images`,
        path: `${__dirname}/src/images`,
      },
    },
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    {
      resolve: `gatsby-plugin-manifest`,
      options: {
        name: `...`,
        short_name: `...`,
        start_url: `/`,
        background_color: `#00b3e6`,
        theme_color: `#00b3e6`,
        display: `minimal-ui`,
        icon: `src/images/favicon.ico`,
        crossOrigin: `use-credentials`
      },
    },
    {
      resolve: `gatsby-plugin-google-tagmanager`,
      options: {
        id: 'GTM-...',
        includeInDevelopment: true
      },
    },
    {
      resolve: 'gatsby-plugin-purgecss',
      options: {
        develop: true,
        printRejected: false,
        purgeOnly : ['src/components/Layout/']
      },
    },
    {
      resolve: `gatsby-plugin-sitemap`,
      options: {
        exclude: [
          `/...`,
        ],
        query: `
        {
          site {
            siteMetadata {
              siteUrl
            }
          }

          allSitePage(filter: {fields: {hideFromSearchEngines: {eq: false}}}) {
            edges {
              node {
                path
                fields {
                  lastmod
                  changefreq
                  priority
                }
              }
            }
          }
        }`,
        serialize: ({ site, allSitePage }) =>
          allSitePage.edges.map(edge => {
            return {
              url: site.siteMetadata.siteUrl + edge.node.path,
              lastmod: edge.node.fields.lastmod,
              changefreq: edge.node.fields.changefreq,
              priority: edge.node.fields.priority,
            }
          }),
      }
    }
  ],
}

package.json: N/A

{
  "name": "abc.com",
  "private": true,
  "description": "...",
  "version": "5.0.0",
  "author": "... <info@abc.com>",
  "browserslist": [
    ">1%",
    "last 4 versions",
    "not dead"
  ],
  "dependencies": {
    "@contentful/rich-text-plain-text-renderer": "^13.1.0",
    "@contentful/rich-text-react-renderer": "^13.2.0",
    "@reach/tabs": "^0.1.6",
    "airbrake-js": "^1.6.8",
    "autoprefixer": "^9.6.1",
    "axios": "^0.19.0",
    "babel-runtime": "^6.26.0",
    "classnames": "^2.2.6",
    "fs-extra": "^7.0.0",
    "gatsby": "^2.13.31",
    "gatsby-image": "^2.2.6",
    "gatsby-plugin-google-fonts": "^1.0.1",
    "gatsby-plugin-google-tagmanager": "^2.1.4",
    "gatsby-plugin-manifest": "^2.2.3",
    "gatsby-plugin-offline": "^2.2.4",
    "gatsby-plugin-purgecss": "^4.0.0",
    "gatsby-plugin-react-helmet": "^3.1.2",
    "gatsby-plugin-sass": "^2.1.3",
    "gatsby-plugin-sharp": "^2.2.8",
    "gatsby-plugin-sitemap": "^2.2.3",
    "gatsby-source-contentful": "^2.1.19",
    "gatsby-source-filesystem": "^2.1.5",
    "gatsby-transformer-remark": "^2.6.7",
    "gatsby-transformer-sharp": "^2.2.4",
    "js-cookie": "^2.2.0",
    "lodash.kebabcase": "^4.1.1",
    "node-sass": "^4.11.0",
    "postcss-flexbugs-fixes": "^4.1.0",
    "program-language-detector": "^0.0.12",
    "prop-types": "^15.7.2",
    "query-string": "^6.8.1",
    "react": "^16.8.2",
    "react-collapsible": "^2.6.0",
    "react-contentful-image": "^1.3.2",
    "react-device-detect": "^1.6.5",
    "react-dom": "npm:@hot-loader/react-dom",
    "react-helmet": "^5.2.1",
    "react-modal": "^3.9.1",
    "react-player": "^1.11.1",
    "react-reveal": "^1.2.2",
    "react-slick": "^0.24.0",
    "react-sticky-el": "^1.0.20",
    "react-syntax-highlighter": "^11.0.2",
    "react-visibility-sensor": "^5.1.0",
    "sass": "^1.22.7",
    "scroll-into-view": "^1.9.7",
    "sharp": "^0.22.1",
    "slick-carousel": "^1.8.1",
    "url-search-params-polyfill": "^7.0.0",
    "uuid": "^3.3.2"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "start": "npm run develop",
    "serve": "gatsby serve",
    "test": "jest src",
    "coverage": "jest src --coverage",
    "storybook": "STORYBOOK_ENV=true start-storybook -p 6006",
    "build-storybook": "build-storybook"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/..."
  },
  "bugs": {
    "url": "https://github.com/.../issues"
  },
  "devDependencies": {
    "@babel/core": "^7.5.5",
    "@storybook/addon-a11y": "^5.1.9",
    "@storybook/addon-actions": "^5.1.9",
    "@storybook/addon-backgrounds": "^5.1.9",
    "@storybook/addon-links": "^5.1.9",
    "@storybook/addon-notes": "^5.1.9",
    "@storybook/addon-storysource": "^5.1.9",
    "@storybook/addon-viewport": "^5.1.9",
    "@storybook/addons": "^5.1.9",
    "@storybook/react": "^5.1.9",
    "@storybook/theming": "^5.1.9",
    "@testing-library/jest-dom": "^4.0.0",
    "@testing-library/react": "^8.0.5",
    "@testing-library/react-hooks": "^1.1.0",
    "axios-mock-adapter": "^1.17.0",
    "babel-eslint": "^10.0.2",
    "babel-jest": "^24.7.1",
    "babel-loader": "^8.0.5",
    "babel-plugin-remove-graphql-queries": "^2.7.2",
    "babel-preset-gatsby": "^0.2.7",
    "cross-env": "^5.2.0",
    "eslint": "^6.0.1",
    "eslint-config-fbjs": "^3.1.0",
    "eslint-plugin-babel": "^5.3.0",
    "eslint-plugin-flowtype": "^3.12.1",
    "eslint-plugin-jsx-a11y": "^6.2.3",
    "eslint-plugin-react": "^7.14.2",
    "eslint-plugin-relay": "^1.3.5",
    "identity-obj-proxy": "^3.0.0",
    "jest": "^24.7.1",
    "jest-localstorage-mock": "^2.4.0",
    "react-test-renderer": "^16.8.6"
  }
}

gatsby-node.js: Can't currently share. gatsby-browser.js: N/A gatsby-ssr.js: N/A

[axe312ger]

It uses gatsby-source-contentfuls createRemoteFileNode to cache it, which stores in PROJECT_DIR/.cache/gatsby-source-filesystem

This will be cached when using gatsby-plugin-netlify-cache, but all files get deleted when the gatsby cache is reset. (Which can happen more often as you would think)

I can see how this can be very bad for huge sites. SQIP and the new video transformer will use node_modules/.cache/XYZ. We could make source-filesystem use that folder as well? Files won't be deleted on Gatsby cache reset and caching on netlify would even work without the netlify-cache plugin.

[tmilewski]

Thanks for the reply, @axe312ger!

The approach of saving it in node_modules/.cache/XYZ would likely help. Long-term we might want to enable a simple command to clear the cache for each plugin as it might not be 100% clear where this data resides.

---

In the meantime, I don't know that the issue is that the cache is being reset. See below for the process:

Clean Build

rm -r .cache; yarn build yields the following:

Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 28
Updated entries  2343
Deleted entries  0
Updated assets  2929
Deleted assets  0
Fetch Contentful data: 11828.563ms

Checking Cache Directories

No cached files to be found.

🔌  2:16 in dotcom/ on develop 
› ls -al .cache/gatsby-source-filesystem      
gls: cannot access '.cache/gatsby-source-filesystem': No such file or directory

🔌  2:16 in dotcom/ on develop 
› ls -al .cache/caches                         
total 0
drwxr-xr-x 18 tmilewski staff  576 Aug 14 09:41 ./
drwxr-xr-x 47 tmilewski staff 1504 Aug 14 09:41 ../
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 default-site-plugin/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 dev-404-page/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-manifest/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-page-creator/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-purgecss/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-sass/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-plugin-sharp/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:41 gatsby-plugin-sitemap/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-source-contentful/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 gatsby-source-filesystem/
drwxr-xr-x 35 tmilewski staff 1120 Aug 14 09:41 gatsby-transformer-remark/
drwxr-xr-x  3 tmilewski staff   96 Aug 14 09:40 gatsby-transformer-sharp/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 internal-data-bridge/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 load-babel-config/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 prod-404/
drwxr-xr-x  2 tmilewski staff   64 Aug 14 09:40 webpack-theme-component-shadowing/

🔌  2:16 in dotcom/ on develop 
› ls -al .cache/caches/gatsby-source-filesystem
total 0
drwxr-xr-x  2 tmilewski staff  64 Aug 14 09:40 ./
drwxr-xr-x 18 tmilewski staff 576 Aug 14 09:41 ../

🔌  2:16 in dotcom/ on develop
› ls -al .cache/caches/gatsby-source-contentful 
total 0
drwxr-xr-x  2 tmilewski staff  64 Aug 14 09:40 ./
drwxr-xr-x 18 tmilewski staff 576 Aug 14 09:41 ../

Second Build

yarn build yields the following, apparently with cached files:

Starting to fetch data from Contentful
Fetching default locale
default locale is : en-US
contentTypes fetched 28
Updated entries  0
Deleted entries  0
Updated assets  0
Deleted assets  0
Fetch Contentful data: 598.905ms

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[axe312ger]

I sometimes have this issue as well, just my sites are not that huge so I can wait when this happens...

Stil... what does @gatsbyjs/inkteam think about changing the cache dir for createRemoteFileNode to node_modules/.cache/...?

[LekoArts]

That's something for v3: https://github.com/gatsbyjs/gatsby/issues/5880

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!