Issue when caching the whole repository

[ehmicky]

This plugin caches PUBLISH_DIR (renamed from BUILD_DIR) which is the "Publish directory" set in the UI or in the configuration file (build.publish). When unset, this defaults to the "build directory", which is (in most cases) the repository root directory.

However caching (and restoring) the whole repository has several issues:

• It caches many directories that are not related to Gatsby but take lots of space, such as node_modules. Those should either not be cached or be cached separately from this plugin.
• It caches directories that are not meant to restored across builds. For example .git (beside being big) is being restored by this plugin, which should not be the case.
• It undoes any build operations inside the repository that might have happened before this plugin starts restoring the cache, leading to conflicts with other plugins, Netlify Build or the buildbot.

In short, caching the whole repository does not seem to be a viable option.

Please note that this is only an issue when no "Publish directory" was set. When one is set, caching it and restoring it should just work.

When no "Publish directory" is set, is there any specific files or directories that we could cache instead to avoid the issues described above?

[jlengstorf]

how does a Gatsby site publish properly if the directory isn't set to public? wouldn't that fail?

Gatsby doesn't let you change the name of public, so we could do something like

const dirToCache = constanst.PUBLISH_DIR === 'public' ? constants.PUBLISH_DIR : path.join(constants.PUBLISH_DIR, 'public');

but I'm confused how a site would deploy at all without specifying where the files to deploy are?

[ehmicky]

When no "Publish directory" is specified, Netlify (with or without the Netlify Build beta) defaults to the "build directory" which (in most cases) is the repository root directory.

I am not sure whether our infrastructure defaults the "Publish directory" to publish when we detect Gatsby is used, I am going to check on this.

About Gatsby:

• Can you confirm the build directory is always called public?
• Is this public directory relative to $PWD, to the package root directory (first parent with a package.json) or to the repository root (first parent with a .git directory)?
• Could we just cache this public directory?

[jlengstorf]

• the build directory is always called public
• the public directory is always at the root of the site, but that can be in a monorepo (e.g. at root, or at packages/site/ or whatever)
• we have to cache both public and .cache

[ehmicky]

Thanks @jlengstorf!

Monorepos are supported through the use of the "Base directory" UI setting (or build.base in netlify.toml). Then this is set, the plugin's current directory will be packages/site/ (for example). Otherwise the repository root will be the plugin's current directory.

I.e. to fix this issue, we should cache ./public and ./.cache.

[ehmicky]

Edit: PUBLISH_DIR should actually work providing it is correctly set. If not correctly set, deploys will fail, so we decided to still use that constant, which means the current value is correct.

I.e. the problem is not coming from the question described above. I am doing more investigating now.

[ehmicky]

After investigating, it seems that everything is good with netlify-plugin-gatsby-cache :+1:

More details here.

[janosh]

@ehmicky I think there actually is something to be fixed here. Just adding

[[plugins]]
  package = "netlify-plugin-gatsby-cache"

to my netlify.toml, I get this error:

9:21:02 AM: ┌─────────────────────────────────────────────────────┐
9:21:02 AM: │ Plugin "netlify-plugin-gatsby-cache" internal error │
9:21:02 AM: └─────────────────────────────────────────────────────┘
9:21:02 AM: ​
9:21:02 AM: ❯ Error message
9:21:02 AM:   Error: Cannot cache /opt/build/repo because it is the current directory (/opt/build/repo) or a parent directory​
9:21:02 AM: ​
9:21:02 AM: ❯ Plugin details
9:21:02 AM:   ID:             netlify-plugin-gatsby-cache
9:21:02 AM:   Version:        0.2.1
9:21:02 AM:   Repository:     git+https://github.com/jlengstorf/netlify-plugin-gatsby-cache.git
9:21:02 AM:   npm link:       https://www.npmjs.com/package/netlify-plugin-gatsby-cache
9:21:02 AM:   Report issues:  https://github.com/jlengstorf/netlify-plugin-gatsby-cache/issues​
9:21:02 AM: ​
9:21:02 AM: ❯ Error location
9:21:02 AM:   In "onPreBuild" event in npm package "netlify-plugin-gatsby-cache"
9:21:02 AM:       at checkSrcPath (/opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/node_modules/@netlify/cache-utils/src/path.js:32:11)
9:21:02 AM:       at getSrcPath (/opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/node_modules/@netlify/cache-utils/src/path.js:18:3)
9:21:02 AM:       at parsePath (/opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/node_modules/@netlify/cache-utils/src/path.js:9:19)
9:21:02 AM:       at restoreOne (/opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/node_modules/@netlify/cache-utils/src/main.js:34:40)
9:21:02 AM:       at /opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/node_modules/@netlify/cache-utils/src/main.js:81:55
9:21:02 AM:       at Array.map (<anonymous>)
9:21:02 AM:       at allowMany (/opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/node_modules/@netlify/cache-utils/src/main.js:81:43)
9:21:02 AM:       at Object.restore (/opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/node_modules/@netlify/cache-utils/src/main.js:94:32)
9:21:02 AM:       at onPreBuild (/opt/build/repo/node_modules/netlify-plugin-gatsby-cache/index.js:14:29)
9:21:02 AM:       at Object.run (/opt/buildhome/.nvm/versions/node/v12.16.1/lib/node_modules/@netlify/build/src/plugins/child/main.js:85:9)

So I tried adding

[build]
  publish = "public"

and now it works. So regarding your question

When no "Publish directory" is set, is there any specific files or directories that we could cache instead to avoid the issues described above?

I think ./public and ./.cache would be good defaults.

[jlengstorf]

@janosh I don't think the site can deploy unless publish is set to public, though, right? because otherwise you'd be deploying all the source files? let me know if your setup was different

[janosh]

You can set the publish directory either from Netlify's deploy settings UI or via netlify.toml. I had been using the former which, I think, this plugin can't easily access.

[ehmicky]

Yes @jlengstorf is correct. netlify-plugin-gatsby-cache assumes that the publish directory is set to public.

This is done automatically by Netlify using deploy settings UI when you start a new Site and Gatsby is detected. It can also be done manually using build.publish in your netlify.toml as you mention @janosh.

Build plugins do have access to the publish directory using constants.PUBLISH_DIR, which is what this plugin is using.

Based on this, the error message you saw above indicates that your publish directory was not set to public which is a requirement for Netlify to deploy it correctly.