Gatsby fails resolving node_modules in monorepo

josefaidt commented 5 years ago

Description

When using a monorepo with lerna and Yarn workspaces I am attempting to set up multiple, different Gatsby sites under a ./starters folder (from scratch or using the cli), and they will all use a shared component library under ./packages folder. The issue seems to be affecting Gatsby itself and how it resolves packages.

I am able to use my existing React stack setup in the same monorepo without issue. Gatsby site fails with and without react and react-dom in root package.json

SKIP TO SOLUTION for using base Gatsby in a monorepo

Steps to reproduce

Setting up gatsby from scratch
1. mkdir starters/gatsby
2. cd starters/gatsby
3. init blank project: yarn -y init
4. install my dependencies: yarn workspace gatsby add gatsby gatsby-image gatsby-plugin-manifest gatsby-plugin-offline gatsby-plugin-react-helmet gatsby-plugin-sharp gatsby-plugin-styled-components gatsby-source-filesystem gatsby-transformer-sharp prop-types styled-components react react-dom react-helmet
5. set up build and develop scripts
6. Run start script. Everything runs great on the first run with 0 files added (no gatsby-* files, no src directory).
7. Quit running script, add gatsby-config.js and src directory from default starter
8. Run develop script. Fails.
Setting up gatsby site via cli
1. cd starters
2. gatsby new new-site
3. Change package.json name to "gatsby" and run: cd .. && yarn workspace gatsby run develop, site runs fine
4. add my component library from packages/
5. lerna clean && lerna bootstrap
6. Run the site again using yarn workspace gatsby run develop and it fails finding the module gatsby/graphql
After site is built, functional, and all package dependencies have been bootstrapped
1. add a new plugin, yarn workspace gatsby add gatsby-transformer-json
2. modify the gatsby-config.js file as necessary
3. clear the site's cache directory
4. Run the site using yarn workspace gatsby run develop
5. Site fails to resolve new module (see stack trace below)

Scripts

{
  "scripts": {
    "start": "yarn develop",
    "develop": "gatsby develop --verbose",
    "build": "gatsby build"
  }
}

Expected result

Running yarn workspace gatsby run develop runs the site as intended. Sometimes this actually works after freshly installing new gatsby site to my starters folder. I can quit, bootstrap my dependencies and rerun the same command to expect the failure describe below (without gatsby as a dependency in the root package.json).

Actual result

Running yarn workspace gatsby run develop Gatsby fails on resolving only one of three packages:

gatsby/graphql (most common)
gatsby-plugin-react-helmet
gatsby-source-filesystem
gatsby-transformer-json

Stack Traces

Stack Trace for `gatsby-plugin-react-helmet`

``` verbose 0.343 set gatsby_log_level: "verbose" verbose 0.346 set gatsby_executing_command: "develop" verbose 0.346 loading local command from: /Users/josef/Documents/playground/{ROOT_PROJECT_FOLDER}/node_modules/gatsby/dist/commands/develop.js verbose 1.584 running command: develop success open and validate gatsby-configs — 0.009 s error Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you need to install its package? Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n eed to install its package? - load.js:102 resolvePlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11 - load.js:114 processPlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20 - load.js:164 config.plugins.forEach.plugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20 - Array.forEach - load.js:163 module.exports [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20 - index.js:56 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21 - Generator.next - debuggability.js:313 Promise._execute [widget-library]/[bluebird]/js/release/debuggability.js:313:9 - promise.js:483 Promise._resolveFromExecutor [widget-library]/[bluebird]/js/release/promise.js:483:18 - promise.js:79 new Promise [widget-library]/[bluebird]/js/release/promise.js:79:10 - index.js:96 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17 - index.js:124 [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36 error UNHANDLED REJECTION Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n eed to install its package? - load.js:102 resolvePlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11 - load.js:114 processPlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20 - load.js:164 config.plugins.forEach.plugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20 - Array.forEach - load.js:163 module.exports [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20 - index.js:56 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21 - Generator.next - debuggability.js:313 Promise._execute [widget-library]/[bluebird]/js/release/debuggability.js:313:9 - promise.js:483 Promise._resolveFromExecutor [widget-library]/[bluebird]/js/release/promise.js:483:18 - promise.js:79 new Promise [widget-library]/[bluebird]/js/release/promise.js:79:10 - index.js:96 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17 - index.js:124 [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36 error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed. Exit code: 1 ```

Stack Trace for `gatsby/graphl`

``` yarn workspace v1.13.0 yarn run v1.13.0 $ gatsby develop gatsby develop Start development server. Watches files, rebuilds, and hot reloads if something changes Options: --verbose Turn on verbose output [boolean] [default: false] --no-color Turn off the color in output [boolean] [default: false] -H, --host Set host. Defaults to localhost [string] [default: "localhost"] -p, --port Set port. Defaults to 8000 [string] [default: "8000"] -o, --open Open the site in your (default) browser for you. [boolean] -S, --https Use HTTPS. See https://www.gatsbyjs.org/docs/local-https/ as a guide [boolean] -c, --cert-file Custom HTTPS cert file (relative path; also required: --https, --key-file). See https://www.gatsbyjs.org/docs/local-https/ [string] [default: ""] -k, --key-file Custom HTTPS key file (relative path; also required: --https, --cert-file). See https://www.gatsbyjs.org/docs/local-https/ [string] [default: ""] --open-tracing-config-file Tracer configuration file (open tracing compatible). See https://www.gatsbyjs.org/docs/performance-tr acing/ [string] -h, --help Show help [boolean] -v, --version Show version number [boolean] error There was a problem loading the local develop command. Gatsby may not be installed. Perhaps you need to run "npm install"? Error: Cannot find module 'gatsby/graphql' - loader.js:611 Function.Module._resolveFilename internal/modules/cjs/loader.js:611:15 - loader.js:537 Function.Module._load internal/modules/cjs/loader.js:537:25 - loader.js:665 Module.require internal/modules/cjs/loader.js:665:17 - v8-compile-cache.js:159 require [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20 - index.js:8 Object. [widget-library]/[babel-plugin-remove-graphql-queries]/index.js:8:17 - v8-compile-cache.js:178 Module._compile [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30 - loader.js:747 Object.Module._extensions..js internal/modules/cjs/loader.js:747:10 - loader.js:628 Module.load internal/modules/cjs/loader.js:628:32 - loader.js:568 tryModuleLoad internal/modules/cjs/loader.js:568:12 - loader.js:560 Function.Module._load internal/modules/cjs/loader.js:560:3 - loader.js:665 Module.require internal/modules/cjs/loader.js:665:17 - v8-compile-cache.js:159 require [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20 - file-parser.js:21 Object. [gatsby]/[gatsby]/dist/internal-plugins/query-runner/file-parser.js:21 :23 - v8-compile-cache.js:178 Module._compile [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30 - loader.js:747 Object.Module._extensions..js internal/modules/cjs/loader.js:747:10 - loader.js:628 Module.load internal/modules/cjs/loader.js:628:32 error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed. Exit code: 1 Command: /Users/josef/.nvm/versions/node/v11.8.0/bin/node Arguments: /usr/local/Cellar/yarn/1.13.0/libexec/lib/cli.js run develop Directory: /Users/josef/Documents/playground/{PROJECT}/starters/gatsby Output: info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command. ```

Stack trace for `gatsby-transformer-json`

``` yarn workspace v1.13.0 yarn run v1.13.0 $ yarn develop $ gatsby develop success open and validate gatsby-configs — 0.008 s error Unable to find plugin "gatsby-transformer-json". Perhaps you need to install its package? Error: Unable to find plugin "gatsby-transformer-json". Perhaps you need to install its package? - load.js:102 resolvePlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11 - load.js:114 processPlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20 - load.js:164 config.plugins.forEach.plugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20 - Array.forEach - load.js:163 module.exports [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20 - index.js:56 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21 - Generator.next - debuggability.js:313 Promise._execute [widget-library]/[bluebird]/js/release/debuggability.js:313:9 - promise.js:483 Promise._resolveFromExecutor [widget-library]/[bluebird]/js/release/promise.js:483:18 - promise.js:79 new Promise [widget-library]/[bluebird]/js/release/promise.js:79:10 - index.js:96 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17 - index.js:124 [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36 error UNHANDLED REJECTION Error: Unable to find plugin "gatsby-transformer-json". Perhaps you need to install its package? - load.js:102 resolvePlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11 - load.js:114 processPlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20 - load.js:164 config.plugins.forEach.plugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20 - Array.forEach - load.js:163 module.exports [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20 - index.js:56 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21 - Generator.next - debuggability.js:313 Promise._execute [widget-library]/[bluebird]/js/release/debuggability.js:313:9 - promise.js:483 Promise._resolveFromExecutor [widget-library]/[bluebird]/js/release/promise.js:483:18 - promise.js:79 new Promise [widget-library]/[bluebird]/js/release/promise.js:79:10 - index.js:96 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17 - index.js:124 [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36 error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed. Exit code: 1 Command: /Users/josef/.nvm/versions/node/v11.8.0/bin/node Arguments: /usr/local/Cellar/yarn/1.13.0/libexec/lib/cli.js run start Directory: /Users/josef/Documents/playground/{PROJECT}/starters/gatsby Output: info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command. ```

Environment

  System:
    OS: macOS 10.14.3
    CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
    Shell: 2.7.1 - /usr/local/bin/fish
  Binaries:
    Node: 11.8.0 - ~/.nvm/versions/node/v11.8.0/bin/node
    Yarn: 1.13.0 - /usr/local/bin/yarn
    npm: 6.5.0 - ~/.nvm/versions/node/v11.8.0/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 72.0.3626.96
    Firefox: 64.0
    Safari: 12.0.3
  npmPackages:
    gatsby: ^2.1.0 => 2.1.0

SOLUTION (well, was)

After much testing and trial/error, I've found the monorepo needs gatsby declared in the root package.json

Once added, run lerna clean && lerna bootstrap
Then run the site, yarn workspace gatsby run develop

I typed this up and continued working thinking it may be my environment, however after adding another plugin the site began to break and show similar symptoms to those described above.

josefaidt commented 5 years ago

Actually this may be my environment. When running yarn workspace gatsby add gatsby-transformer-json the dependency gets added to the correct package.json but the dependency does not actually get installed.

Does anyone have any insight to why this may be happening?

josefaidt commented 5 years ago

I've exhausted myself trying to figure this one out. I don't think this has to deal directly with Gatsby or its package, but I do think it's odd that I'm having issues with only a handful of plugins. The bulk of the issue has been resolve by adding gatsby as a dependency in the project root (not sure why).

The plugins I am having issues with:

gatsby-transformer-json
gatsby-transformer-remark

I'm going to leave this issue open for now in case anyone has any insight

pieh commented 5 years ago

This is very weird, because folks working on some features use yarn workspaces quite often and didn't seem to hit problems with it. It seems like workspaces isn't correctly installing deps and setting symlinks? Maybe issue lies with yarn itself not correctly setting this up? Did you try updating/downgrading it? Also did you try to verify how node_modules in site itself look like - do they have those packages installed / symlinked?

josefaidt commented 5 years ago

I agree this is very odd, and I thought I set everything up to the spec. I'm removing lerna from the equation to see if the issue persists. The dependencies weren't being installed at all. They are added to the root's lockfile and the package's package.json file, but the package was nowhere to be found in any node_modules folders.

I'm going to play around with this some more after learning about the nohoist workspace option.

josefaidt commented 5 years ago

@pieh

Ok, I ended up getting this working. Using @jedrichards snippet from his monorepo example:

"workspaces": {
    "nohoist": [
      "gatsby"
    ]
  }

I didn't know the nohoist option existed as it is not mentioned on the official yarn documentation but in a yarn blog post. BUT this didn't work for me given my directory structure:

.
├── README.md
├── components
│   ├── package.json
│   └── src
│       ├── SharedComponent
│       └── index.js
├── package.json
├── sites
│   ├── gatsby-app
│   │   ├── LICENSE
│   │   ├── README.md
│   │   ├── gatsby-browser.js
│   │   ├── gatsby-config.js
│   │   ├── gatsby-node.js
│   │   ├── gatsby-ssr.js
│   │   ├── node_modules
│   │   ├── package.json
│   │   ├── public
│   │   └── src
│   └── gatsby-app2
│       ├── LICENSE
│       ├── README.md
│       ├── gatsby-browser.js
│       ├── gatsby-config.js
│       ├── gatsby-node.js
│       ├── gatsby-ssr.js
│       ├── node_modules
│       ├── package.json
│       └── public
└── yarn.lock

The trick was to use a globbing pattern in the nohoist array like so,

"workspaces": {
    "nohoist": [
      "**/gatsby",
      "gatsby/**",
      "**/gatsby-*",
      "gatsby-*/**",
      "**/*-gatsby",
      "*-gatsby/**"
    ]
  }

I feel like this definitely needs to be mentioned in the Gatsby docs.

DSchau commented 5 years ago

@josefaidt I'm still kind of unsure why there's an issue there. That nohoist looks pretty gnarly. I think in this case it probably makes sense to investigate what exactly breaks with that particular directory structure rather than documenting this scenario--which seems a bit niche-y and may just lead to confusion.

For instance, I structured a recent repository I created as a monorepo with Yarn workspaces, and I didn't really have any issues whatsoever with resolving Gatsby. https://github.com/DSchau/gatsby-drupal-workshop

Is there an easily discernible difference or something we can fix here? If at all possible, sharing a repository, rather than documenting its structure, would be super helpful!

josefaidt commented 5 years ago

At first I thought this was an issue with Gatsby and how it resolves modules as I did not have an issue with a pure React build. That is why I raised the issue, however I now know that it most likely is not a Gatsby-specific issue, but rather a "quirk", if you will, to using it this way.

I took a look at your repo and I'm honestly surprised you haven't had any issues without any hoisting. But what was weird was that originally my issue was affecting specific gatsby packages.

I'm still not quite sure what the root cause of the issue is, and unfortunately I cannot share the original repo but I made a proof of concept: https://github.com/josefaidt/monorepo-gatsby

Considering this issue is resolved and in the end not entirely relevant to Gatsby I am closing it.

jedrichards commented 5 years ago

@josefaidt There were a few tiny errors in your proof of concept above. Check it out: https://github.com/josefaidt/monorepo-gatsby/pull/1

carllippert commented 4 years ago

@josefaidt did this proof of concept work? I am trying to get a lerna monorepo set up and am running into about a days worth of issues seemingly related to staticquery's not being picked up if they are not in main src folder. In demo i see you had all your quries in site folder but was wondering if you went further with this?

josefaidt commented 4 years ago

@carllippert yes, however I use yarn without lerna now. My personal site uses a monorepo with a theme, demo site, and app site. In app/content/pages/resume.mdx you can see I'm using a page query outside of the source folder

carllippert commented 4 years ago

@josefaidt did removing lerna aid in this? I would have to do a lot of digging but i think when you use the page-creator plugin it then can pickup pageQueries but yet basically just scans src folder for StaticQuery and useStaticQueryHooks thus killing any components say in a shared ui package. I use page creator to generate pages in a dif package and that seemingly works its just the underlying components on the page. Anyways thanks for giving me something to look over for inspiration. 👍

timbakkum commented 4 years ago

I had a similar issue in a monorepo with a gatsby project in a seperate package: storybook with a custom webpack config using file-loader in one package and gatsby project in another package. Gatsby requires v 1.1.11 of file-loader while the other packages use v 3 or higher. Yarn put 1 version of file-loader in the storybook package's node_modules but did not put/hoist a version in the global node_modules, therefor the gatsby package could not find file-loader.

Using the fix described by @josefaidt gatsby is able to resolve file-loader again (when importing local fonts in my case). I don't know if this issue lies with yarn workspaces or with the way gatsby dependencies are listed/required, but hopefully this sheds some more light on it for somebody.

gatsbyjs / gatsby