Setup in monorepo (`gatsby` dependency is required?)

[stephanschubert]

I have a monorepo set up with lerna (which shouldn't matter) and moved all (dev)dependencies into the root package.json. But it is required that gatsby is added to the package's own package.json.

I'm not running npm i inside the (sub-)package—I just added it.

Are there any (legacy?) reasons why this check is required or could it be removed/replaced with a more flexible one?

[pieh]

If package (site in case) uses dependency, then it should declare that dependency. If you want to avoid manually installing dependencies in each individual package please take a look at lerna bootstrap that will handle that. Another useful thing to consider is using yarn workspaces.

Gatsby doesn't really require anything, so there is no other way to determine if current working directory is actually root of gatsby project - if we would remove that check you could run gatsby build/develop anywhere and it would probably build empty site which is definitely not great DX

[stephanschubert]

I don't use lerna bootstrap nor yarn workspaces for different reasons. (FYI: Moving all deps into root package.json allows me to just run npm i packages/* to setup everything up.)

But back to topic: The current check does not ensure the current directory is actually a valid (as in buildable) Gatsby project. A lot of things can be wrong or missing in any case..?

[sidharthachatterjee]

Like @pieh mentioned, Gatsby doesn't really require anything so the current check for gatsby as a dependency or dev dependency makes sense because it is a pretty strong indicator that the current folder is a Gatsby project if it depends on it.

If I understand correctly, you want to avoid manually installing dependencies for every individual package (for which you're using npm i packages/*)

lerna bootstrap does the same thing:

npm install all external dependencies of each package. Symlink together all Lerna packages that are dependencies of each other. npm run prepublish in all bootstrapped packages. npm run prepare in all bootstrapped packages.

On a separate note, it is a good practice for individual packages to list their own dependencies so that they are self contained and can work in isolation.

I'm closing this issue for now but please feel free to reopen if there is anything else.

[stephanschubert]

@sidharthachatterjee Yes, that's what I want(ed) to avoid. Just for the record: "my" approach is a suggestion from lerna maintainers.

Besides that moving all deps into one root package.json avoids issues with different versions. Personally I take the tradeoff of having to put the deps back when publishing.

Anyway, I can live with this check but wanted to raise the question whether it's still useful.

[pieh]

You can also skip using gatsby-cli and call build/develop commands directly with some small custom wrapper: here's entry point for build https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/commands/build.js - you can write very small js file to setup BuildArgs (that normally are setup by cli)

I do think that tradeoff for not checking local package.json is not worth supporting this pretty niche use case - that's the only sanity check we can do :/

[sidharthachatterjee]

Thank you for bringing it up anyway, @stephanschubert

If there's more interest in the usecase, we'll relook at this in the future. In the mean time, happy building 🙂

[stephanschubert]

@pieh Thanks for the hint, I will look into that.

[josefaidt]

Hi all, after much trial and error getting a similar setup running I noticed without declaring gatsby in the root package.json file gatsby sites in a monorepo will not run - failing resolving modules such as gatsby/graphql and gatsby-plugin-react-helmet.

I have a whole issue typed up (see below) but figured I could tack on to this issue. I did not see this note in the documentation, and was wondering if it could be added somewhere?

My issue summary

## Description When using a monorepo with lerna and Yarn workspaces I am attempting to set up multiple, different Gatsby sites under a `./starters` folder (from scratch *or* using the cli), and they will all use a shared component library under `./packages` folder. The issue seems to be affecting Gatsby itself and how it resolves hoisted packages. I am able to use my existing React stack setup in the same monorepo without issue. Gatsby site fails with and without `react` and `react-dom` in root `package.json` [SKIP TO SOLUTION](###SOLUTION) ### Steps to reproduce - Setting up gatsby from scratch 1. `mkdir starters/gatsby` 2. `cd starters/gatsby` 3. init blank project: `yarn -y init` 4. install my dependencies: `yarn workspace gatsby add gatsby gatsby-image gatsby-plugin-manifest gatsby-plugin-offline gatsby-plugin-react-helmet gatsby-plugin-sharp gatsby-plugin-styled-components gatsby-source-filesystem gatsby-transformer-sharp prop-types styled-components react react-dom react-helmet` 5. set up build and develop scripts 6. Run `start` script. Everything runs great on the first run with 0 files added (no `gatsby-*` files, no `src` directory). 7. Quit running script, add `gatsby-config.js` and `src` directory from default starter 8. Run `develop` script. Fails. - Setting up gatsby site via cli 1. `cd starters` 2. `gatsby new new-site` 3. Change package.json name to "gatsby" and run: `cd .. && yarn workspace gatsby run develop`, site runs fine 4. add my component library from `packages/` 5. `lerna clean && lerna bootstrap` 6. Run the site again using `yarn workspace gatsby run develop` and it fails finding the module `gatsby/graphql` #### Scripts ```json { "scripts": { "start": "yarn develop", "develop": "gatsby develop --verbose", "build": "gatsby build" } } ``` ### Expected result Running `yarn workspace gatsby run develop` runs the site as intended. Sometimes this actually works after freshly installing new gatsby site to my `starters` folder. I can quit, bootstrap my dependencies and rerun the same command to expect the failure describe below (without gatsby as a dependency in the root package.json). ### Actual result Running `yarn workspace gatsby run develop` Gatsby fails on resolving *only* one of three packages: - `gatsby/graphql` (most common) - `gatsby-plugin-react-helmet` - `gatsby-source-filesystem` Specifically: #### Stack Trace for `gatsby-plugin-react-helmet` ``` verbose 0.343 set gatsby_log_level: "verbose" verbose 0.346 set gatsby_executing_command: "develop" verbose 0.346 loading local command from: /Users/{USER}/Documents/{ROOT_PROJECT_FOLDER}/node_modules/gatsby/dist/commands/develop.js verbose 1.584 running command: develop success open and validate gatsby-configs — 0.009 s error Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you need to install its package? Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n eed to install its package? - load.js:102 resolvePlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11 - load.js:114 processPlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20 - load.js:164 config.plugins.forEach.plugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20 - Array.forEach - load.js:163 module.exports [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20 - index.js:56 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21 - Generator.next - debuggability.js:313 Promise._execute [widget-library]/[bluebird]/js/release/debuggability.js:313:9 - promise.js:483 Promise._resolveFromExecutor [widget-library]/[bluebird]/js/release/promise.js:483:18 - promise.js:79 new Promise [widget-library]/[bluebird]/js/release/promise.js:79:10 - index.js:96 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17 - index.js:124 [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36 error UNHANDLED REJECTION Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n eed to install its package? - load.js:102 resolvePlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11 - load.js:114 processPlugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20 - load.js:164 config.plugins.forEach.plugin [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20 - Array.forEach - load.js:163 module.exports [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20 - index.js:56 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21 - Generator.next - debuggability.js:313 Promise._execute [widget-library]/[bluebird]/js/release/debuggability.js:313:9 - promise.js:483 Promise._resolveFromExecutor [widget-library]/[bluebird]/js/release/promise.js:483:18 - promise.js:79 new Promise [widget-library]/[bluebird]/js/release/promise.js:79:10 - index.js:96 [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17 - index.js:124 [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36 error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed. Exit code: 1 ``` #### Stack Trace for `gatsby/graphl` ``` yarn workspace v1.13.0 yarn run v1.13.0 $ gatsby develop gatsby develop Start development server. Watches files, rebuilds, and hot reloads if something changes Options: --verbose Turn on verbose output [boolean] [default: false] --no-color Turn off the color in output [boolean] [default: false] -H, --host Set host. Defaults to localhost [string] [default: "localhost"] -p, --port Set port. Defaults to 8000 [string] [default: "8000"] -o, --open Open the site in your (default) browser for you. [boolean] -S, --https Use HTTPS. See https://www.gatsbyjs.org/docs/local-https/ as a guide [boolean] -c, --cert-file Custom HTTPS cert file (relative path; also required: --https, --key-file). See https://www.gatsbyjs.org/docs/local-https/ [string] [default: ""] -k, --key-file Custom HTTPS key file (relative path; also required: --https, --cert-file). See https://www.gatsbyjs.org/docs/local-https/ [string] [default: ""] --open-tracing-config-file Tracer configuration file (open tracing compatible). See https://www.gatsbyjs.org/docs/performance-tr acing/ [string] -h, --help Show help [boolean] -v, --version Show version number [boolean] error There was a problem loading the local develop command. Gatsby may not be installed. Perhaps you need to run "npm install"? Error: Cannot find module 'gatsby/graphql' - loader.js:611 Function.Module._resolveFilename internal/modules/cjs/loader.js:611:15 - loader.js:537 Function.Module._load internal/modules/cjs/loader.js:537:25 - loader.js:665 Module.require internal/modules/cjs/loader.js:665:17 - v8-compile-cache.js:159 require [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20 - index.js:8 Object. [widget-library]/[babel-plugin-remove-graphql-queries]/index.js:8:17 - v8-compile-cache.js:178 Module._compile [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30 - loader.js:747 Object.Module._extensions..js internal/modules/cjs/loader.js:747:10 - loader.js:628 Module.load internal/modules/cjs/loader.js:628:32 - loader.js:568 tryModuleLoad internal/modules/cjs/loader.js:568:12 - loader.js:560 Function.Module._load internal/modules/cjs/loader.js:560:3 - loader.js:665 Module.require internal/modules/cjs/loader.js:665:17 - v8-compile-cache.js:159 require [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20 - file-parser.js:21 Object. [gatsby]/[gatsby]/dist/internal-plugins/query-runner/file-parser.js:21 :23 - v8-compile-cache.js:178 Module._compile [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30 - loader.js:747 Object.Module._extensions..js internal/modules/cjs/loader.js:747:10 - loader.js:628 Module.load internal/modules/cjs/loader.js:628:32 error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed. Exit code: 1 Command: /Users/josef/.nvm/versions/node/v11.8.0/bin/node Arguments: /usr/local/Cellar/yarn/1.13.0/libexec/lib/cli.js run develop Directory: /Users/josef/Documents/playground/widget-library/starters/gatsby Output: info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command. ``` ### Environment ``` System: OS: macOS 10.14.3 CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz Shell: 2.7.1 - /usr/local/bin/fish Binaries: Node: 11.8.0 - ~/.nvm/versions/node/v11.8.0/bin/node Yarn: 1.13.0 - /usr/local/bin/yarn npm: 6.5.0 - ~/.nvm/versions/node/v11.8.0/bin/npm Languages: Python: 2.7.10 - /usr/bin/python Browsers: Chrome: 72.0.3626.96 Firefox: 64.0 Safari: 12.0.3 npmPackages: gatsby: ^2.1.0 => 2.1.0 ``` ### SOLUTION After much testing and trial/error, I've found the monorepo *needs* gatsby declared in the root `package.json` - Once added, run `lerna clean && lerna bootstrap` - Then run the site, `yarn workspace gatsby run develop`