[fsevents bug] Stuck at "source and transform nodes" / "createPagesStatefully" on MacOS

[ehowey]

Description

I am working on a theme, local build was working fine with no problems, and recently upgraded all the dependencies to Gatsby 2.14.0 and both gatsby develop and gatsby build hang at source and transform nodes in my local dev environment.

Interestingly it works and builds on Netlify. This would point to it being something on my system. I have deleted the node modules folders in the workspaces and the root workspace folder and done a fresh yarn command. I also deleted the yarn.lock and package.lock files...not sure if this would cause problems.

Steps to reproduce

Theme repo is here: Gatsby-Theme-Catalyst-Core

Starter repo is here: Gatsby-Starter-Catalyst-Core

I have this setup in a yarn workspace for development however the same issue occurs if you do a fresh install of the starter using gatsby new my-catalyst-starter-core https://github.com/ehowey/gatsby-starter-catalyst-core

Expected result

Build successfully

Actual result

yarn workspace v1.17.3 yarn run v1.17.3 $ gatsby develop success open and validate gatsby-configs - 0.122 s success load plugins - 1.964 s success onPreInit - 0.073 s success initialize cache - 0.056 s success copy gatsby files - 0.242 s success onPreBootstrap - 0.087 s ⠙ source and transform nodes

Environment

System: OS: macOS High Sierra 10.13.6 CPU: (2) x64 Intel(R) Core(TM)2 Duo CPU P8600 @ 2.40GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 12.9.1 - /usr/local/bin/node Yarn: 1.17.3 - /usr/local/bin/yarn npm: 6.11.2 - /usr/local/bin/npm Languages: Python: 2.7.16 - /usr/local/bin/python Browsers: Chrome: 76.0.3809.100 Firefox: 67.0.2 Safari: 12.1.2 npmGlobalPackages: gatsby-cli: 2.7.40

[ehowey]

Updated - I tested commenting out the gatsby-node.js file and the build progressed on one occasion, but then I tried again and doing this and it got stuck in the same spot again.

Is there any chance this is due to my ancient computer, 2010 Macbook Pro (upgraded to 8GB Ram and an SSD), hasn't stopped me yet but I know its days are limited. This is a hobby for me and I have young kids so the dollars haven't been there to spend on a new computer since this one has worked fine with the upgraded Ram and SSD.

[ehowey]

I have tried to revert back to gatsby 2.13.52 which was the last version I was stable on, I have also tried reverting to react 16.8.6.

Interestingly I had it build successfully one time when I reverted react to 16.8.6 but after that first time it hung up at the same spot which is really odd behaviour to me.

[ehowey]

I can also, rarely, for no reason I can discern get it to build fine. There does not seem to be a rhyme or reason for when this happens. I spent a couple of hours looking for patterns or errors that might be causing this but didn't find anything. I reviewed https://github.com/gatsbyjs/gatsby/issues/6654 and tried some of the items in there to no avail.

[ehowey]

This is got to me one of the most odd bugs I have ever encountered because the behaviour seems to change without there being discernible change in the code. In some cases the build hangs at source and transform nodes(60% of time), in some cases it hangs at Create Pages Statefully(20%), in some cases it builds successfully (20%). I have had it display all three of these behaviours without changing a line of code.

I have replicated this behaviour in gatsby-starter-blog as well, which is really odd. Again it makes me think this is an issue on my end locally. Interestingly it only hangs at createPagesStatefully.

I am now starting to think that it has something to do with a library that got updated by homebrew automatically - which one I have no idea but it is the only thing I can think of which I have not tested.

Here is a list of the things I have tried so far:

• Changing node versions, tried 12, 10 and 8

• Reverting to a previous point in my commit history that was working - it still fails now.

• Commenting out plugins and areas of the gatsby-config

• Commenting out the contents of my gatsby-node.js

• Test it again on Netlify, still built successfully, which is what makes me think it must have something to do with my computer.

• Deleting the 4 pages in my src/pages directory and putting in a very basic index.mdx file

• Deleting all the node_module and lock files, reinstalling

• Restarting computer

• Creating a fresh yarn workspace with new clones of the theme/starter from Github.

• Testing gatsby-starter-blog, similar behaviour but it hangs on createPagesStatefully

[steinitz]

Hello,

I can do little but confirm that I'm seeing this issue on a number of gatsby starters. And, as you point out, it sometimes just decides to build or to stop building, hanging at either "source and transform nodes" or createPagesStatefully.

Quite frustrating and leading to trying the most outrageous attempts to fix it.

[georgiee]

I did not witness this problem but this sounds awkward and I would really like to know the reason for this behaviour on your side.

Preparation You should try to use the node debugger to get to the root of the problem. A task in your package.json would look like this. If you are using VSCode you can enable "Auto Attach" to use the internal debugger together with this task (make sure you use the internal terminal for this purpose)

"start:debug": "node --inspect=127.0.0.1:9232 node_modules/.bin/gatsby develop",

Debugging will of course work with any IDE just make sure you attach your debugger correctly.

1. Variant: Minimal Reproductiion You have mentioned createPagesStatefully as the source of the problem. If you are sure that's the source of the problem maybe you could create a very tiny project to reproduce it. Ditch all the starters, just use gatsby directly and use createPagesStatefully through gatsby-node.js with some examples that mimic the things your starters are doing. Then start gatsby and debug it through node inspect.

That way you might find where it hangs.

2. Alternative: Inside your project/starter You could of course try to debug the problem inside your project with the same technique. But maybe you have multiple problems that are stacking and blur your view on the cause of problems. So I would always try to get a minimal reproduction before starting to debug.

Good luck.

[ehowey]

So...I ran into some weird behaviour with the lock files. Maybe someone who knows more could point me in the right direction. In the process of trying to get to a minimal working build I basically stripped down to a "hello world" Gatsby install and it still wasn't working which was really odd.

Where it got even more strange was that the actual gatsby-starter-hello-world does work and build fine on my computer. BUT...if I delete the lock file and get yarn to rebuild a new lock file then the build starts failing, hanging at source and transform nodes. I could get my stripped down and minimal implementation to build if I copied the lock file from "hello-world" and used this. So my current theory is that there is some kind of version problem in my lock files that is causing this issue but I am stuck here.

I also deleted all of my homebrew installs and reinstalled node, yarn, git, etc. Just to make sure it wasn't somekind of funny business there.

[rheinardkorf]

Thanks @ehowey for raising this.... I thought it was just me because it is quite intermittent (but happens more than 50% of the time). Did the same as you to try and debug the situation. When I hang on ⠙ source and transform nodes it usually means having to kill my terminal.

If I find something I'll let you know. And I'll watch this thread too.

[steinitz]

@georgiee - thank you for the --inspect info. I'm going to see if I can get node debugging working with WebStorm.

I also like your idea of a minimal reproduction. But that will take time while I understand Gatsby a little deeper.

Currently it's hanging on "source and transform nodes". Rarely, it makes it to createPagesStatefully and hangs there. Otherwise it builds normally.

[hbthen3rd]

I am currently facing the same problem after doing "yarn upgrade" to fix a vulnerable dependency. Here's my set up:

System: OS: macOS 10.15 CPU: (4) x64 Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz Shell: 5.7.1 - /bin/zsh Binaries: Node: 12.8.1 - /usr/local/bin/node Yarn: 1.17.3 - /usr/local/bin/yarn npm: 6.10.3 - /usr/local/bin/npm Languages: Python: 2.7.16 - /usr/local/bin/python Browsers: Chrome: 76.0.3809.132 Firefox: 68.0.2 Safari: 13.0 npmPackages: gatsby: ^2.13.42 => 2.14.7 gatsby-background-image: ^0.8.3 => 0.8.6 gatsby-image: ^2.2.7 => 2.2.15 gatsby-plugin-manifest: ^2.2.4 => 2.2.10 gatsby-plugin-netlify: ^2.1.4 => 2.1.10 gatsby-plugin-netlify-cms: ^4.1.6 => 4.1.12 gatsby-plugin-offline: ^2.2.5 => 2.2.10 gatsby-plugin-react-helmet: ^3.1.2 => 3.1.5 gatsby-plugin-react-svg: ^2.1.1 => 2.1.2 gatsby-plugin-sass: ^2.1.3 => 2.1.12 gatsby-plugin-sharp: ^2.2.9 => 2.2.18 gatsby-plugin-typography: ^2.3.2 => 2.3.5 gatsby-plugin-webfonts: ^1.1.0 => 1.1.0 gatsby-remark-images: ^3.1.10 => 3.1.19 gatsby-remark-relative-images-v2: ^0.1.5 => 0.1.5 gatsby-remark-responsive-iframe: ^2.2.5 => 2.2.10 gatsby-source-filesystem: ^2.1.6 => 2.1.18 gatsby-transformer-remark: ^2.6.12 => 2.6.19 gatsby-transformer-sharp: ^2.2.5 => 2.2.12

[hbthen3rd]

I am currently facing the same problem after doing "yarn upgrade" to fix a vulnerable dependency. Here's my set up:

Update: I manage to fix things by restoring my old "yarn.lock" file.

[steinitz]

@hbthen3rd

Update: I manage to fix things by restoring my old "yarn.lock" file.

My experience is that the problem can go away for no clear reason only to return later. You may find the problem returning despite restoring yarn.lock. Keep us posted.

[ehowey]

Here is a minimal reproduction using gatsby-starter-hello-world:

https://github.com/ehowey/gatsby-test-lockfiles

The current lock file included in the repo is the one failing for me. I have also included copies in the repo of yarn-working.lock and yarn-notworking.lock. Hopefully that makes it easier for someone to troubleshoot.

My current environment:

System: OS: macOS High Sierra 10.13.6 CPU: (2) x64 Intel(R) Core(TM)2 Duo CPU P8600 @ 2.40GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 10.16.3 - /usr/local/bin/node Yarn: 1.17.3 - /usr/local/bin/yarn npm: 6.10.3 - /usr/local/bin/npm Languages: Python: 2.7.10 - /usr/bin/python Browsers: Chrome: 76.0.3809.132 Firefox: 67.0.2 Safari: 12.1.2 npmPackages: gatsby: ^2.13.73 => 2.15.0 npmGlobalPackages: gatsby-cli: 2.7.40

[sharvit]

I am experiencing the same issue.

Some direction I have found:

1. Downgrading gatsby-plugin-sitemap from 2.2.9 to 2.2.6 solved the issue with yarn develop.

   • It is odd because gatsby-plugin-sitemap should not affect development build.

2. yarn build still doesn't work but when I remove gatsby-plugin-sitemap form my config, yarn build works again.

[ehowey]

@sharvit I can report that this did not work in my case. Glad it fixed it for you, I do think it has something ultimately to do with the lock files and some weird version issue deep in the bowels of the lock file.

I have managed to get back to a working build, most of the time, maybe 8/10 times by reverting to an old lock file and doing some copy and paste. I can now also CTRL+C to force quit the build if it is hanging which I couldn't do at one point in this process. I don't know what fixes it in the lock file but I think the clues are in the repository I posted above with two copies of a lock file, one that works and one that doesn't. This is an odd beast of a bug. Usually in computer land things reassuringly work or don't work.

[ehowey]

@steinitz Any luck on your end? I had what you mention happen. It seemed to get better, pretty good but not perfect and now is failing almost every time again. Pretty frustrating.

[steinitz]

@ehowey

As you might imagine, due to the on-again off-again nature of this problem, I'm hesitant to report. Here's a case in point:

After deleting the node_modules directory, deleting yarn.lock then running npx yarn-upgrade-all and yarn install all was well.

Then, just now, in response to your message, I tried yarn start Result: it hangs again at source and transform nodes

I suppose there are two sensible things to do:

1. take @georgiee's advice to get Webstorm's debugging working with node --inspect and see if I can spot any obvious problems
2. set Gatsby aside for a while to see if some smart person solves the problem

[steinitz]

Update:

ctl-C'd the above stuck process (that didn't used to stop the stuck process which was doubly annoying).
Then yarn start got stuck on createPagesStatefully.
ctl-C'd again, another yarn start - stuck on source and transform nodes ctl-C'd one more time (for the hell of it) - this time yarn start worked and it's running

Don't know what to make of that, but there it is.

[ehowey]

This is similar to what I am seeing, tonight it seems worse maybe building successfully 1/10 times or less. From a programming/coding perspective this is super odd behaviour. Anecdotally it seemed to be working better a few days ago at 2.15.1 then today at 2.15.6. I also wonder what commonalities we all share that is causing this bug to happen. Can you run the gatsby info --clipboard command and post it?

It obviously isn't super widespread otherwise there would be a flood of reports but it also isn't just me like I originally thought. We all seem to be using yarn as well, which is a requirement for me as I am working on themes in a yarn workspace. I am still able to reproduce this on the gatsby-starter-hello-world so I believe it is some dependency issue or conflict in the core gatsby files.

[steinitz]

@ehowey here's the gatsby info you requested

System: OS: macOS 10.14.6 CPU: (4) x64 Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 12.9.1 - /usr/local/bin/node Yarn: 1.17.3 - /usr/local/bin/yarn npm: 6.11.2 - /usr/local/bin/npm Languages: Python: 2.7.16 - /usr/local/bin/python Browsers: Chrome: 76.0.3809.132 Firefox: 68.0.1 Safari: 12.1.2 npmPackages: gatsby: ^2.14.3 => 2.14.7 gatsby-image: ^2.2.14 => 2.2.15 gatsby-plugin-feed: ^2.3.9 => 2.3.9 gatsby-plugin-i18n: ^1.0.1 => 1.0.1 gatsby-plugin-less: ^3.0.2 => 3.0.4 gatsby-plugin-manifest: ^2.2.9 => 2.2.10 gatsby-plugin-offline: ^2.2.10 => 2.2.10 gatsby-plugin-react-helmet: ^3.1.5 => 3.1.5 gatsby-plugin-robots-txt: ^1.5.0 => 1.5.0 gatsby-plugin-sharp: ^2.2.18 => 2.2.18 gatsby-plugin-sitemap: ^2.2.9 => 2.2.9 gatsby-remark-images: ^3.1.19 => 3.1.19 gatsby-remark-prismjs: ^3.3.9 => 3.3.9 gatsby-source-filesystem: ^2.1.18 => 2.1.18 gatsby-transformer-remark: ^2.6.19 => 2.6.19 gatsby-transformer-sharp: ^2.2.12 => 2.2.12 npmGlobalPackages: gatsby-cli: 2.7.40

[goblindegook]

I was having the same issue: site built flawlessly on Netlify, but hung on my development machine with both gatsby build and gatsby develop.

After playing around with package versions, I noticed that reverting gatsby-plugin-sitemap from version 2.2.10 to 2.2.9 fixed the problem for me. Oddly enough, some people here seem to have issues with 2.2.9, which could mean the problem resides somewhere else.

Edit: Spoke too soon, Gatsby still hangs at the "source and transform nodes" and "createPagesStatefully" steps, though much less frequently.

[steinitz]

@goblindegook this seems a common pattern with this particular issue. Because the issue comes and goes, seemingly with the weather, the time of day, time since boot up... you can believe something you've done has fixed it - only to have it recur.

[domLocalHeroes]

Experiencing this issue too, downgraded gatsby-plugin-sitemap to 2.2.6 and that seems to have fixed it

[namukang]

FWIW, I'm also experiencing this but use neither yarn nor gatsby-plugin-sitemap.

My gatsby info in case it helps:

  System:
    OS: macOS 10.14.6
    CPU: (4) x64 Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 12.9.1 - /usr/local/bin/node
    npm: 6.11.2 - /usr/local/bin/npm
  Languages:
    Python: 3.7.4 - /usr/local/opt/python/libexec/bin/python
  Browsers:
    Chrome: 76.0.3809.132
    Firefox: 68.0.1
    Safari: 12.1.2
  npmPackages:
    gatsby: ^2.15.1 => 2.15.1 
    gatsby-cli: ^2.7.41 => 2.7.41 
    gatsby-image: ^2.2.15 => 2.2.15 
    gatsby-plugin-catch-links: ^2.1.6 => 2.1.6 
    gatsby-plugin-google-tagmanager: ^2.1.7 => 2.1.7 
    gatsby-plugin-manifest: ^2.2.12 => 2.2.12 
    gatsby-plugin-offline: ^3.0.0 => 3.0.0 
    gatsby-plugin-react-helmet: ^3.1.5 => 3.1.5 
    gatsby-plugin-sass: ^2.1.12 => 2.1.12 
    gatsby-plugin-sharp: ^2.2.18 => 2.2.18 
    gatsby-remark-images: ^3.1.19 => 3.1.19 
    gatsby-source-filesystem: ^2.1.18 => 2.1.18 
    gatsby-transformer-remark: ^2.6.19 => 2.6.19 
    gatsby-transformer-sharp: ^2.2.12 => 2.2.12 
    gatsby-transformer-yaml: ^2.2.7 => 2.2.7 
  npmGlobalPackages:
    gatsby-cli: 2.7.41

[karlhorky]

For me, cleaning the cache with gatsby clean solved the issue.

[ehowey]

I am still doing some hunting, trying to figure this out. Still a problem for me. Does anyone know if this might relate to the switch from babel 7.0.0 -> 7.5.5. This switch happened around the time I started experiencing this bug and coincides with me beginning to see problems...I have been trying to get yarn resolutions to work in order to peg the babel versions at 7.0.0 but haven't had success with this yet.

[ladyisidore]

I think I can offer some insight--I started having this problem when, halfway through adding plugins to a project, I updated gatsby-cli in another terminal window. Running gatsby clean in my project worked.

[JaKXz]

from https://github.com/gatsbyjs/gatsby/issues/6385#issuecomment-531949380 - I am also seeing this issue but gatsby clean did not solve it. It seems like it might just be the CLI printout is hanging, which is why resizing the terminal fixes it?? 🤷‍♂ 😕 ❓❗️

[ehowey]

Resizing the terminal window does seem to fix it for me! I have not tested it super extensively, can some other people having this problem try this as well? What an odd bug and potentially odd solution.

[emilyaviva]

I am facing the exact same issue—gatsby gets stuck on "source and transform nodes" or "createPagesStatefully" often, and I'm not using any source plugins. I only recently encountered the "resize terminal window" fix and am 140% baffled as to how in the heck this fixes it, but it does. This seems like a pretty serious issue!

[ehowey]

@JaKXz - Thank-you! This was driving me nuts. The fix does seem to be resizing the terminal window in VS Code. Just drag it up or down a bit and the develop task chugs along happily. I tested in a couple of different cases both yarn and npm, workspaces and not. Seemed to be working in all those cases for me. It also seems to freeze or hang at createPagesStatefully more often than source and transform nodes now for me. I am going to leave the issue open for now, this may need to be fixed in a less "hacky" way by someone with more know how than me.

[MaximeHeckel]

@ehowey I have the same issue and moving the terminal window in vscode works (couldn't believe it when reading this issue until I tried myself). Do you know if it only happens on VS Code?

[goblindegook]

I have this issue on iTerm 2, so it's not VS Code specific.

[domLocalHeroes]

My issue has also been in iTerm 2

[steinitz]

Webstorm terminal, Mac Terminal, iTerm

[ehowey]

Did the resizing terminal fix work for all of you in different dev environments?

[MaximeHeckel]

On my end, resizing the terminal worked on iTerm and VSCode (took a few tries to reproduce the issue on iTerm)

[emilyaviva]

The resizing terminal trick works reliably for me in iTerm 2.

[SilencerWeb]

Yep, resizing iTerm 2 works perfectly

[HaveF]

If resizing works, I suspect this bug is related buffer, somewhere needs stdout flush.

[Js-Brecht]

This kind of seems like a kernel+shell related issue. Probably some way node interacts with your kernel and/or your shell. I only say this because I can't seem to replicate the issues using Linux or Windows. I can't seem to find any known issues, so either a) it's some combination of code patterns unique to Gatsby + the interaction with the system, or b) I just don't know the right question to ask yet.

If resizing the terminal fixes the issue, then it might be some glitch with between node and kqueue causing a block in the event loop. Resizing the terminal sends the process a SIGWINCH signal, which interrupts the event loop, which frees it up and allows it to continue.

Can you try running kill -SIGWINCH ${pid} on the running process when it freezes? Should act the same as resizing the terminal.

I'm also interested in whether or not this happens in all shells. Based on the information so far, this has been failing in bash and zsh, and this is probably one of the common factors between all of the terminal emulators. Can you guys try one or more of sh, csh, ksh, tcsh, etc... ?

If the problem occurs in all of the shells, then I would guess that it's the way the kernel is handling node's event loop. This could also be why it's an intermittent problem. If some function gets less processor time (maybe because of variable system load), it might block for too long, and maybe the kernel tries to reuse an event node somewhere, resulting in a deadlock? I'm not super familiar with the internals of the api, but I bet that source and transform nodes is fairly filesystem IO intensive. That means there's probably a lot of offloading happening, so who knows what's really happening on lower levels.

I think it would be a good idea to try to narrow down the surface area of this bug. It occurs most frequently in source and transform nodes, it looks like, so maybe try narrowing it down to what plugin is blocking. Try adding the following lines to node_modules/gatsby/dist/utils/api-runner-node.js:

+ 347       if (api === `sourceNodes`) {
+ 348         debugger;
+ 349       }
  350       resolve(runAPI(plugin, api, Object.assign({}, args, {
  351         parentSpan: apiSpan
  352       })));

Then run node inspect node_modules/.bin/gatsby develop. It will break as soon as it starts, so you have to hit c when you get to the debug> prompt, and wait for it to continue. When it breaks on that debugger line, write exec console.log(plugin), and note what it says in resolve. Then hit c to continue. Just do this until it hangs... if it hangs.

Because of the nature of the event loop, I bet that it won't even hang while debugging. Those interrupts are probably enough to keep it on a track. Async bugs can be a real pain to track down. If it doesn't hang while using the debugger, replace that debugger line with:

reporter.log(plugin.resolve);

Hopefully that will turn something up. It would be nice to see what plugin is causing the block. If we can figure that out, we might be able to figure out what needs to be optimized/refactored in order to sort this out.

[andrico1234]

Resizing works for me too, I use zsh as my VSCode shell.

[ehowey]

@Js-Brecht - Thank-you for taking the time for such a detailed response. I was not sure exactly where I was supposed to input kill -SIGWINCH ${pid}. I couldn't do it during the build process.

Withe the debugger the following code came out...beyond me to make much sense of it. I actually got stuck in the debugger but .exit still worked.

⠋ source and transform nodes
break in /Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/utils/api-runner-node.js:365
 363       return new Promise(resolve => {
 364         if (api === `sourceNodes`) {
>365           debugger;
 366         }
 367         resolve(

< {
<   resolve: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/internal-plugins/internal-data-bridge',
<   id: 'a5079d69-ba80-53dc-82f9-0f440bd5448c',
<   name: 'internal-data-bridge',
<   version: '1.0.0',
<   pluginOptions: { plugins: [] },
<   nodeAPIs: [ 'sourceNodes', 'onCreatePage' ],
<   browserAPIs: [],
<   ssrAPIs: []
< }
< ⠋ source and transform nodes
debug> undefined
debug> c
break in /Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/utils/api-runner-node.js:365
 363       return new Promise(resolve => {
 364         if (api === `sourceNodes`) {
>365           debugger;
 366         }
 367         resolve(
⠦ source and transform nodes
break in /Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/utils/api-runner-node.js:365
 363       return new Promise(resolve => {
 364         if (api === `sourceNodes`) {
>365           debugger;
 366         }
 367         resolve(
{
<   resolve: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-plugin-mdx',
<   id: '479d2e25-5a33-5a59-a290-cf1877d39ee5',
<   name: 'gatsby-plugin-mdx',
<   version: '1.0.43',
<   pluginOptions: {
<     plugins: [ [Object] ],
<     extensions: [ '.md', '.mdx' ],
<     defaultLayouts: {
<       default: '/Users/erichowey/Coding/gatsby-catalyst/themes/gatsby-theme-catalyst-core/src/components/layout.js'
<     },
<     gatsbyRemarkPlugins: [ [Object], [Object], [Object] ],
<     remarkPlugins: [ [Function: slug] ]
<   },
<   nodeAPIs: [
<     'sourceNodes',
<     'onCreateNode',
<     'onCreatePage',
<     'onCreateWebpackConfig',
<     'resolvableExtensions',
<     'preprocessSource',
<     'onCreateBabelConfig',
<     'onPreBootstrap',
<     'onPostBootstrap'
<   ],
<   browserAPIs: [ 'wrapRootElement' ],
<   ssrAPIs: [ 'wrapRootElement' ],
<   pluginFilepath: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-plugin-mdx'
< }
< ⠦ source and transform nodes
debug> undefined
{
<   resolve: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-plugin-mdx',
<   id: '479d2e25-5a33-5a59-a290-cf1877d39ee5',
<   name: 'gatsby-plugin-mdx',
<   version: '1.0.43',
<   pluginOptions: {
<     plugins: [ [Object] ],
<     extensions: [ '.md', '.mdx' ],
<     defaultLayouts: {
<       default: '/Users/erichowey/Coding/gatsby-catalyst/themes/gatsby-theme-catalyst-core/src/components/layout.js'
<     },
<     gatsbyRemarkPlugins: [ [Object], [Object], [Object] ],
<     remarkPlugins: [ [Function: slug] ]
<   },
<   nodeAPIs: [
<     'sourceNodes',
<     'onCreateNode',
<     'onCreatePage',
<     'onCreateWebpackConfig',
<     'resolvableExtensions',
<     'preprocessSource',
<     'onCreateBabelConfig',
<     'onPreBootstrap',
<     'onPostBootstrap'
<   ],
<   browserAPIs: [ 'wrapRootElement' ],
<   ssrAPIs: [ 'wrapRootElement' ],
<   pluginFilepath: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-plugin-mdx'
< }
< ⠦ source and transform nodes
debug> undefined
debug> c
break in /Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/utils/api-runner-node.js:365
 363       return new Promise(resolve => {
 364         if (api === `sourceNodes`) {
>365           debugger;
 366         }
 367         resolve(
{
<   resolve: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-transformer-sharp',
<   id: '822bdf7b-a95a-5885-9351-158705910ac3',
<   name: 'gatsby-transformer-sharp',
<   version: '2.2.16',
<   pluginOptions: { plugins: [] },
<   nodeAPIs: [
<     'onCreateNode',
<     'setFieldsOnGraphQLNodeType',
<     'onPreExtractQueries',
<     'sourceNodes'
<   ],
<   browserAPIs: [],
<   ssrAPIs: [],
<   pluginFilepath: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-transformer-sharp'
< }

⠋ source and transform nodes
break in /Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/utils/api-runner-node.js:365
 363       return new Promise(resolve => {
 364         if (api === `sourceNodes`) {
>365           debugger;
 366         }
 367         resolve(

< {
<   resolve: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/internal-plugins/internal-data-bridge',
<   id: 'a5079d69-ba80-53dc-82f9-0f440bd5448c',
<   name: 'internal-data-bridge',
<   version: '1.0.0',
<   pluginOptions: { plugins: [] },
<   nodeAPIs: [ 'sourceNodes', 'onCreatePage' ],
<   browserAPIs: [],
<   ssrAPIs: []
< }
< ⠋ source and transform nodes
debug> undefined
debug> c
break in /Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby/dist/utils/api-runner-node.js:365
 363       return new Promise(resolve => {
 364         if (api === `sourceNodes`) {
>365           debugger;
 366         }
 367         resolve(
{
<   resolve: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-source-filesystem',
<   id: '339022db-842c-55bd-8e87-d84bd74a175a',
<   name: 'gatsby-source-filesystem',
<   version: '2.1.26',
<   pluginOptions: { plugins: [], name: 'src', path: 'src/' },
<   nodeAPIs: [ 'sourceNodes', 'setFieldsOnGraphQLNodeType' ],
<   browserAPIs: [],
<   ssrAPIs: [],
<   pluginFilepath: '/Users/erichowey/Coding/gatsby-catalyst/node_modules/gatsby-source-filesystem'
< }
< ⠋ source and transform nodes
debug> undefined
⠹ source and transform nodes
debug> exec console.log(plugin)
c
c

[rm-rf-etc]

macOS Sierra, using Terminal, resizing worked for me.

[Js-Brecht]

@ehowey Just so I know I'm understanding you correctly, it did hang while you were in the debugger? In that case, it looks to me like gatsby-source-filesystem is the culprit, which makes sense to me... especially because of existing, related, gatsby issues.

I'd like to see if the plugin can handle a test run. If it hangs while running tests, then I think that will be an easy way to see where it is failing. If not, well... will just have to do some dissecting/debugging, which will be difficult for me, since I don't have MacOS.

Can you download the main gatsby repo, and run the gatsby-source-filesystem tests?

> git clone git@github.com:gatsbyjs/gatsby.git gatsby-js
> cd gatsby-js
> yarn bootstrap
> yarn jest -- -i './packages/gatsby-source-filesystem/src'

Also, are you doing all of this with your minimal reproduction repository? I see that gatsby-plugin-mdx was run twice, so that tells me that it is not a barebones repository. In a perfect world, it shouldn't matter, but I think it would be better to run as simple a setup as possible. If you can't get it to fail reliably with the minimal repo, then use whichever one fails most consistently (in the same place, every time (or close to))

😉

---

kill -SIGWINCH ${pid} must be run from another shell instance. When the build hangs, open another terminal window/tab, and run the command from there. This little snippet should work:

pid=$(ps -ef |grep -E 'node.*index\.js develop' |awk '{ print $2 }').
kill -SIGWINCH $pid

# Can also be run like this
kill -SIGWINCH $(ps -ef |grep -E 'node.*index\.js develop' |awk '{ print $2 }')

If there are errors, try running the commands seperately:

# Run first
ps -ef grep -E 'node.*index\.js develop'
# Example output
     UID     PID    PPID  TTY        STIME COMMAND
********   39928       1 cons0    06:47:38 /path/to/node /path/to/gatsby-cli/lib/index.js develop
# Second column is the pid you want
kill -SIGWINCH 39928

---

If you come up empty-handed after running the tests, I think it will be useful to have a core dump, so we can get a stack trace. But, one step at a time.

[ehowey]

@Js-Brecht Sorry for the slower responses...this is really just a side hobby I do in the spare time in the evenings.

So I ran the same thing inside of the the gatsby hello world starter - as bare bones as you can get. Sorry I ran it before in my main workspace on a project I had been having problems with. I have previously replicated it on starters though so I dont think it relates to a plugin but rather to something in the core of gatsby.

It hangs on source and transform nodes giving me the following output:

< success onPreBootstrap - 0.102 s
⠋ source and transform nodes
break in node_modules/gatsby/dist/utils/api-runner-node.js:362
 360       return new Promise(resolve => {
 361         if (api === `sourceNodes`) {
>362           debugger
 363         }
 364         resolve(

< {
<   resolve: '/Users/erichowey/Coding/my-hello-world-starter/node_modules/gatsby/dist/internal-plugins/internal-data-bridge',
<   id: 'a5079d69-ba80-53dc-82f9-0f440bd5448c',
<   name: 'internal-data-bridge',
<   version: '1.0.0',
<   pluginOptions: { plugins: [] },
<   nodeAPIs: [ 'sourceNodes', 'onCreatePage' ],
<   browserAPIs: [],
<   ssrAPIs: []
< }
< ⠋ source and transform nodes
debug> undefined

< success source and transform nodes - 25.137 s

Here is a dump from the gatsby info command in case that is helpful:

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (2) x64 Intel(R) Core(TM)2 Duo CPU     P8600  @ 2.40GHz
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 12.9.1 - /usr/local/bin/node
    Yarn: 1.17.3 - /usr/local/bin/yarn
    npm: 6.10.3 - /usr/local/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 76.0.3809.132
    Firefox: 67.0.2
    Safari: 13.0.1
  npmPackages:
    gatsby: ^2.15.22 => 2.15.22 
  npmGlobalPackages:
    gatsby-cli: 2.7.47

[ehowey]

As a side note - when I use your debug command it hangs on source and transform nodes. When I use gatsby develop it is mostly hanging at createPagesStatefully now for me. Sorry I know this is a really odd one. To be honest I am not sure how much work it is worth from your end of things. The resizing the terminal window trick works every time for me and others. It is a hacky fix but it must not be affecting that many users otherwise the issues would be flooded.

[christopherrobinson]

This has started happening over here as well. The resize the terminal trick resolves it; very odd!

[antonfrolovsky]

+1 Doesn't work in iTerm2, but work in Terminal 🤷‍♂️

[Js-Brecht]

@ehowey The majority of what is happening during the build is defined by one plugin or another. Gatsby ships with them as dependencies; I suppose they could be considered core plugins. The Gatsby core exposes an API, which looks for endpoints defined in the plugins, and passes them a series of arguments which include actions/objects that are defined in the core. So yes, what is happening could be happening in the core, but it first has to call out to a plugin, and those plugins define a specific context for the API. I'm trying to determine the context that is resulting in the build hanging, and also need to verify that the issue is not occurring in the plugin itself.

Can I get you to change these lines?

 360       return new Promise(resolve => {
-361         if (api === `sourceNodes`) {
-362           debugger
-363         }
+364         reporter.log(`${api}\t${plugin.name}`)
 365         resolve(

Please run this, and copy/paste the entire output (might even just attach it to your response as a .txt file). It will be considerably more verbose, and a lot of the information will probably be unnecessary, but 🤷‍♂.

---

After you've done that, I'd like to see if increasing node's thread pool makes a difference. I've noticed other issues that mention the same, or similar issues. Mostly they occur in source and transform, and some talk about that stage taking forever, or completely locking up. So, I want to see if it happens to be some kind of deadlock in filesystem io... asynchronous file system access is offloaded by node to different threads, and, by default, node only has a pool of 4 threads for userspace. If those fill up, and it needs to do any more offloading, the tasks will queue up in the main event loop, waiting for a thread. This could, potentially, bring the entire program to a standstill... until a thread becomes available. Gatsby maintains a cache on the filesystem, so perhaps there's a collision occurring somewhere, and if there's some kind of deadlock happening, then perhaps the threads are waiting for a timeout/interrupt before moving on, and if there's dozens (or hundreds, even) of filesystem access events, they could all be waiting for the same timeout, and causing everything to pile up. You can see that this could cause wait times to grow rather quickly.

Increasing the pool size might help alleviate some of the traffic... or, it could just happen the same way, just with more threads 😆. Try the following command:

UV_THREADPOOL_SIZE=10 gatsby develop