search

Js-Brecht / gatsby-plugin-pnpm

Provides PNPM compatible module resolvers to Webpack for Gatsby
MIT License
48 stars 6 forks source link

Fragment from shadowed components does not exists #9

Closed TheGuit closed 3 years ago

TheGuit commented 3 years ago

Hi,

We are facing an issue with pnpm and Gatsby.

We have a folder structure like this :

When trying to launch develop or build we have and error : 

 ERROR #85908  GRAPHQL

There was an error in your GraphQL query:

The fragment "XXX" does not exist.

or

There was an error in your GraphQL query:

Unknown type "node__cde".

Have you any idea that could help us to use pnpm with Gatsby ?

Regards,

Js-Brecht commented 3 years ago

Have you confirmed that this only happens when you use pnpm?

TheGuit commented 3 years ago

Yes,

What we try to achieve is to reduce space usage of our gatsby installation. We have 600 Drupal sites as CMS backend, that use the same gatsby base code to produce static sites, we have a process to build them on-demand :

Actually gatsby + node_modules is about ~590 Mo, copiyng/cleaning strategy could take really long time (15 to 30 minutes).

One of our idea was to use Yarn pnp, Yarn 2 or pnpm to have less data to copy (using symlink or thing like that).

But at the moment it's not very effective :(

Js-Brecht commented 3 years ago

Are you able to share the repository, or a reproduction of the problem? I'm pretty certain that this is going to be an issue in the core, but I can at least investigate it and get more information.

Gatsby scans the directories of plugins looking for fragments, so maybe there's some issue when using pnpm + shadowing that's making it fail to resolve the directories/modules correctly. I currently use pnpm for every Gatsby project I use (periodically, I use Yarn 2), and I haven't had this problem yet... but I also don't use shadowing much.

Chacalbis commented 3 years ago

Hi @Js-Brecht, This is a repository which is a simple reproduction of our project and represents the state of our investigations.

TheGuit commented 3 years ago

To complete @Chacalbis comment.

We tests multiples alternative (Yarn 1, Yarn 1 + pnp, Yarn 2, pnpm) : https://github.com/TheGuit/gatsby-example-pm-errors/tree/yarn2-with-pnp

The readme explain our actual problem with Yarn 2.

Our problem with pnpm doesn't appear in our simplified test, we continue our investigation on pnpm, but it doesn't seems to fix our space problem.

Js-Brecht commented 3 years ago

@TheGuit I will take a look, but I just had one quick note about pnpm: the node_modules for a project will report space used because the files contained within it are hard links to the global store, so they are reporting the size of the actual node on the disk... assuming the store and the node_modules are on the same partition. However, once all packages exist in the global store, you should see no increase in total disk space usage after dependency installation in a project. More information here

If your global store and your project exist on different partitions then the files will be copied, by necessity. You cannot create hard links across partitions.

Similar kind of thing applies for yarn 2. You can set up a global store, and tell your installation to use that instead of the local project's .yarn/cache directory.

Js-Brecht commented 3 years ago

I am unable to reproduce the problem using the branch pnpm and branch pnpm-gpo. Not sure why... everything seems to work correctly for me. I am using pnpm version 6.11.5 and node version 12.21.0

The issue with yarn 2 is because of the readdir method that true-case-path uses to check the case of the path you've provided it. It wants to iterate through each folder, starting from root, looking for the provided path segment in each folder (case insensitive). If that provided path segment doesn't exist, it fails, assuming that the file doesn't exist.

Since yarn 2 uses a virtual fs, readdir is not returning the folder contents accurately, until you get into the actual contents of the virtual dir. For example:

If my base path is: process.cwd() My relative path is: .yarn/$$virtual/gatsby-theme-parent-virtual-0b5977071c/1/themes/gatsby-theme-parent/src/pages/404.js

So, true-case-path will loop through every segment of process.cwd() + "/.yarn/$$virtual/gatsby-theme-parent-virtual-0b5977071c/1/themes/gatsby-theme-parent/src/pages/404.js" until it successfully exhausts the path (case correcting as it goes), or it hits a segment where a node doesn't exist.

The problem occurs at .yarn/$$virtual through .yarn/$$virtual/gatsby-theme-parent-virtual-0b5977071c... those are virtual containers, and readdir is always returning the contents of .yarn for each of them instead. Once it hits .yarn/$$virtual/gatsby-theme-parent-virtual-0b5977071c/1, everything is fine.

This is a problem that would need to be addressed in true-case-path. Interestingly, I requested a change a while back while working on some rush ticket that might help address this problem, but wouldn't solve it entirely. What it needs to do is allow path segments that don't exist in readdir output, and continue checking the rest anyway as it progresses.

TheGuit commented 3 years ago

I am unable to reproduce the problem using the branch pnpm and branch pnpm-gpo. Not sure why... everything seems to work correctly for me. I am using pnpm version 6.11.5 and node version 12.21.0

Yes, as I say yesterday the problem doesn't show up in our simplified repository.

So we made new test yesterday on our original repository. We found that pnpm was doing something "weird", it use one package with the same name from registry instead of our local package in a workspace, we stick version of our local module with something random. At the moment pnpm is working on our repository.

I have although understand your explanation about hardlink, so yes, it could be suitable for our initial search about reducing disk usage.

Thank for all your information about Yarn 2, I think a lot of work needs to be done to ensure full Gatsby compatibility.

Js-Brecht commented 3 years ago

Oh, I see. I’m sorry, I guess I overlooked where you said it was working in the simplified repo.

About pnpm using a version from the registry: you could try using version protocol workspace:* for workspace dependencies, so that they are explicitly linked. More info on that here.

There’s also a few .npmrc settings that I usually turn on, to make things explicit:

link-workspace-packages=deep
prefer-workspace-packages=true
shared-workspace-lockfile=true

More info on those options here

I’d also be careful using the shamefully-hoist option. It can wind up causing problems, just like you run into with npm and yarn’s hoisting, especially if you start to get a lot of dependencies in the workspace.

TheGuit commented 3 years ago

Hi Jeremy,

Sorry for my late reply, thanks for all your tips and for your time.

We have a working version of PNPM at the moment, with shamefully-hoist option, we will try to remove this options in a near future, but for the moment our sites don't build without this option. We have to identify all dependency not properly listed or one that need explicit hoisting.

Again, thanks for your work on pnpm + gatsby and your time on this issue.

I close for the moment as we have a functional version.

Regards,