Memory limits, slow performance and hangs while trying to use the webpack loader

Hi There! I was taking a shot at this very cool library! 🎉 With the 0.18 release I thought it was worthwhile to take a look. We are currently using graphql-code-generator to generate our types but I though: How nice would it be if we don't need a separate process and everything runs through webpack. A smoother and faster development experience would be nice.

We're running the tools in a monorepo setup, which complicates things a bit and I'm not sure if my issues are related to this.

Running with graphql-code-generator

We're currently running a custom version of near-operation-file which generates FileName.gql.ts files, which isn't as good as the FileName.graphql.d.ts files, we don't have the cache setup that graphql-let offers.

graphql codegen.yml setup (expand for details)

```yml generates: # Generate a schema file ../../packages/graphql/generated/schema.graphqls: schema: - ${NEXT_PUBLIC_GRAPHQL_ENDPOINT} - '**/*.graphqls' - '../../packages/magento-*/**/*.graphqls' - '../../packages/mollie-magento-payment/**/*.graphqls' - '../../packagesDev/**/*.graphqls' plugins: - schema-ast - add config: includeDirectives: true commentDescriptions: false content: '#' # Generate a types file ../../packages/graphql/generated/types.ts: schema: - ${NEXT_PUBLIC_GRAPHQL_ENDPOINT} - '**/*.graphqls' - '../../packages/magento-*/**/*.graphqls' - '../../packages/mollie-magento-payment/**/*.graphqls' - '../../packagesDev/**/*.graphqls' plugins: - typescript - typescript-apollo-client-helpers - add config: enumsAsTypes: true content: '/* eslint-disable */' # Generate a fragments.json ../../packages/graphql/generated/fragments.json: schema: - ${NEXT_PUBLIC_GRAPHQL_ENDPOINT} - '**/*.graphqls' - '../../packages/magento-*/**/*.graphqls' - '../../packages/mollie-magento-payment/**/*.graphqls' - '../../packagesDev/**/*.graphqls' plugins: - fragment-matcher config: apolloClientVersion: 3 # Generate .gql.ts files for each GraphQL file ../../: schema: - ${NEXT_PUBLIC_GRAPHQL_ENDPOINT} - '**/*.graphqls' - '../../packages/magento-*/**/*.graphqls' - '../../packages/mollie-magento-payment/**/*.graphqls' - '../../packagesDev/**/*.graphqls' documents: - '**/*.graphql' - '../../packages/**/*.graphql' # - '../../packagesDev/**/*.graphql' plugins: - '@graphcommerce/graphql-codegen-relay-optimizer-plugin' - typed-document-node - typescript-operations - add preset: '@graphcommerce/graphql-codegen-near-operation-file' presetConfig: extension: .gql.ts baseTypesPath: ~@graphcommerce/graphql injectables: true config: skipTypename: true dedupeOperationSuffix: true content: '/* eslint-disable */' ```

$ NEXT_PUBLIC_GRAPHQL_ENDPOINT=.mesh/schema.graphql node_modules/.bin/graphql-codegen --require dotenv/config
  ✔ Parse configuration
  ✔ Generate outputs
✨  Done in 7.84s.

Running with graphql-let

We've been running this schema

.graphql-let.yml coniguration (expand for details)

```yml schema: - ${NEXT_PUBLIC_GRAPHQL_ENDPOINT} - '**/*.graphqls' - '../../packages/**/*.graphqls' #- '.packages/mollie-magento-payment/**/*.graphqls' documents: - 'components/**/*.graphql' - '../../packages/**/*.graphql' plugins: - typed-document-node - typescript-operations - add config: skipTypename: true dedupeOperationSuffix: true content: '/* eslint-disable */' cacheDir: .cache/graphql-let/examples/magento-graphql respectGitIgnore: false ```

Before my first run, I've modified all GraphQL files to have the # import MyFragment from bla.graphql declarations. We're making heavy use of fragments.

yarn graphql-let --require dotenv/config (expand for details)

``` NEXT_PUBLIC_GRAPHQL_ENDPOINT=.mesh/schema.graphql yarn graphql-let --require dotenv/config [graphql-let] Processing 251 codegen... <--- Last few GCs ---> [65061:0x104b00000] 149454 ms: Scavenge (reduce) 4010.0 (4105.2) -> 4009.8 (4105.2) MB, 7.5 / 0.0 ms (average mu = 0.160, current mu = 0.089) allocation failure [65061:0x104b00000] 149464 ms: Scavenge (reduce) 4010.4 (4102.2) -> 4010.2 (4103.2) MB, 5.7 / 0.0 ms (average mu = 0.160, current mu = 0.089) allocation failure [65061:0x104b00000] 149472 ms: Scavenge (reduce) 4010.8 (4105.2) -> 4010.6 (4105.2) MB, 5.6 / 0.0 ms (average mu = 0.160, current mu = 0.089) allocation failure <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x101319fc5 node::Abort() (.cold.1) [.nvm/versions/node/v14.18.1/bin/node] 2: 0x1000b6169 node::Abort() [.nvm/versions/node/v14.18.1/bin/node] 3: 0x1000b62df node::OnFatalError(char const*, char const*) [.nvm/versions/node/v14.18.1/bin/node] 4: 0x100200ba7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [.nvm/versions/node/v14.18.1/bin/node] 5: 0x100200b43 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [.nvm/versions/node/v14.18.1/bin/node] 6: 0x1003aec55 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [.nvm/versions/node/v14.18.1/bin/node] 7: 0x1003b06fa v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [.nvm/versions/node/v14.18.1/bin/node] 8: 0x1003abe25 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [.nvm/versions/node/v14.18.1/bin/node] 9: 0x1003a9750 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [.nvm/versions/node/v14.18.1/bin/node] 10: 0x1003b7e7a v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [.nvm/versions/node/v14.18.1/bin/node] 11: 0x1003b7f01 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [.nvm/versions/node/v14.18.1/bin/node] 12: 0x100385cd2 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [.nvm/versions/node/v14.18.1/bin/node] 13: 0x100705d08 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [.nvm/versions/node/v14.18.1/bin/node] 14: 0x100a8deb9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [.nvm/versions/node/v14.18.1/bin/node] ```

After upping the memory limit (although 4GB for 250 files seems excessive) it does go through. In practice I can't up the memory limit as not all build systems have more than 4GB om memory available.. So this creates an issue

NODE_OPTIONS=--max-old-space-size=8192 yarn graphql-let --require dotenv/config (expand for details)

``` NODE_OPTIONS=--max-old-space-size=8192 NEXT_PUBLIC_GRAPHQL_ENDPOINT=.mesh/schema.graphql yarn graphql-let --require dotenv/config [graphql-let] Done processing 251 GraphQL documents. ✨ Done in 141.37s. ```

The initial compilation takes 18 times as long as graphql-code-generator. There seems to be some room for optimization 😉

Using graphql-let via webpack hangs indefinitely

I've added the following to my next.config.js:

  webpack: (config, { defaultLoaders }) => {
    config.module.rules.push({
      test: /\.graphql$/,
      exclude: /node_modules/,
      use: [defaultLoaders.babel, { loader: 'graphql-let/loader' }],
    })

    config.module.rules.push({
      test: /\.graphqls$/,
      exclude: /node_modules/,
      use: ['graphql-let/schema/loader'],
    })
    return config
  },

When starting next dev I get the following output and after a few minutes it still hangs here:

yarn run v1.19.1
$ /Users/paulhachmang/Sites/reachdigital-next/node_modules/.bin/next dev
ready - started server on 0.0.0.0:3000, url: http://localhost:3000
info  - Loaded env from /Users/paulhachmang/Sites/reachdigital-next/examples/magento-graphcms/.env
warn  - You have enabled experimental feature(s).
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

warn  - Disabled SWC as replacement for Babel because of custom Babel configuration ".babelrc" https://nextjs.org/docs/messages/swc-disabled
> [PWA] PWA support is disabled
info  - automatically enabled Fast Refresh for 2 custom loaders
info  - Using external babel configuration from /Users/paulhachmang/Sites/reachdigital-next/examples/magento-graphcms/.babelrc
[BABEL] Note: The code generator has deoptimised the styling of /Users/paulhachmang/Sites/reachdigital-next/packages/graphql/generated/types.ts as it exceeds the max of 500KB.
[graphql-let] Processing codegen for ../../packages/magento-cart/components/InlineAccount/InlineAccount.graphql...

If I take a look at my activity monitor it seems that the node process isn't doing anything. It isn't growing in memory or using any CPU. Something isn't working as expected.

Again, don't know if this is all because I'm using a monorepo setup and it is just a faulty configuration on my end. Would really love to get this to work though :)

piglovesyou / graphql-let