JIT setting causes Javascript heap out of memory error on mutation

[emanuelet]

Hi,

I found a pattern where when the JIT compiler is engaged by a mutation, that triggers a loop which in turn causes an out of memory error.

The mutation were generated with sequelize-graphql-schema and once I disabled jit, it all worked perfectly.

I have attached a CPU profile and in case I can send an heapdump that is pretty big tho. CPU-20210304T121833.zip

[mcollina]

Thanks for reporting @emanuelet. We would need a minimal reproduction to be able to open a bug report to graphql-jit (and trying to fix it).

[emanuelet]

You're right.

First of all here's a gist of my fastify setup.

For reproduction:

1. Run the server
2. Execute the mutation auto-generated from the model like so

   mutation UpdateProvider($provider: providerInput!) {
   providerEdit(provider: $provider) {
     id
   }
   }

3. execute another query
4. execute the same mutation from before with a new value
5. execute another query
6. the server should still spin for a few more seconds before throwing a Heap out of memory error and crash.

I tried changing the jit value from 1 to 2 and I saw that in that case I was able to do 1 more round of mutation and query before the server crashed.

[mcollina]

It does seem the combination of those dependency is not really expected:

$ npm i mercurius fastify sequelize fastify-autoload config graphql sequelize-graphql-schema @graphql-tools/schema @graphql-tools/stitch
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! Found: graphql@15.5.0
npm ERR! node_modules/graphql
npm ERR!   graphql@"*" from the root project
npm ERR!   peer graphql@"^14.0.0 || ^15.0.0" from @graphql-tools/schema@7.1.3
npm ERR!   node_modules/@graphql-tools/schema
npm ERR!     @graphql-tools/schema@"*" from the root project
npm ERR!     @graphql-tools/schema@"^7.1.2" from @graphql-tools/stitch@7.4.0
npm ERR!     node_modules/@graphql-tools/stitch
npm ERR!       @graphql-tools/stitch@"*" from the root project
npm ERR!   2 more (@graphql-tools/stitch, mercurius)
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! sequelize-graphql-schema@"*" from the root project
npm ERR!
npm ERR! Conflicting peer dependency: graphql@14.7.0
npm ERR! node_modules/graphql
npm ERR!   peer graphql@"^14.1.1" from sequelize-graphql-schema@0.1.71
npm ERR!   node_modules/sequelize-graphql-schema
npm ERR!     sequelize-graphql-schema@"*" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR! See /Users/matteo/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/matteo/.npm/_logs/2021-03-05T09_49_45_179Z-debug.log

Could you try to upload a full repository? I have not been able to run your gist.

After some tries I got to:

(node:17353) UnhandledPromiseRejectionWarning: Error: Cannot use GraphQLSchema "[object GraphQLSchema]" from another module or realm.

Ensure that there is only one instance of "graphql" in the node_modules
directory. If different versions of "graphql" are the dependencies of other
relied on modules, use "resolutions" to ensure only one version is installed.

https://yarnpkg.com/en/docs/selective-version-resolutions

Duplicate "graphql" modules cannot be used at the same time since different
versions may have different capabilities and behavior. The data from one
version used in the function from another could produce confusing and
spurious results.
    at instanceOf (/Users/matteo/temp/mercurius-jit-bug/node_modules/mercurius/node_modules/graphql/jsutils/instanceOf.js:29:13)
    at isSchema (/Users/matteo/temp/mercurius-jit-bug/node_modules/mercurius/node_modules/graphql/type/schema.js:42:34)
    at assertSchema (/Users/matteo/temp/mercurius-jit-bug/node_modules/mercurius/node_modules/graphql/type/schema.js:46:8)
    at validateSchema (/Users/matteo/temp/mercurius-jit-bug/node_modules/mercurius/node_modules/graphql/type/validate.js:42:28)
    at Object.<anonymous> (/Users/matteo/temp/mercurius-jit-bug/node_modules/mercurius/index.js:204:36)
    at /Users/matteo/temp/mercurius-jit-bug/node_modules/fastify/lib/hooks.js:142:22
    at _encapsulateThreeParam (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:545:7)
    at Boot.timeoutCall (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:447:5)
    at Boot.callWithCbOrNextTick (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:428:19)
    at Task.release (/Users/matteo/temp/mercurius-jit-bug/node_modules/fastq/queue.js:147:16)
    at worked (/Users/matteo/temp/mercurius-jit-bug/node_modules/fastq/queue.js:199:10)
    at Boot.timeoutCb (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:456:7)
    at /Users/matteo/temp/mercurius-jit-bug/node_modules/fastify/fastify.js:454:9
    at Object._encapsulateThreeParam (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:545:7)
    at Boot.timeoutCall (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:447:5)
    at Boot.callWithCbOrNextTick (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:428:19)
    at release (/Users/matteo/temp/mercurius-jit-bug/node_modules/fastq/queue.js:147:16)
    at Object.resume (/Users/matteo/temp/mercurius-jit-bug/node_modules/fastq/queue.js:80:7)
    at /Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/boot.js:167:18
    at /Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/plugin.js:269:7
    at done (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/plugin.js:201:5)
    at check (/Users/matteo/temp/mercurius-jit-bug/node_modules/avvio/plugin.js:225:9)

If you could reproduce it by removing most of those dependencies it would be great, thanks.

[emanuelet]

That's quite weird. I tried creating a new folder and run the same npm install command you run and I got through.

$ npm i mercurius fastify sequelize fastify-autoload config graphql sequelize-graphql-schema @graphql-tools/schema @graphql-tools/stitch
npm WARN saveError ENOENT: no such file or directory, open '/home/Personal Projects/test/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/home/Personal Projects/test/package.json'
npm WARN sequelize-graphql-schema@0.1.71 requires a peer of sequelize@^4.41.2 but none is installed. You must install peer dependencies yourself.
npm WARN sequelize-graphql-schema@0.1.71 requires a peer of graphql@^14.1.1 but none is installed. You must install peer dependencies yourself.
npm WARN graphql-relay@0.5.5 requires a peer of graphql@^0.5.0 || ^0.6.0 || ^0.7.0 || ^0.8.0-b || ^0.9.0 || ^0.10.0 || ^0.11.0 || ^0.12.0 || ^0.13.0 but none is installed. You must install peer dependencies yourself.
npm WARN dataloader-sequelize@1.7.10 requires a peer of sequelize@^3.24.6 || ^4.0.0 || ^5.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN graphql-sequelize@9.2.0 requires a peer of graphql@^0.5.0 || ^0.6.0 || ^0.7.0 || ^0.8.0 || ^0.9.0 || ^0.10.0 || ^0.11.0 || ^0.12.0 || ^0.13.0 || ^14 but none is installed. You must install peer dependencies yourself.
npm WARN test No description
npm WARN test No repository field.
npm WARN test No README data
npm WARN test No license field.

+ graphql@15.5.0
+ config@3.3.4
+ sequelize@6.5.0
+ fastify@3.13.0
+ @graphql-tools/schema@7.1.3
+ @graphql-tools/stitch@7.4.0
+ mercurius@7.2.0
+ fastify-autoload@3.5.2
+ sequelize-graphql-schema@0.1.71
added 218 packages from 197 contributors and audited 218 packages in 16.235s

33 packages are looking for funding
  run `npm fund` for details

found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

I should point out also that in my project package.json I don't specify or require a specific version of the graphql package.

I;m using node v12.18.1 and npm 6.14.5

[nherment]

@emanuelet I'm going through this and trying to reproduce your issue but it would be of great help if you could provide a fully working example. As Matteo mentioned, the provided gist does not run.

[jamesmeneghello]

Getting it in one of our projects as well. Occurs when the Input arg of a mutation has a circular dependency, in our instance:

e.g. chatMessage.tradeCompany.trade.tradeCompanies[0].trade.tradeCompanies[0]... until OOM.

[mcollina]

If you would provide a script to reproduce this reliably it would be amazing. This is likely to be a bug in graphql-jit.

[mohammed-bahumaish]

Hi,

I found a pattern where when the JIT compiler is engaged by a mutation, that triggers a loop which in turn causes an out of memory error.

The mutation were generated with sequelize-graphql-schema and once I disabled jit, it all worked perfectly.

I have attached a CPU profile and in case I can send an heapdump that is pretty big tho. CPU-20210304T121833.zip

i had the same issue. JavaScript heap out of memory when using Jit. some mutations work. and big mutations fail.

[mohammed-bahumaish]

that's the error message. i hope it help.

`

                api:dev: <--- Last few GCs --->
                api:dev: 
                api:dev: [26391:0x50bb970]    28692 ms: Mark-sweep 2025.5 (2079.1) -> 2006.4 (2064.5) MB, 142.5 / 0.0 ms  (average mu = 0.182, current mu = 0.128) allocation failure scavenge might not succeed
                api:dev: [26391:0x50bb970]    28867 ms: Mark-sweep 2033.0 (2075.5) -> 2011.7 (2069.6) MB, 141.4 / 0.0 ms  (average mu = 0.187, current mu = 0.191) allocation failure scavenge might not succeed
                api:dev: 
                api:dev: 
                api:dev: <--- JS stacktrace --->
                api:dev: 
                api:dev: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
                api:dev:  1: 0xb09c10 node::Abort() [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  2: 0xa1c193 node::FatalError(char const*, char const*) [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  3: 0xcf8dde v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  4: 0xcf9157 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  5: 0xeb09f5  [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  6: 0xeb14d6  [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  7: 0xebf9fe  [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  8: 0xec0440 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev:  9: 0xec33be v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev: 10: 0xe848fa v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev: 11: 0x11fd646 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/root/.nvm/versions/node/v16.15.1/bin/node]
                api:dev: 12: 0x15f20b9  [/root/.nvm/versions/node/v16.15.1/bin/node]

`

[mcollina]

We often need a reproducible example, e.g. some code that allows someone else to recreate your problem by just copying and pasting it. If it involves more than a couple of different file, create a new repository on GitHub and add a link to that.