Javascript heap out of memory - circular references?

[wabiloo]

I tried to open our current OpenAPI spec (json file) with Enforcer. After a few minutes, I got a FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I think this may be due to the fact that there are circular references in our spec. I could not find an option to make dereferencing ignore circular references (which is possible with swagger parser: https://github.com/APIDevTools/swagger-parser/blob/main/docs/README.md#circular-refs)

If it's not that, it may be the shear size of our spec (500+ paths, 1000+ schemas). Unfortunately I cannot share the spec, at least not publically. (do contact me privately if you'd like to have a copy for testing)

Note: I used very simple code, from your documentation:

const Enforcer = require('openapi-enforcer')

Enforcer('../../openapi.json', { fullResult: true })
    .then(function ({ error, warning }) {
        if (!error) {
            console.log('No errors with your document')
            if (warning) console.warn(warning)
        } else {
            console.error(error)
        }
    })

[Gi60s]

Hello, @wabiloo. Thanks for the issue. The library does correctly identify and link support circular references, so I'm guessing it has to do with the size of your API.

I'm happy to take a look at this and it may even lead to memory optimizations. Will you create a private Github repo, put your definition into it, and then give me read access to the repo?

[wabiloo]

Thanks for offering to take a look. I have sent you an invitation to a private repo, which only has that one file in it (called "openapi-clients.json")

[Gi60s]

Thanks for being willing to help debug this too.

I was able to run this successfully on my machine although it took 37 seconds to run.

I found this on stack overflow that talks about this error you're seeing: https://stackoverflow.com/questions/53230823/fatal-error-ineffective-mark-compacts-near-heap-limit-allocation-failed-javas

One thing that caught my eye is one answer mentioned updating the version of NodeJS fixed the problem. I checked my node version and I'm running 14.17.0.

In that same stack overflow page there are other alternatives too.

I'll run a profile too and see what else I can learn about this.

[wabiloo]

Ah, I didn't think of trying with more recent versions of Node. I use v12.18.3.

Note however that I'm hoping to be able to wrap this (with Webpack) in a browser-based tool (to allow navigation / discovery of the spec), so I'm not sure I'd get much control over version in the first place...

[Gi60s]

Ah, if you're hoping to get this running on the browser then you'll likely run into loading issues. The library can load via HTTP, HTTPS, or the file system. The file system is what makes browsers choke.

There is a version 2 of this library in the works that is about half done. It is an isomorphic application that will allow it to be run on the browser or on NodeJS. It also provides for custom loaders in case you need to load content that is behind an authenticated endpoint.

You can check out the library on the v2 branch of this repo.

[wabiloo]

Thanks, will have a look.

Did you learn anything from running a profile though? I won't pretend that it's similar but when I use the swagger parser's dereference function, I don't get this type of issue

[Gi60s]

I did learn some things, but nothing that is easily addressable in v1. I will be learning from this for v2 though so if all goes well we should see some significant performance improvements there too.