MEMORY LEAK ISSUE

[vijayseid]

Hi team, facing an memory leak issue, memory keep on increasing & reaches the max heap.

Test Scenario:

JMETER : 1000 threads, duration 72000.

[GavinChenYan]

@vijayseid Do you have detail test case? And the error message detail? We will take a look

[GavinChenYan]

@vijayseid Please take a look you code, for temp solution, please verify : OpenApiValidator openApiValidator = new OpenApiValidator(),
this should be defined at application or instance level, try not new it for every request

[vijayseid]

Thanks for the suggestion @GavinChenYan. Doing the same, OpenApiValidator instance is created on application level.

Test Scenarios: Test Load : 1000 User threads, 30 Ramp up seconds, Durations 24hours, loop count : Infinite. Test Tool : Jmeter No of HTTP Request : 1

NOTE:

• Using Spring Boot application with JAVA 17, and limit the heap memory to 2GB.
• My spring application has only one function, & that is the openapi schema validation. NO OTHER CODE BLOCKS PRESENT.
• Using JConsole to monitor the heap memory

When I Start the test, Heap memory graphs seems to up & down only, but the OLD GEN Graph slowly claims up & not comes down. So Slowly my heap memory gets increasing & reached the max very soon. Facing an Exception in thread "main" java.lang.OutOfMemoryError: Java heap space issue on my spring boot application.

please take a loot at this graph, old gen keep on increasing.

For confirming this is to be a memory leak issue, tried the same type of load to another lib (SwaggerValidatior/ OpenApiInteractionValidator), there Clearly I can see the Heap & Old Gen heap goes up & comes down after sometime. You can see that graph as well here

So it's clearly says some memory leak on this lib.

Few tries to identify the memory leak issue, Request Entity & Status Objects are set to null after validation

If anything require from my side, please let me know & will try to submit here.

Reason to Stick on this lib is, It has Path variable validation & FailFast Feature, which is not available on Swagger/OpenApiInteractionValidator

Your Suggestion @GavinChenYan & Fix help me to solve my issue & I will be ready for Production ˶ᵔ ᵕ ᵔ˶

Thanks for the support.

[vijayseid]

Last Night run the perf testing again, here the screenshot of that @GavinChenYan. Almost reached the 2GB heap within 9hr 30min, soon reach the 2gb heap & ends in OUTOFMEMORY issue.

Heap Memory :

Old Gen :

Jmeter Result :

[vijayseid]

Heap memory reaches max & application failed after 16hour of testing

Old Gen:

[vijayseid]

@GavinChenYan

22/05 result :

[GavinChenYan]

@vijayseid Will try to have a new release for this issue. Sorry for the delay.

And would you mind share you code for how to initial the OpenApiValidator? which constructor you are using?

Are you using:

public OpenApiValidator(String openapiPath)

[vijayseid]

@GavinChenYan thanks for supporting

Using this constructor

public OpenApiValidator(InputStream openapi) {
    spec = new BufferedReader(new InputStreamReader(openapi, StandardCharsets.UTF_8)).lines().collect(Collectors.joining("\n"));
    openApiHelper = new OpenApiHelperCustom(spec);
    schemaValidator = new SchemaValidator(openApiHelper.openApi3);
}

[GavinChenYan]

@vijayseid

I had some test on my side.

And I think it could be inputstream close issue which cause the memory problem.

Basically. as long as you properly handle and close the input stream, there should not be a memory leak problem in this specific code snippet.

However, it's important to ensure that the input stream is closed correctly, especially if this code is executed multiple times or if there are other parts of your application that may not release resources properly. If the input stream is not closed in some other part of your code, it could potentially lead to memory leaks.

I created a PR for closing inputstream: https://github.com/mservicetech/openapi-schema-validation/pull/26

I am going to have a new release today.

But for your case, you are using inputstream which initialled by yourself which shouldn't be controlled by library. Can you please make sure you close the inputsteam by yourself ?

Thanks,

Gavin

[GavinChenYan]

leave it open for now

[vijayseid]

@GavinChenYan One thing to add here, I have modified the the OpenApiValidator class & used the modified class as well. In modified OpenApiValidator class,

public OpenApiValidatorCustom(String openapiPath) {
    //InputStream in = this.getClass().getClassLoader().getResourceAsStream(openapiPath);
    spec =openapiPath;
            //new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8)).lines().collect(Collectors.joining("\n"));
    openApiHelper = new OpenApiHelperCustom(spec);
    schemaValidator = new SchemaValidator(openApiHelper.openApi3);
}

Instead of passing the openapipath, I'm passing the openapi spec & directly assigned to spec. This try also giving the memory leak.

NOTE: I don't have any OpenApiHelper dependency previously, after adding that dependency, memory leaks is too better. Is that memory leak because of missing dependency.

I closed the inputstream as well, but that didn't help me. But OpenApiHelper dependency helps me to reduce 90% of memory, yeah I can say there is no leat at all, but have to verify again.

Also lowered the dependency versions & you can see the results here,

    <!-- https://mvnrepository.com/artifact/org.openapi4j/openapi-parser -->
    <dependency>
        <groupId>org.openapi4j</groupId>
        <artifactId>openapi-parser</artifactId>
        <version>1.0.5</version>
    </dependency>
    <dependency>
        <groupId>com.networknt</groupId>
        <artifactId>openapi-helper</artifactId>
        <version>2.0.32</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.networknt/json-overlay -->
    <dependency>
        <groupId>com.networknt</groupId>
        <artifactId>json-overlay</artifactId>
        <version>2.1.2</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.networknt/openapi-parser -->
    <dependency>
        <groupId>com.networknt</groupId>
        <artifactId>openapi-parser</artifactId>
        <version>2.1.2</version>
    </dependency>

Previously, old gen graph seems to be flat & slowly claims up, but now you can see the old gen graph goes up & down constantly for the constant load

Have to figure out, which helps me to fix this.

[GavinChenYan]

@vijayseid if you are using

OpenApiValidatorCustom(String openapiPath)

It may cause memory issue since the inputstream close issue. We have changed the code with try/catch resource block to close the inputstream.

For the dependency, I don't think it will cause the memory leak issue. But we will take a look @stevehu The openapi-helper has been moved to openapi-parse module. Do we have any change on it?

[vijayseid]

@GavinChenYan OpenApiValidatorCustom(String openapiSpec) With this I’m not using an inputstream… Directly passing the spec & assigned it to spec Like this, I’m sure after the dep only it resolves my memory leak issue. Will confirm it with again today or tomorrow.

public OpenApiValidatorCustom(String openapiSpec) { spec =openapiSpec; openApiHelper = new OpenApiHelperCustom(spec); schemaValidator = new SchemaValidator(openApiHelper.openApi3); }

[vijayseid]

@GavinChenYan But you also right, have to close the inputstream, after closing it performs too good & TPS also increased

[GavinChenYan]

@vijayseid Thanks for the confirmation. Please let us know the result. And we will release new version soon.

[vijayseid]

@GavinChenYan

TPS : 123.5/sec, earlier around 75-89/sec. Improved after closing the inputstream, Memory graph also seems to fine. Will plan to run for 48hrs perf test.

Will share those results after the test completes.

Thanks.