search

redskap / swagger-brake

Swagger contract checker for breaking API changes
Apache License 2.0
57 stars 16 forks source link

java.lang.OutOfMemoryError upon first try #67

Closed pkunze closed 1 year ago

pkunze commented 1 year ago

I just tried out swagger-brake and am getting the following Exception:

java -jar .\swagger-brake-2.3.0-cli.jar --old-api=swagger_3.1.json --new-api=swagger_3.2.json  --output-formats=STDOUT,JSON,HTML
Loading old API from swagger_3.1.json
Loading new API from swagger_3.2.json
Successfully loaded APIs
Starting the check for breaking API changes
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.TreeMap.addEntry(Unknown Source)
        at java.base/java.util.TreeMap.put(Unknown Source)
        at java.base/java.util.TreeMap.put(Unknown Source)
        at java.base/java.util.TreeSet.add(Unknown Source)
        at java.base/java.util.AbstractCollection.addAll(Unknown Source)
        at java.base/java.util.TreeSet.addAll(Unknown Source)
        at java.base/java.util.TreeSet.<init>(Unknown Source)
        at io.redskap.swagger.brake.core.model.SchemaBuilder.build(SchemaBuilder.java:109)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.transformSchema(SchemaTransformer.java:105)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.internalTransform(SchemaTransformer.java:53)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.lambda$getSchemaAttributes$0(SchemaTransformer.java:121)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer$$Lambda$138/0x0000000800e3fda0.apply(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.getSchemaAttributes(SchemaTransformer.java:126)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.transformSchema(SchemaTransformer.java:100)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.internalTransform(SchemaTransformer.java:53)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.lambda$getSchemaAttributes$0(SchemaTransformer.java:121)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer$$Lambda$138/0x0000000800e3fda0.apply(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)

I would glady share both json files, but I'd rather do so privately via e-mail. If this is relevant: we are generating the everything swagger related using swashbuckle on asp.net core 6.0.

galovics commented 1 year ago

Hi @pkunze, thanks for reporting the issue.

Have you tried specifying the Xmx for the JVM? Try it like this:

java -Xmx1024m -jar .\swagger-brake-2.3.0-cli.jar --old-api=swagger_3.1.json --new-api=swagger_3.2.json  --output-formats=STDOUT,JSON,HTML

This'll give the JVM 1024 MB of memory. If still experiencing the issue, you could try increasing this number but that'll indicate a huge API file (which shouldn't be an issue, just calling it out).

Let me know.

pkunze commented 1 year ago

Hi @galovics,

Thanks for the quick reply! I just did as instructed resulting in 

java -Xmx1024m -jar .\swagger-brake-2.3.0-cli.jar --old-api=swagger_3.1.json --new-api=swagger_3.2.json  --output-formats=STDOUT,JSON,HTML
Loading old API from swagger_3.1.json
Loading new API from swagger_3.2.json
Successfully loaded APIs
Starting the check for breaking API changes
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.stream.ReferencePipeline.map(Unknown Source)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.getSchemaAttributes(SchemaTransformer.java:119)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.transformSchema(SchemaTransformer.java:100)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.internalTransform(SchemaTransformer.java:53)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.lambda$getSchemaAttributes$0(SchemaTransformer.java:121)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer$$Lambda$138/0x0000000800e38468.apply(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.getSchemaAttributes(SchemaTransformer.java:126)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.transformSchema(SchemaTransformer.java:100)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.internalTransform(SchemaTransformer.java:53)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.lambda$getSchemaAttributes$0(SchemaTransformer.java:121)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer$$Lambda$138/0x0000000800e38468.apply(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.getSchemaAttributes(SchemaTransformer.java:126)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.transformSchema(SchemaTransformer.java:100)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.internalTransform(SchemaTransformer.java:53)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.internalTransform(SchemaTransformer.java:43)
        at io.redskap.swagger.brake.core.model.transformer.SchemaTransformer.lambda$getSchemaAttributes$0(SchemaTransformer.java:121)

The API Files are approx. 288KB If that matters. I also tried using a lower mayor version of swagger-brake with the same result. Let me knowif you need the files and if so where I can send them to :)

pkunze commented 1 year ago

Also... it feels like some kind of infinite loop. The Console where I executed the command pretty much ate up all my notebooks ram (consumed more then 8GB according to task manager) and cpu (99%) 🚀🚀🚀

galovics commented 1 year ago

Thanks @pkunze, it seems like there's an infinite loop for sure. You can send the file here arnold_galovics (at) docktape (dot) com

pkunze commented 1 year ago

I did, thank you!

pkunze commented 1 year ago

@galovics did you already find time to have a look at this? I would gladly take a look myself if it is not a super-complicated fix, however, I guess if you want me to I would need some help getting started since I am not a Java developer ;)

galovics commented 1 year ago

@pkunze I'm playing around with it. I even tried to run your OpenAPI contracts with 10G memory but it's still not enough. I'm still trying to figure out whats going on.

pkunze commented 1 year ago

I tried it with 32gb as well with the same result.

I will also try to narrow down what endpoint/schema part might cause this.

galovics commented 1 year ago

Yeah, I think it's probably a recursive schema somewhere but I wasn't able to narrow down yet either.

galovics commented 1 year ago

I'm still analyzing memory dumps but I can already see that one of the problems in the OpenAPI definition you've shared is the fact that it's having a combinatorial explosion in terms of attributes. For example let's consider this as an attribute: declaredType.assembly.definedTypes.baseType.assembly.customAttributes.constructor.module.assembly.definedTypes.assembly.entryPoint.module.assembly.definedTypes.typeInitializer.module.assembly.entryPoint.methodHandle.value

Now imagine how many other attribute combinations there can be; and this is for many of the attributes. I've made some minor optimizations but it's still not analyzable for swagger-brake.

pkunze commented 1 year ago

Are attributes a Swagger/OpenAPI concept or do you mean .NET Attributes? In the latter case I could try find and exclude the affected schema parts.

galovics commented 1 year ago

I mean the Swagger schema attributes. As you can see in my previous comment, the Assembly schema has been reused in many other schema definitions and for all of those there's a number of attribute combinations.

For some testing purposes, I've used a 2MB big API definition to see if swagger-brake can cope with it. I did introduce a few optimizations but works, even without increasing the max heap size: https://github.com/redskap/swagger-brake/pull/80

Going back, I think the reason that your definition runs out of memory is the number of attribute combinations since everything is referencing everything. I'm not really sure how to solve it short-term without refactoring the whole transformation phase along with the breaking rule checks but that'd be a huge work.

pkunze commented 1 year ago

I just found this which seems to reflect what you describe. I'll play around with it next week and let you know if there is any progress (currently at the beach 😎🏝️).

galovics commented 1 year ago

Oh, so this is generated schema from C#; that answers it. Yeah, internal classes have been included in the schema definitions which in this case is wrong (I thought you want to mimic a very generic API).

Let me know if you find anything.

P.S. enjoy your time off :))

pkunze commented 1 year ago

I can now confirm that swagger-break works, when fixing the mentioned issue, where sort of internal .NET System Types leaked into the checked json file.

I guess we can fix this issue. Thanks for your help!

galovics commented 1 year ago

Sounds good @pkunze, thanks for the info.