Trying to parse and map a particular Open API spec crashes the APP due to memory

[mkmishra89]

We are using AMF parser version 5.0.1 in java and below is our code to parse and transform oas file

our application is getting crash due to memory issue. Could you please help us. oas file size is 3.7MB and memory allocated in CF is 1024 MB.

below is the sample code which we are using to parse and transform the oas file.

@DisplayName("AMF Parser demo") @Test void parseOASFile() throws ExecutionException, InterruptedException { final AMFBaseUnitClient client = OASConfiguration.OAS30().baseUnitClient(); final BaseUnit unresolvedModel = client.parse("file://src/test/resources/tags/openapispecwithtags.json").get().baseUnit(); final BaseUnit resolvedModel = client.transform(unresolvedModel).baseUnit(); final Document document = (Document) resolvedModel; final WebApi api = (WebApi) document.encodes(); //to read api type Optional domainExtension = api.customDomainProperties().stream().filter(property -> property.definedBy().name().value().contains("api-type") ).findAny(); domainExtension.ifPresent(extension -> String.valueOf(((ScalarNode) extension.extension()).value())); //process each endpoint api.endPoints().forEach(endPoint -> endPoint.operations().forEach(currOperation -> { readValuesForEachOperation(currOperation, endPoint.path().value()); })); }

private void readValuesForEachOperation(Operation currOperation, String value) {
    // reads server details
    AMFElementClient client = OASConfiguration.OAS30().elementClient();
    currOperation.servers().forEach(server -> {
        JsonOutputBuilder<StringWriter> jsonBuilder = JsonOutputBuilder.apply();
        client.renderToBuilder(server, jsonBuilder);
        JSONObject serverJson = new JSONObject(jsonBuilder.result().toString());
    });
    //read securitySchemes
    document.declares().forEach(domainElement -> {
                if (domainElement instanceof SecurityScheme) {
                    SecurityScheme securityScheme = (SecurityScheme) domainElement;
                }
            });
    //read request objects
            //currOperation.request().queryParameters()
            //currOperation.request().headers()
            //currOperation.request().uriParameters()
         Optional<Payload> payload = currOperation.request().payloads().stream().filter(p ->
            p.mediaType().value().equalsIgnoreCase(APPLICATION_JSON)).findAny();
         //transform payload into another object
    // read response objects
    currOperation.responses().forEach(response -> readResponse(response));
}

private void readResponse(Response response) {
    // read response headers
    for (Parameter header : response.headers()) {
        //do something
    }
    //response payload
    Optional<Payload> payload = response.payloads().stream().filter(p ->
                    p.mediaType().value().equalsIgnoreCase(APPLICATION_JSON))
            .findAny();
    String jsonSchema = OASConfiguration.OAS30().elementClient().toJsonSchema((AnyShape) payload.get().schema());
    // read this json object and create another object

}

[mkmishra89]

We have added below code for references

[mkmishra89]

@tomsfernandez could you please have a look.

[mkmishra89]

sampleoas.txt

[Vivek-Vishal]

@tomsfernandez could you please have a look.

Hello @tomsfernandez , Request your assistance to help on this issue. It is a blocker for us as we are unable to work with any open api specifications of higher file sizes.

[mkmishra89]

@tomsfernandez Could you please help us on the above issue. It is blocker for us not able to work with big file.

Please let us know if you need any details from our end.

[mkmishra89]

@tomsfernandez @jstoiko could you please help us on the above issue it blocker for us.

[mkmishra89]

@arielmirra @hghianni could you please look above issue and help us for the same.

[tomsfernandez]

Hello @mkmishra89. We understand that this issue is a blocker for you and your team but we haven't had the opportunity to check this out yet. I'll try to check it out today. Please stop the spamming as it won't change how fast we resolve / check this issue out.

[tomsfernandez]

@mkmishra89 The next time please follow the bug issue template

[tomsfernandez]

@arielmirra will check this when he has some available time

[arielmirra]

Hi @mkmishra89! I've been investigating your case, running the provided API through several profilers and analyzing time, CPU and memory usage.

Here are the results with AMF in Java, and only parsing, without transformation:

And here are the results when parsing and transforming the API:

The graphs show that there isn't any unusual behavior in AMF regarding resources usage, but rather that it needs a lot of resources to parse the provided API, which is normal with APIs this large, especially when there are a lot of types, inheritances, and endpoints. In total, your API is 86,833 lines long, which proves to be a challenge for any parser. Even the swagger official editor freezed up when trying to parse it:

My recommendations are, in no particular order:

• Try to simplify the API:
  • reduce the amount of types and endpoints being used
  • in terms of file transfer and memory allocation, some memory can be saved by removing all examples, removing or reducing descriptions/summaries/tags or any documentation fields that do not affect API functionality.
• If the API it's still very large after applying some changes, an increase in resources should be the next logical option.

I'm closing this issue as it's not a bug, reducing memory allocation is an enhancement request that we already have and been working on.

Hope this analysis provided useful. Thank you!