search

swagger-api / swagger-codegen

swagger-codegen contains a template-driven engine to generate documentation, API clients and server stubs in different languages by parsing your OpenAPI / Swagger definition.
http://swagger.io
Apache License 2.0
16.73k stars 6.02k forks source link

[ANY LANG] The incoming YAML document exceeds the limit: 3145728 code points. #11980

Open lorre851 opened 1 year ago

lorre851 commented 1 year ago
Description

When executing the following command: java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i ../swagger.json -l html2 -o target

The following exception occurs, no mather which language (-l) is exported to:

io.swagger.v3.parser.util.DeserializationUtils$SnakeException: Exception safe-checking yaml content  (maxDepth 2000, maxYamlAliasesForCollections 2147483647)
    at io.swagger.v3.parser.util.DeserializationUtils$CustomSnakeYamlConstructor.getSingleData(DeserializationUtils.java:438)
    at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:477)
    at org.yaml.snakeyaml.Yaml.load(Yaml.java:406)
    at io.swagger.v3.parser.util.DeserializationUtils.readYamlTree(DeserializationUtils.java:211)
    at io.swagger.v3.parser.util.DeserializationUtils.deserializeIntoTree(DeserializationUtils.java:143)
    at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:165)
    at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:104)
    at io.swagger.v3.parser.converter.SwaggerConverter.readResult(SwaggerConverter.java:111)
    at io.swagger.v3.parser.converter.SwaggerConverter.readLocation(SwaggerConverter.java:85)
    at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:16)
    at io.swagger.codegen.v3.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:612)
    at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:386)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points.
    at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:342)
    at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:263)
    at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:694)
    at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:185)
    at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:57)
    at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:43)
    at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:136)
    at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:116)
    at org.yaml.snakeyaml.composer.Composer.composeScalarNode(Composer.java:239)
    at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:208)
    at org.yaml.snakeyaml.composer.Composer.composeKeyNode(Composer.java:347)
    at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:332)
    at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
    at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
    at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
    at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
    at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
    at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
    at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
    at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
    at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
    at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
    at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
    at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
    at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
    at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
    at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
    at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
    at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
    at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
    at org.yaml.snakeyaml.composer.Composer.getNode(Composer.java:134)
    at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:160)
    at io.swagger.v3.parser.util.DeserializationUtils$CustomSnakeYamlConstructor.getSingleData(DeserializationUtils.java:415)
    ... 12 common frames omitted
14:39:19.137 [Thread-0] ERROR i.s.v.p.util.DeserializationUtils - Error parsing content
com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: The incoming YAML document exceeds the limit: 3145728 code points.
 at [Source: (StringReader); line: 99391, column: 16]
    at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:425)
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:539)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:98)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:23)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
    at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4772)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3124)
    at io.swagger.v3.parser.util.DeserializationUtils.readYamlTree(DeserializationUtils.java:232)
    at io.swagger.v3.parser.util.DeserializationUtils.deserializeIntoTree(DeserializationUtils.java:143)
    at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:165)
    at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:104)
    at io.swagger.v3.parser.converter.SwaggerConverter.readResult(SwaggerConverter.java:111)
    at io.swagger.v3.parser.converter.SwaggerConverter.readLocation(SwaggerConverter.java:85)
    at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:16)
    at io.swagger.codegen.v3.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:612)
    at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:386)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points.
    at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:342)
    at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:263)
    at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:694)
    at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:185)
    at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:195)
    at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:419)
    ... 16 common frames omitted
14:39:19.311 [Thread-0] WARN  io.swagger.v3.parser.OpenAPIV3Parser - Exception while parsing:
com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: The incoming YAML document exceeds the limit: 3145728 code points.
 at [Source: (StringReader); line: 99391, column: 16]
    at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:425)
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:539)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:98)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:23)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
    at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4772)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3124)
    at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:167)
    at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:104)
    at io.swagger.v3.parser.converter.SwaggerConverter.readResult(SwaggerConverter.java:111)
    at io.swagger.v3.parser.converter.SwaggerConverter.readLocation(SwaggerConverter.java:85)
    at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:16)
    at io.swagger.codegen.v3.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:612)
    at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:386)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points.
    at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:342)
    at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:263)
    at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:694)
    at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:185)
    at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:195)
    at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:419)
    ... 14 common frames omitted
14:39:19.314 [Thread-0] INFO  i.s.c.v.i.CodegenIgnoreProcessor - No .swagger-codegen-ignore file found.
Exception in thread "Thread-0" java.lang.RuntimeException: missing OpenAPI input!
    at io.swagger.codegen.v3.DefaultGenerator.generate(DefaultGenerator.java:777)
    at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:388)
    at java.base/java.lang.Thread.run(Thread.java:833)

The swagger.json for our monolith of an application is 145 288 lines long.

Swagger-codegen version

3.0.36

Swagger declaration file content or url

I am not a liberty to share this .json file due to an NDA. The file has 145 288 lines and is about 4MB in size.

Command line used for generation

java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i ../swagger.json -l html2 -o target

openjdk 17.0.5 2022-10-18 Ubuntu 22.04.1 LTS x86_64

Steps to reproduce
  1. execute the command above with a large swagger.json file
  2. see the exception occur that's mentioned above
Related issues/PRs

N/A

Suggest a fix/enhancement

Increase upper limit or make configurable.

skwashp commented 1 year ago

snakeyaml introduced this change to patch CVE-2022-25857. Given the use case for swagger-codegen there should be a better default or a configuration option to control this.

oprudkyi commented 1 year ago

it seems like it has been fixed.

something like 

/usr/bin/java -DmaxYamlCodePoints=99999999 -jar /opt/swagger-codegen/swagger-codegen-cli-3.jar generate ...

works without errors

nishuiaee commented 10 months ago

it seems like it has been fixed.

something like

/usr/bin/java -DmaxYamlCodePoints=99999999 -jar /opt/swagger-codegen/swagger-codegen-cli-3.jar generate ...

works without errors java -DmaxYamlCodePoints=99999999 -cp cus-openapi-generator-1.0.0.jar:openapi-generator-cli.jar \ org.openapitools.codegen.OpenAPIGenerator generate Exception safe-checking yaml content (maxDepth 2000, maxYamlAliasesForCollections 2147483647) it also error