Unable to generate anything with multiple files yaml spec

JFCote commented 7 years ago

Description

I'm trying to generate the JavaPlayFramework with a very simple multiple files yaml spec but it doesn't work. I also tried with other server and I receive the same error. I was advised to try to validate the spec using the brand new "validate" function but I still receive the same error.

What is strange is that I am able to look at this spec using swagger-editor in local. There is not error at all. Just put both file in the swagger-editor folder and see for yourself.

There are 2 files (find the content of them below):

swagger.yaml
definitions.yaml

Here is the error I get every time

[main] INFO io.swagger.parser.Swagger20Parser - reading from swagger-editor/swagger.yaml
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -7
        at java.lang.String.substring(Unknown Source)
        at io.swagger.models.refs.GenericRef.computeSimpleRef(GenericRef.java:86)
        at io.swagger.models.refs.GenericRef.<init>(GenericRef.java:29)
        at io.swagger.models.properties.RefProperty.set$ref(RefProperty.java:56)
        at io.swagger.models.properties.RefProperty.<init>(RefProperty.java:18)
        at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:214)
        at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:286)
        at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:137)
        at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:39)
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3584)
        at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3508)
        at io.swagger.parser.util.SwaggerDeserializer.property(SwaggerDeserializer.java:974)
        at io.swagger.parser.util.SwaggerDeserializer.definition(SwaggerDeserializer.java:789)
        at io.swagger.parser.ResolverCache.loadRef(ResolverCache.java:135)
        at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:34)
        at io.swagger.parser.processors.ExternalRefProcessor.processRefProperty(ExternalRefProcessor.java:119)
        at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:83)
        at io.swagger.parser.processors.PropertyProcessor.processRefProperty(PropertyProcessor.java:34)
        at io.swagger.parser.processors.PropertyProcessor.processProperty(PropertyProcessor.java:21)
        at io.swagger.parser.processors.ResponseProcessor.processResponse(ResponseProcessor.java:21)
        at io.swagger.parser.processors.OperationProcessor.processOperation(OperationProcessor.java:45)
        at io.swagger.parser.processors.PathsProcessor.processPaths(PathsProcessor.java:101)
        at io.swagger.parser.SwaggerResolver.resolve(SwaggerResolver.java:50)
        at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:67)
        at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:420)
        at io.swagger.codegen.cmd.Generate.run(Generate.java:232)
        at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:43)

Swagger-codegen version

Latest master

Swagger declaration file content or url

swagger.yaml

swagger: '2.0'

info:
  version: "0.1.0"
  title: Title
  description: ""
  contact:
    name: me
    email: hello@foo.com
    url: http://www.foo.com
schemes:
  - http
  - https
basePath: /api
host: "localhost:9000"
consumes:
  - application/json
produces:
  - application/json
paths:
  /actors:
    delete:
      tags:
        - Actor
      summary: Kill an actor in the actor system (MAW)
      description: ""
      operationId: killActor
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/DefaultMetaOnlyResponse"
    get:
      tags:
        - Actor
      summary: Request the status of an actor
      description: ""
      operationId: requestPrintStatus
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/DefaultMetaOnlyResponse"

definitions.yaml

EmptyObject:
  type: object
  description: "Sample Description"
Error:
  type: object
  description: "Sample Description"
  required:
    - type
  properties:
    type:
      type: string
      description: "Sample Description"
      default: "GenericException"
      example: "GenericException"
    itemInfo:
      type: object
      description: "Sample Description"
      items:
        $ref: '#/ItemId'
    details:
      type: string
      description: "Sample Description"
      example: "Could not update that field"
    exception:
      type: string
      description: "Sample Description"
      example: "DBException + stack trace"
ItemId:
  type: object
  description: "Sample Description"
  required:
    - id
    - type
  properties:
    id:
      type: string
      description: "Sample Description"
      example: "45"
    type:
      type: string
      description: "playlist"
      example: "5667"
ResponseMeta:
  type: object
  description: "Sample Description"
  required:
    - code
  properties:
    code:
      type: string
      description: "Sample Description"
      default: Ok
      enum:
      - Ok
      example: Ok
    detail:
      type: string
      description: "Sample Description"
      example: "this is some detail about the error or the success"
    exception:
      type: string
      description: "Sample Description"
      example: "IOException + stack trace"
    errors:
      type: array
      description: "Sample Description"
      items:
        $ref: '#/Error'
ActorInfo:
  type: object
  description: "Sample Description"
  required:
    - actorPath
  properties:
    actorPath:
      type: string
      example: "/user/synchronizer"
DefaultMetaOnlyResponse:
  type: object
  required:
    - meta
  properties:
    meta:
      $ref: '#/ResponseMeta'

Command line used for generation

java -jar swagger-codegen.jar generate -i swagger-editor/swagger.yaml -l java-play-framework -o generatedServer -DhideGenerationTimestamp=true

Steps to reproduce

Just run the command line above

jimschubert commented 7 years ago

I'd never seen external refs defined outside of a definitions key. I checked swagger-parser, and it should be supported.

I ran the following against the csharp generator with a parent definitions key in your definitions.yaml, and it generated for me.

swagger.yaml:

swagger: '2.0'

info:
  version: "0.1.0"
  title: Title
  description: ""
  contact:
    name: me
    email: hello@foo.com
    url: http://www.foo.com
schemes:
  - http
  - https
basePath: /api
host: "localhost:9000"
consumes:
  - application/json
produces:
  - application/json
paths:
  /actors:
    delete:
      tags:
        - Actor
      summary: Kill an actor in the actor system (MAW)
      description: ""
      operationId: killActor
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/definitions/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/definitions/DefaultMetaOnlyResponse"
    get:
      tags:
        - Actor
      summary: Request the status of an actor
      description: ""
      operationId: requestPrintStatus
      parameters:
        - in: body
          name: body
          required: true
          schema:
            $ref: "./definitions.yaml#/definitions/ActorInfo"
      responses:
        200:
          description: OK
          schema:
            $ref: "./definitions.yaml#/definitions/DefaultMetaOnlyResponse"

definitions.yaml:

definitions:
  EmptyObject:
    type: object
    description: "Sample Description"
  Error:
    type: object
    description: "Sample Description"
    required:
      - type
    properties:
      type:
        type: string
        description: "Sample Description"
        default: "GenericException"
        example: "GenericException"
      itemInfo:
        type: object
        description: "Sample Description"
        items:
          $ref: '#/definitions/ItemId'
      details:
        type: string
        description: "Sample Description"
        example: "Could not update that field"
      exception:
        type: string
        description: "Sample Description"
        example: "DBException + stack trace"
  ItemId:
    type: object
    description: "Sample Description"
    required:
      - id
      - type
    properties:
      id:
        type: string
        description: "Sample Description"
        example: "45"
      type:
        type: string
        description: "playlist"
        example: "5667"
  ResponseMeta:
    type: object
    description: "Sample Description"
    required:
      - code
    properties:
      code:
        type: string
        description: "Sample Description"
        default: Ok
        enum:
        - Ok
        example: Ok
      detail:
        type: string
        description: "Sample Description"
        example: "this is some detail about the error or the success"
      exception:
        type: string
        description: "Sample Description"
        example: "IOException + stack trace"
      errors:
        type: array
        description: "Sample Description"
        items:
          $ref: '#/definitions/Error'
  ActorInfo:
    type: object
    description: "Sample Description"
    required:
      - actorPath
    properties:
      actorPath:
        type: string
        example: "/user/synchronizer"
  DefaultMetaOnlyResponse:
    type: object
    required:
      - meta
    properties:
      meta:
        $ref: '#/definitions/ResponseMeta'

When I run without the parent definitions key, I also receive an error:

$ ./bin/csharp-petstore.sh
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[main] INFO io.swagger.parser.Swagger20Parser - reading from /Users/jim/temp/issues/swagger-codegen/5753/swagger.yaml
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -7
    at java.lang.String.substring(String.java:1919)
    at io.swagger.models.refs.GenericRef.computeSimpleRef(GenericRef.java:86)
    at io.swagger.models.refs.GenericRef.<init>(GenericRef.java:29)
    at io.swagger.models.properties.RefProperty.set$ref(RefProperty.java:56)
    at io.swagger.models.properties.RefProperty.<init>(RefProperty.java:18)
    at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:214)
    at io.swagger.util.PropertyDeserializer.propertyFromNode(PropertyDeserializer.java:286)
    at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:137)
    at io.swagger.util.PropertyDeserializer.deserialize(PropertyDeserializer.java:39)
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3584)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3508)
    at io.swagger.parser.util.SwaggerDeserializer.property(SwaggerDeserializer.java:974)
    at io.swagger.parser.util.SwaggerDeserializer.definition(SwaggerDeserializer.java:789)
    at io.swagger.parser.ResolverCache.loadRef(ResolverCache.java:135)
    at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:34)
    at io.swagger.parser.processors.ExternalRefProcessor.processRefProperty(ExternalRefProcessor.java:119)
    at io.swagger.parser.processors.ExternalRefProcessor.processRefToExternalDefinition(ExternalRefProcessor.java:83)
    at io.swagger.parser.processors.PropertyProcessor.processRefProperty(PropertyProcessor.java:34)
    at io.swagger.parser.processors.PropertyProcessor.processProperty(PropertyProcessor.java:21)
    at io.swagger.parser.processors.ResponseProcessor.processResponse(ResponseProcessor.java:21)
    at io.swagger.parser.processors.OperationProcessor.processOperation(OperationProcessor.java:45)
    at io.swagger.parser.processors.PathsProcessor.processPaths(PathsProcessor.java:101)
    at io.swagger.parser.SwaggerResolver.resolve(SwaggerResolver.java:50)
    at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:67)
    at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:431)
    at io.swagger.codegen.cmd.Generate.run(Generate.java:239)
    at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:43)

It seems to me like this would be covered by the tests hitting relative-file-references in swagger-parser, although those are json-only and this could be a yaml issue.

You may want to track swagger-api/swagger-parser#342, as it looks like it's missing functionality (or perhaps a bug) in the YAML processing.

jimschubert commented 7 years ago

I should mention, current master was updated today (9 hours ago, issue reported 7 hours ago) to use swagger-parser 1.0.29. I tested this both with 1.0.28 and 1.0.29 and received the same error when there's no parent definitions key in the externally referenced file.

fehguy commented 7 years ago

Could be the case that we need to add a test to the parser for this. It should be handled...

JFCote commented 7 years ago

@jimschubert and @fehguy : I'm pretty sure it should be supported by swagger-codegen because it seems to be supported by the swagger-editor and swagger-ui (and probably by the OpenApi spec itself).

I followed this article on how to split a single yaml file into multiple files and as you can see, there is no more "definitions" key once it is split and it make sense.

https://apihandyman.io/writing-openapi-swagger-specification-tutorial-part-8-splitting-specification-file/

I'm pretty sure more and more people will start to split their files and will follow this blog post and swagger-codegen must be compliant.

Is this an easy fix? I don't know anything about the parser unfortunately so I don't think I can contribute myself.

jimschubert commented 7 years ago

@JFcote I don't think it's a swagger-codegen issue. The third line from the bottom of my stack trace is in CodegenConfigurator, which calls

Swagger swagger = new SwaggerParser().read(inputSpec, authorizationValues, true);

The exception that's thrown is in swagger-core, and looks like it has to do with getting the name of a reference. As @fehguy said, it should be covered.

I'm heading out on vacation today, but I'll try to look later.

jimschubert commented 7 years ago

@fehguy I tracked down the problem. This is an issue with swagger-parser. I'm on vacation for a week, so I was wondering if you could look into a fix?

The issue is that the code is getting into GenericRef#computeSimpleRef with a RefFormat.INTERNAL and RefType.DEFINITION for which the prefix is defined as #/definitions/ by the RefType enum. If an internal type definition is a shorter length than #/definitions/ count, line 82 of GenericRef:

result = ref.substring(prefix.length());

Throws an error. For example, n the sample case of this issue, it's doing "#/Error".substring("#/definitions/".length()).

It's weird to me that swagger-parser loosely allows external refs to be defined as #/Error rather than #/definitions/Error, and after stepping through the code I can understand why I've only ever seen it as #/definitions/Error.

If I rename #/Error in the sample above to #/ErrorErrorError, swagger-codegen generates successfully.

I haven't worked in the swagger-parser codebase, so I wouldn't really know where to start with the fix. Intuitively, I'd think the fix would be to just strictly enforce the prefix. However, I'm not too familiar with the spec on external refs and it looks like the test files aren't prefixed with #/definitions/. Maybe the issue is that RefFormat needs an EXTERNAL option which can be special cased in computeSimpleRef?

2ricecrackerfolder commented 7 years ago

Hi Jim,

I added the "definitions:" in the parent hierarchy as mentioned above. In the UI I was able to display my api, but when I ran the code generator in java, everything ran and the response models were not created. They were referenced in the other models but not created. They would have this name: SnapAdYamldefinitionssnapAd. Is this related? Does the code-gen not support multiple files?

Greg

jimschubert commented 7 years ago

@2ricecrackerfolder the external yaml generated for me in the local filesystem with the examples above. I didn't test it with hosted specs and external definitions Is that how you're generating?

2ricecrackerfolder commented 7 years ago

@jimschubert Thanks for your quick reply! I also modified my configuration to what they have above. The yaml did generate with no errors but was missing the response model files when i dug through the generated src.

2ricecrackerfolder commented 7 years ago

I have a large api with a lot of definitions. I am splitting the files, and it seems like the response models would not generate.

jimschubert commented 7 years ago

@2ricecrackerfolder my assumption would be that you may have a validation error somewhere.

You can check with the online validator by appending your url: https://online.swagger.io/validator/debug?url=

For example, here's a swagger definition with a validation error: https://online.swagger.io/validator/debug?url=https://quay.io/api/v1/discovery

I don't think the above definition would have a problem when generating client code, but if you have lots of errors it's more likely.

fehguy commented 7 years ago

@jimschubert looks like a bug in swagger-core is triggering this. Bummer, we just did a release today! But please see here:

https://github.com/swagger-api/swagger-core/pull/2255

fehguy commented 7 years ago

Updating to swagger-parser 1.0.30-SNAPSHOT to see if this fixes the issue

JFCote commented 7 years ago

@fehguy : Let me know when it's in the master, I will do some testing.

fehguy commented 7 years ago

You should be able to update this:

https://github.com/swagger-api/swagger-codegen/blob/master/pom.xml#L890

to 1.0.30-SNAPSHOT. Can you try that and report back? Thanks a bunch! The logic was a little weird in the current release so I'm pretty sure this will fix it.

JFCote commented 7 years ago

@fehguy : I have tested it with the example I give above and the generation was successful! So I'm pretty sure we have a good fix with the new swagger-parser version. Please update this bug when you will update the parser in the master! Thanks!

fehguy commented 7 years ago

OK Great! I'll do the release today. We have to do the whole gamut including swagger-core too...

francescobianco commented 6 years ago

@JFCote @jimschubert @fehguy I sugget to build file using command line toolkit that link multiple yaml files with $include tag . Look this https://github.com/javanile/yamlinc add in build task on gulp or grunt

JFCote commented 6 years ago

@francescobianco It's already fixed

rajkash commented 6 years ago

Hi @JFCote ,

Which version of swagger-codegen is this fixed in ? I am still getting the same error for swagger-codegen-maven-plugin 2.2.2 version

francescobianco commented 6 years ago

Hi @rajkash please use https://github.com/javanile/yamlinc to build your multiple Yaml files documentation in this way your files are quite clean and arranged as you want. I ask @JFCote @jimschubert @fehguy: why swagger parser need $ref but as this limitations:

A common misconception is that $ref is allowed anywhere in an OpenAPI specification file. Actually $ref is only allowed in places where the OpenAPI 3.0 Specification explicitly states that the value may be a reference. For example, $ref cannot be used in the info section and directly under paths:

https://swagger.io/docs/specification/using-ref/

swagger-api / swagger-codegen