[BUG] [codegen] external $ref in parameters not getting resolved

[jphorx]

Description

When attempting to generate code from an OpenAPI document that included $ref , all of the references failed to be resolved and are included in files in the same directory. I've seen other's report this issue before, but it is supposed to be resolved in the version I tested with.

openapi-generator version

3.3.4

OpenAPI declaration file content or url

https://github.com/jdegre/5GC_APIs.git

Command line used for generation

% java -jar openapi-generator-cli.jar generate -i ./TS29514_Npcf_PolicyAuthorization.yaml -g java-vertx -o ./PCF/PolicyAuth

Steps to reproduce

Working in Ubuntu 16.04 ( if that matters ).

I followed the instructions here ( https://openapi-generator.tech/docs/installation ) and installed the JAR, to pull down the 3.3.4 version of the library. I cloned OpenAPI yaml specifications from this repo: https://github.com/jdegre/5GC_APIs.git

Tried to generate with this command:

% java -jar openapi-generator-cli.jar generate -i ./TS29514_Npcf_PolicyAuthorization.yaml -g java-vertx -o ./PCF/PolicyAuth

The output was as follows:

[main] WARN io.swagger.v3.parser.OpenAPIV3Parser - Exception while reading: java.lang.NullPointerException: null at io.swagger.v3.parser.ResolverCache.updateLocalRefs(ResolverCache.java:162) at io.swagger.v3.parser.ResolverCache.loadRef(ResolverCache.java:152) at io.swagger.v3.parser.processors.ExternalRefProcessor.processRefToExternalResponse(ExternalRefProcessor.java:205) at io.swagger.v3.parser.processors.ResponseProcessor.processReferenceResponse(ResponseProcessor.java:76) at io.swagger.v3.parser.processors.ResponseProcessor.processResponse(ResponseProcessor.java:38) at io.swagger.v3.parser.processors.OperationProcessor.processOperation(OperationProcessor.java:56) at io.swagger.v3.parser.processors.PathsProcessor.processPaths(PathsProcessor.java:83) at io.swagger.v3.parser.OpenAPIResolver.resolve(OpenAPIResolver.java:49) at io.swagger.v3.parser.OpenAPIV3Parser.readLocation(OpenAPIV3Parser.java:53) at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:19) at org.openapitools.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:552) at org.openapitools.codegen.cmd.Generate.run(Generate.java:354) at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:62) [main] WARN o.o.codegen.utils.ModelUtils - Failed to get the schema name: ./TS29571_CommonData.yaml#/components/schemas/RouteToLocation [main] WARN o.o.codegen.utils.ModelUtils - Failed to get the schema name: ./TS29571_CommonData.yaml#/components/schemas/RouteToLocation [main] WARN o.o.codegen.utils.ModelUtils - Failed to get the schema name: ./TS29571_CommonData.yaml#/components/schemas/PresenceInfo [main] WARN o.o.codegen.utils.ModelUtils - Failed to get the schema name: ./TS29571_CommonData.yaml#/components/schemas/PresenceInfo . . LOTS more warnings like these, omitted to keep this concise . [main] WARN o.o.codegen.utils.ModelUtils - Failed to get the schema name: ./TS29571_CommonData.yaml#/components/responses/503 [main] WARN o.o.codegen.utils.ModelUtils - Failed to get the schema name: ./TS29571_CommonData.yaml#/components/responses/default Exception in thread "main" org.openapitools.codegen.SpecValidationException: There were issues with the specification. The option can be disabled via validateSpec (Maven/Gradle) or --skip-validate-spec (CLI). | Error count: 1, Warning count: 0 Errors: -null

at org.openapitools.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:569)
at org.openapitools.codegen.cmd.Generate.run(Generate.java:354)
at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:62)

Related issues/PRs

This one seems very similar: https://github.com/OpenAPITools/openapi-generator/issues/455

Suggest a fix

[auto-labeler[bot]]

👍 Thanks for opening this issue! 🏷 I have applied any labels matching special text in your issue.

The team will review the labels and make any necessary changes.

[loganknecht]

+1 on this. I really wish I could use references 😭

[jmini]

From the stacktrace, it seems that the problem is located in Swagger-Parser library.

Please have a look at how Swagger-Parser is parsing your file and try to produce a minimal example. Sample Java Snippet:

OpenAPIParser openApiParser = new OpenAPIParser();
ParseOptions options = new ParseOptions();
options.setResolve(true);
options.setFlatten(true);

OpenAPI openAPI = openApiParser.readLocation(inputSpec, null, options).getOpenAPI();
String string = Yaml.mapper().writerWithDefaultPrettyPrinter().writeValueAsString(openAPI);

System.out.println(string);

Maven coordinates of the dependency for your test project:

<dependency>
    <groupId>io.swagger.core.v3</groupId>
    <artifactId>swagger-core</artifactId>
    <version>2.0.6</version>
</dependency>

Let me know if you need help.

[jphorx]

I've modified the petstore.yaml example to reproduce the issue. Original found here: https://github.com/OAI/OpenAPI-Specification/tree/master/examples/v3.0/petstore.yaml.

petstore.zip

[jphorx]

As suggested, I wrote some code to directly parse the OpenAPI spec files from this location ( https://github.com/jdegre/5GC_APIs.git ), with the target being a locally corrected version of TS29514_Npcf_PolicyAuthorization.yaml.

When using the swagger-parser directly, I was able to consume this yaml, including the external references successfully.

The crude test program I wrote to try processing the same yaml files is attached as well, just run the jar and add an argument that is the qualified yaml file name. swagger-parser-bug.zip

I've attached just the files yaml files I used below 5GC_APIs.zip

[jphorx]

I've modified the petstore.yaml example to reproduce the issue. Original found here: https://github.com/OAI/OpenAPI-Specification/tree/master/examples/v3.0/petstore.yaml.

petstore.zip

I found a mistake in these yaml files, they should be ignored.

[jphorx]

I have re-tested this with the v4.0.0-beta2 release and was able to process all of the models, so this was addressed somehow?

[jweisman]

My test is not working with the latest v4.0.0-beta2 release.

java -jar modules/openapi-generator-cli/target/openapi-generator-cli.jar generate -i https://raw.githubusercontent.com/swagger-api/swagger-codegen/50d5cf092cb05e76d88f8900664c0a248fdd0d95/modules/swagger-codegen/src/test/resources/3_0_0/petstore.json -g ruby -o ~/tmp/ruby

Results in:

...
[main] WARN  o.o.codegen.utils.ModelUtils - Failed to get the schema name: #/ExtUser
[main] WARN  o.o.codegen.utils.ModelUtils - #/ExtUser is not defined
...

In swagger-codegen I had to use setResolveFully to get it to work and added a PR for that setting.

[afobo]

I had the same issue with openapi-generator-maven-plugin version 3.3.4 and 5G API spec (namely, TS29510_Nnrf_NFManagement.yaml). Root cause is the issue in swagger-parser fixed in swagger-api/swagger-parser@a989c2a

So, upgrade of swagger-parser to 2.0.5 solved the problem:

        <plugin>
            <groupId>org.openapitools</groupId>
            <artifactId>openapi-generator-maven-plugin</artifactId>
            <version>3.3.4</version>
            <dependencies>
                <dependency>
                    <groupId>io.swagger.parser.v3</groupId>
                    <artifactId>swagger-parser</artifactId>
                    <version>2.0.5</version>
                </dependency>
            </dependencies>
        </plugin>

[jmini]

Can you retry with 4.0.0-beta3? we have updated the swagger-parser version with #2262.

[Mistgun]

@jmini I have encountered similar issue, on v.4.0.0-beta3 it happens:

[main] WARN  io.swagger.v3.parser.OpenAPIV3Parser - Exception while reading:
java.lang.RuntimeException: Unable to load RELATIVE ref: ../../../components.yml path:
  at io.swagger.v3.parser.util.RefUtils.readExternalRef(RefUtils.java:204)
  at io.swagger.v3.parser.ResolverCache.loadRef(ResolverCache.java:119)
  at io.swagger.v3.parser.processors.ParameterProcessor.processParameters(ParameterProcessor.java:85)
  at io.swagger.v3.parser.processors.OperationProcessor.processOperation(OperationProcessor.java:39)
  at io.swagger.v3.parser.processors.PathsProcessor.processPaths(PathsProcessor.java:84)
  at io.swagger.v3.parser.OpenAPIResolver.resolve(OpenAPIResolver.java:49)
  at io.swagger.v3.parser.OpenAPIV3Parser.readLocation(OpenAPIV3Parser.java:53)
  at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:19)
  at org.openapitools.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:606)
  at org.openapitools.codegen.cmd.Generate.run(Generate.java:367)
  at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:60)
Caused by: java.lang.RuntimeException: Could not find ../../../components.yml on the classpath
  at io.swagger.v3.parser.util.ClasspathHelper.loadFileFromClasspath(ClasspathHelper.java:31)
  at io.swagger.v3.parser.util.RefUtils.readExternalRef(RefUtils.java:198)
  ... 10 common frames omitted
Exception in thread "main" org.openapitools.codegen.SpecValidationException: There were issues with the specification. The option can be disabled via validateSpec (Maven/Gradle) or --skip-validate-spec (CLI).
 | Error count: 1, Warning count: 0
Errors:
  -Unable to load RELATIVE ref: ../../../components.yml path: /Users/x/x/doc

  at org.openapitools.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:626)
  at org.openapitools.codegen.cmd.Generate.run(Generate.java:367)
  at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:60) 

Even though, validation command passes successfully.. I tried the same with version 3.2.0 and it worked.

[jweisman]

Hi @jmini - it's still not working in v4.0.0-beta3 release.

java -jar modules/openapi-generator-cli/target/openapi-generator-cli.jar generate -i https://raw.githubusercontent.com/swagger-api/swagger-codegen/50d5cf092cb05e76d88f8900664c0a248fdd0d95/modules/swagger-codegen/src/test/resources/3_0_0/petstore.json -g ruby -o ~/tmp/ruby

Results in:

...
[main] WARN  o.o.codegen.utils.ModelUtils - Failed to get the schema name: #/ExtUser
[main] WARN  o.o.codegen.utils.ModelUtils - Failed to get the schema name: #/ExtUser
...

The test definition is available so you can run the same command to test it. Thanks!

[jmini]

@jweisman Thank you for the feedback... Sorry for the delay. I am trying to understand what is wrong here. Were is the spec that you are using as Input?

[cvgaviao]

@jmini, this issue was resolved only in the last week in the swagger-parser repository. I think you have not published that yet...

[jmini]

I have prepared PR https://github.com/OpenAPITools/openapi-generator/pull/2775 to update Swagger-Parser

[jweisman]

@jmini - the spec I'm using is https://raw.githubusercontent.com/swagger-api/swagger-codegen/50d5cf092cb05e76d88f8900664c0a248fdd0d95/modules/swagger-codegen/src/test/resources/3_0_0/petstore.json

The offending line is:

"application/json": {
              "schema": {
                "$ref": "./ext_user.json#/ext_user"
              }

The referenced file exists in the same path.

[jmini]

I have updated Swagger-Parser. Can you check the latest 4.0.0-SNAPSHOT version of OpenAPI Generator?

[lacksfish]

For the impatient:

To fix my problem, I've used https://github.com/APIDevTools/json-schema-ref-parser (after turning my main yaml spec into json) and de-referenced all the $ref's automatically with the aforementioned tool.

Then, I was able to generate the client code :)

PSA: This is a hotfix and any code generated should be well tested.

[jweisman]

Hi @jmini - I downloaded the (now release) version of 4.0.0 and I'm getting the same behavior.

java -jar modules/openapi-generator-cli/target/openapi-generator-cli.jar generate -i https://raw.githubusercontent.com/swagger-api/swagger-codegen/50d5cf092cb05e76d88f8900664c0a248fdd0d95/modules/swagger-codegen/src/test/resources/3_0_0/petstore.json -g ruby -o ~/tmp/ruby

Results in:

...
[main] WARN  o.o.codegen.utils.ModelUtils - Failed to get the schema name: #/ExtUser
[main] WARN  o.o.codegen.utils.ModelUtils - Failed to get the schema name: #/ExtUser
...

The test definition is available so you can run the same command to test it. Thanks!

Note also that:

In swagger-codegen I had to use setResolveFully to get it to work and added a PR for that setting.

[ghilainm]

I have the same problem with multiple level of JSON references using only local files with version 4.0.0. Settings in ParseOptions resolve = true and resolveFully = true seems to solve the issue.

I had to override the class on the classpath to be able to change this option. It should maybe be exposed in the plugin options in the future.

[jmini]

With 4.0.3 we have updated Swagger-Parser, and I think that they now better handle reference. Can you please retry?

[jmini]

In swagger-codegen I had to use setResolveFully to get it to work

The problem with setResolveFully is that it is de-referencing the $ref:

With this input:

openapi: 3.0.1
info:
  title: ping test
  version: '1.0'
servers:
  - url: 'http://localhost:9999/'
paths:
  /ping:
    get:
      operationId: pingGet
      responses:
        '201':
          description: OK
          content:
            '*/*':
               schema:
                 $ref: "#/components/schemas/MainObj"
components:
  schemas:
    SomeObj:
      type: object
      properties:
        id:
          type: integer
          format: int64
    MainObj:
      type: object
      properties:
        lorem:
          $ref: "#/components/schemas/SomeObj"
        ipsum:
          type: string

The output of Swagger-Parser is

openapi: 3.0.1
info:
  title: ping test
  version: "1.0"
servers:
- url: http://localhost:9999/
paths:
  /ping:
    get:
      operationId: pingGet
      responses:
        201:
          description: OK
          content:
            '*/*':
              schema:
                type: object
                properties:
                  lorem:
                    type: object
                    properties:
                      id:
                        type: integer
                        format: int64
                  ipsum:
                    type: string
components:
  schemas:
    SomeObj:
      type: object
      properties:
        id:
          type: integer
          format: int64
    MainObj:
      type: object
      properties:
        lorem:
          type: object
          properties:
            id:
              type: integer
              format: int64
        ipsum:
          type: string

In the operation, there is no information anymore about the models defined in components/schema that are used in pingGet operation. The components section could be completely ignored (the schemas there are unused).

I did not try it, but my guess is that OpenAPI-Generator will generator other models than the one defined in components/schema.

I do not believe that this is what the users want.

[jweisman]

Hi @jmini - same with 4.0.3.

java -jar modules/openapi-generator-cli/target/openapi-generator-cli.jar generate -i https://raw.githubusercontent.com/swagger-api/swagger-codegen/50d5cf092cb05e76d88f8900664c0a248fdd0d95/modules/swagger-codegen/src/test/resources/3_0_0/petstore.json -g ruby -o ~/tmp/ruby

Results in:

...
[main] WARN  o.o.codegen.utils.ModelUtils - Failed to get the schema name: #/ExtUser
[main] WARN  o.o.codegen.utils.ModelUtils - Failed to get the schema name: #/ExtUser
...

The test definition is available so you can run the same command to test it. Thanks!

Out of curiosity- does this sample spec work for you? Perhaps it's something I'm doing wrong?

[nmandya]

Is there any update on this? I just tried with both version 4.2.2 and 4.2.3 and the schemas defined under components/schemas and referenced using $ref are not included in the generated HTML documentation. Thanks.

[stephengroat]

Also having this issue, tried to make a replica case in swagger-parser with https://github.com/swagger-api/swagger-parser/pull/1338

[RaduFurnea]

I'm having the same problem when migrating an existent spec from version 2 to 3, and I would really like to avoid having to move all models defined in the common file to each spec file.

[lacksfish]

If you scroll up a bit, I offered a hacky quick fix, which worked for me.

[stephengroat]

thanks for the reminder @lacksfish

here's a small node script that implements the idea, run with node app.js INPUT_FILE_JSON OUTPUT_FILE_JSON:

const $RefParser = require("@apidevtools/json-schema-ref-parser");
const fs = require('fs');

let args = process.argv.slice(2);
let rawdata = fs.readFileSync(args[0]);
let input = JSON.parse(rawdata);
$RefParser.dereference(input, (err, schema) => {
  if (err) {
    console.error(err);
  }
  else {
    let data = JSON.stringify(schema, null, 2);
    fs.writeFileSync(args[1], data);
  }
})

[mutech]

What is the status of this issue?

EDIT: From what I can see, the problem is that when resolving repeated refs, the parser uses the original base URL (the root) instead or resolving relative to the referencing document.

Sorry, I'm ranting a bit here.

I would think such a file resolution issue would be important enough to fix it before 20 months.

I'm currently generating a JSON-schema for each tool that uses schema and falls apart for different reasons, I have sliced, slashed, splitted, merged and partially merged schema, I am patching output with regexes and I am now doing the same for each openapi tool I am using, because no single tool is able to understand any schema dialect and handle it correctly.

I so miss the time when I could work with XSD.