search

swagger-api / swagger-parser

Swagger Spec to Java POJOs
http://swagger.io
Apache License 2.0
777 stars 525 forks source link

OAS3 Swagger parser: Error on external http references #1142

Open gerardbosch opened 5 years ago

gerardbosch commented 5 years ago

I have my API definition in a YAML file and I tried to use absolute references to make references to common or shared API "components" (or fragments) hosted on a dedicated repository and publicly accessible via http in order to reuse those components in several API definitions.

But it looks that the generator (openapi-generator-maven-plugin) fails to reach these remote API definitions when I run it in a corporate network. I've tried the same outside the work network and it works fine. So that's why I think it could be a problem with the proxy. I must point that other maven plugins such as WGET:

<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
<version>1.4.1</version>

do work with the same network and setup, that's what I think it is not a Maven or network issue.

The result is that it can't reach the referenced components via http and generation fails.

I'm using the Maven openapi-generator-maven-plugin:3.3.4 to generate my API artifacts from YAML definitions.

The maven is run by corporate Jenkins CI (but I insist that maven wget plugin works fine in the same setup)

The remote component URL is something like this: http://myprivateorganization.foo/bitbucket/projects/FOO/repos/api-foo/raw/api-foo.yaml?at=refs%2Ftags%2Fv1.0 (which is a versioned tag of the API), so I reference it inside API definition (yaml) like the following:

components:
  schemas:
    Foo:
      type: object
      properties:
        status:
          $ref: 'http://myorganization.com/bitbucket/projects/FOO/repos/api-foo/raw/api-foo.yaml?at=refs%2Ftags%2Fv1.0#/components/schemas/SomeComponent'

I provide the stack-trace of the generation in case can help:

[INFO] --- openapi-generator-maven-plugin:3.3.4:generate (default) @ psd2-vas-contract ---
[ERROR] unable to read
java.io.FileNotFoundException: http://myprivateorganization.foo/bitbucket/projects/FOO/repos/api-foo/raw/api-foo.yaml
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0 (HttpURLConnection.java:1835)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream (HttpURLConnection.java:1440)
    at io.swagger.v3.parser.util.RemoteUrl.urlToString (RemoteUrl.java:148)
    at io.swagger.v3.parser.util.RefUtils.readExternalRef (RefUtils.java:156)
    at io.swagger.v3.parser.ResolverCache.loadRef (ResolverCache.java:118)
    at io.swagger.v3.parser.processors.ExternalRefProcessor.processRefToExternalSchema (ExternalRefProcessor.java:48)
    at io.swagger.v3.parser.processors.SchemaProcessor.processReferenceSchema (SchemaProcessor.java:201)
    at io.swagger.v3.parser.processors.SchemaProcessor.processPropertySchema (SchemaProcessor.java:102)
    at io.swagger.v3.parser.processors.SchemaProcessor.processSchemaType (SchemaProcessor.java:53)
    at io.swagger.v3.parser.processors.SchemaProcessor.processSchema (SchemaProcessor.java:38)
    at io.swagger.v3.parser.processors.ComponentsProcessor.processSchemas (ComponentsProcessor.java:221)
    at io.swagger.v3.parser.processors.ComponentsProcessor.processComponents (ComponentsProcessor.java:71)
    at io.swagger.v3.parser.OpenAPIResolver.resolve (OpenAPIResolver.java:50)
    at io.swagger.v3.parser.OpenAPIV3Parser.readLocation (OpenAPIV3Parser.java:53)
    at io.swagger.parser.OpenAPIParser.readLocation (OpenAPIParser.java:19)
    at org.openapitools.codegen.config.CodegenConfigurator.toClientOptInput (CodegenConfigurator.java:552)
    at org.openapitools.codegen.plugin.CodeGenMojo.execute (CodeGenMojo.java:607)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:497)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

[WARNING] Exception while reading:
java.lang.RuntimeException: Unable to load URL ref: http://myprivateorganization.foo/bitbucket/projects/FOO/repos/api-foo/raw/api-foo.yaml path: C:\Users\user\repos\myproject\generator-api-contracts-java\target
    at io.swagger.v3.parser.util.RefUtils.readExternalRef (RefUtils.java:183)
    at io.swagger.v3.parser.ResolverCache.loadRef (ResolverCache.java:118)
    at io.swagger.v3.parser.processors.ExternalRefProcessor.processRefToExternalSchema (ExternalRefProcessor.java:48)
    at io.swagger.v3.parser.processors.SchemaProcessor.processReferenceSchema (SchemaProcessor.java:201)
    at io.swagger.v3.parser.processors.SchemaProcessor.processPropertySchema (SchemaProcessor.java:102)
    at io.swagger.v3.parser.processors.SchemaProcessor.processSchemaType (SchemaProcessor.java:53)
    at io.swagger.v3.parser.processors.SchemaProcessor.processSchema (SchemaProcessor.java:38)
    at io.swagger.v3.parser.processors.ComponentsProcessor.processSchemas (ComponentsProcessor.java:221)
    at io.swagger.v3.parser.processors.ComponentsProcessor.processComponents (ComponentsProcessor.java:71)
    at io.swagger.v3.parser.OpenAPIResolver.resolve (OpenAPIResolver.java:50)
    at io.swagger.v3.parser.OpenAPIV3Parser.readLocation (OpenAPIV3Parser.java:53)
    at io.swagger.parser.OpenAPIParser.readLocation (OpenAPIParser.java:19)
    at org.openapitools.codegen.config.CodegenConfigurator.toClientOptInput (CodegenConfigurator.java:552)
    at org.openapitools.codegen.plugin.CodeGenMojo.execute (CodeGenMojo.java:607)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:497)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: java.io.FileNotFoundException: http://myprivateorganization.foo/bitbucket/projects/FOO/repos/api-foo/raw/api-foo.yaml
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0 (HttpURLConnection.java:1835)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream (HttpURLConnection.java:1440)
    at io.swagger.v3.parser.util.RemoteUrl.urlToString (RemoteUrl.java:148)
    at io.swagger.v3.parser.util.RefUtils.readExternalRef (RefUtils.java:156)
    at io.swagger.v3.parser.ResolverCache.loadRef (ResolverCache.java:118)
    at io.swagger.v3.parser.processors.ExternalRefProcessor.processRefToExternalSchema (ExternalRefProcessor.java:48)
    at io.swagger.v3.parser.processors.SchemaProcessor.processReferenceSchema (SchemaProcessor.java:201)
    at io.swagger.v3.parser.processors.SchemaProcessor.processPropertySchema (SchemaProcessor.java:102)
    at io.swagger.v3.parser.processors.SchemaProcessor.processSchemaType (SchemaProcessor.java:53)
    at io.swagger.v3.parser.processors.SchemaProcessor.processSchema (SchemaProcessor.java:38)
    at io.swagger.v3.parser.processors.ComponentsProcessor.processSchemas (ComponentsProcessor.java:221)
    at io.swagger.v3.parser.processors.ComponentsProcessor.processComponents (ComponentsProcessor.java:71)
    at io.swagger.v3.parser.OpenAPIResolver.resolve (OpenAPIResolver.java:50)
    at io.swagger.v3.parser.OpenAPIV3Parser.readLocation (OpenAPIV3Parser.java:53)
    at io.swagger.parser.OpenAPIParser.readLocation (OpenAPIParser.java:19)
    at org.openapitools.codegen.config.CodegenConfigurator.toClientOptInput (CodegenConfigurator.java:552)
    at org.openapitools.codegen.plugin.CodeGenMojo.execute (CodeGenMojo.java:607)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:497)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
gerardbosch commented 5 years ago

I could not solve the issue, BUT.. I took another aproach and instead of referencing the remote API-components from Bitbucket, I managed to publish my API definitions to the corporate Artifactory, which gave me a different URL to reference from the Bitbucket's one. Doing that way, the generator has been able to read remote definition references and therefore build successfully.

So I started to think more about the possibility that the "bug" is in the way the generator handles this kind of URLs: http://myorganization.com/bitbucket/projects/FOO/repos/api-foo/raw/api-foo.yaml?at=refs%2Ftags%2Fv1.0

My problem is solved so far, but I think the issue still makes sense.

P.S. The URL above can be accessed correctly with curl or browser.

lscorcia commented 4 months ago

Same issue here. The plugin doesn't respect maven proxy settings when resolving URLs, which I guess is a twofold problem - this component has no proxy settings configuration, and the plugin does not pass that config to the parser. I was able to workaround the issue by explicitly passing the proxy parameters to the maven command line:

mvn clean package -DskipTests -Dhttp.proxyHost=10.184.xxx.xxx -Dhttp.proxyPort=8080 -Dhttps.proxyHost=10.184.xxx.xxx -Dhttps.proxyPort=8080

The URLConnection class automatically picks up those parameters.