Generate model classes from the OpenAPI specification

[HofmeisterAn]

@galvesribeiro Maintaining parts of the Docker API (models) by hand is cumbersome and error prone (#577). As you have mentioned in #568 you are working on a new implementation to generate the model classes.

Why don't we use the existing OpenAPI specification instead? OpenAPI can generate the classes for us.

[galvesribeiro]

That is what I thought we should use. The problem os that last time I've checked, no .Net yaml parsers was able to parse those files from Docker as they were no standards.

I don't believe just the OpenAPI manifest is enough to do all we do Today but, indeed, if we can parse their yaml and have all models without relying on the Go code, I'm all up for it. Then we can source generate with Roslyn the remaining pieces.

So, in order to validade that, can you make sure any Yaml parser like YamlDotNet can parse those files properly?

If they fixed it, then we're good to have the model extracted from that Yaml and then complete it with our custom generated code.

[HofmeisterAn]

That is what I thought we should use. The problem os that last time I've checked, no .Net yaml parsers was able to parse those files from Docker as they were no standards.

It's part of OpenAPI, generated by openapi-generator-cli-5.0.0.jar. It supports many different languages. They generate the API, model, client and even scaffold tests.

So, in order to validade that, can you make sure any Yaml parser like YamlDotNet can parse those files properly?

If they fixed it, then we're good to have the model extracted from that Yaml and then complete it with our custom generated code.

I don't understand why we need that (until now), but I can take a look at it. The YML looks fine though (linting).

[galvesribeiro]

That generator create non-optimized code and We really dont want to rely on Java tools. We need .Net native tooling for this.

The best of worlds would be to generate the POCO types for the requests. The real client we would handcraft and use HttpClientFactory.

If you can read their Yaml and generate the models as pure POCOs with Roslyn we can optimize the client and even generate parts of it as well.

A Source generator which creates the POCOs from the yaml file and customize endpoints as we seem fit, would bring us to another level of maintainability having only the Yaml as our dependency.

[HofmeisterAn]

If you can read their Yaml and generate the models as pure POCOs with Roslyn we can optimize the client and even generate parts of it as well.

Microsoft.OpenApi.Readers looks promising. We get all the information we are maintaining by hand in the Go files (modeldefs.go, specgen.go) and even more. What do think about that?

Here is a small example:

const string OpenApiSpecUri = "https://docs.docker.com/engine/api/v1.41.yaml";

using (var httpClient = new HttpClient())
{
  using (var openApiSpecStream = await httpClient.GetStreamAsync(OpenApiSpecUri)
    .ConfigureAwait(false))
  {
    var openApiSpecStreamReader = new OpenApiStreamReader(new OpenApiReaderSettings());

    var openApiSpec = await openApiSpecStreamReader.ReadAsync(openApiSpecStream)
      .ConfigureAwait(false);

    foreach (var path in openApiSpec.OpenApiDocument.Paths)
    {
      foreach (var operation in path.Value.Operations)
      {
        var parameters = operation.Value.Parameters.Select(parameter =>
          string.Join(',', parameter.In.Value, parameter.Name, parameter.Schema.Type, parameter.Description));

        Debug.WriteLine(path.Key);
        Debug.WriteLine(operation.Key);
        Debug.WriteLine(string.Join(Environment.NewLine, parameters));
        Debug.WriteLine(Environment.NewLine);
      }
    }

    foreach (var schema in openApiSpec.OpenApiDocument.Components.Schemas)
    {
      var properties = schema.Value.Properties.Select(property =>
        string.Join(',', property.Key, property.Value.Type, property.Value.Description));

      Debug.WriteLine(schema.Key);
      Debug.WriteLine(string.Join(Environment.NewLine, properties));
      Debug.WriteLine(Environment.NewLine);
    }
  }

  return 0;
}

/build
Post
Query,dockerfile,string
Query,t,string
Query,extrahosts,string
Query,remote,string
Query,q,boolean
Query,nocache,boolean
Query,cachefrom,string
Query,pull,string
Query,rm,boolean
Query,forcerm,boolean
Query,memory,integer
Query,memswap,integer
Query,cpushares,integer
Query,cpusetcpus,string
Query,cpuperiod,integer
Query,cpuquota,integer
Query,buildargs,string
Query,shmsize,integer
Query,squash,boolean
Query,labels,string
Query,networkmode,string
Header,Content-type,string
Header,X-Registry-Config,string
Query,platform,string
Query,target,string
Query,outputs,string
...
/build/prune
Post
Query,keep-storage,integer
Query,all,boolean
Query,filters,string
...
ContainerConfig
Hostname,string
Domainname,string
User,string
AttachStdin,boolean
AttachStdout,boolean
AttachStderr,boolean
ExposedPorts,object
Tty,boolean
OpenStdin,boolean
StdinOnce,boolean
Env,array
Cmd,array
Healthcheck,object
ArgsEscaped,boolean
Image,string
Volumes,object
WorkingDir,string
Entrypoint,array
NetworkDisabled,boolean
MacAddress,string
OnBuild,array
Labels,object
StopSignal,string
StopTimeout,integer
Shell,array
...

[HofmeisterAn]

I wrote a small test to detect differences between the OpenAPI specification and the configuration in modeldefs.go. I was able to detect some differences, #577 is one of them. I'll create a pull request that contains the recent OpenAPI specification. I didn't check the other way around (I just had a quick look), but it looks like that some configurations in modeldefs.go are no longer available. Furthermore a lot of APIs aren't available or covered with Docker.DotNet yet.

I still think it's weird not using the OpenAPI tools. I don't understand why we're "restricted" to native .NET tools. The OpenAPI specification incl. their tooling is the de facto standard and used in various programing languages among (probably) millions of projects. Even the Docker Go files are generated by OpenAPI tooling.

We are developing a "solution" that relies on the Go files generated by OpenAPI tooling to then generate .NET classes 🤡.

---

Here is a small example of the output of my test run:

Diffs in POST /commit
Found additional parameters in the OpenAPI specification: 1
  container:string:query

Diffs in GET /containers/*/logs
Found additional parameters in the OpenAPI specification: 1
  until:integer:query

I fixed all of these. I'll create a pull request soon.

---

This is the entire log (it contains already the fixes mentioned above). They might not be 100% accurate, but will probably help to find inconsistencies:

Show test output

```txt ~55 Docker REST methods found in the Docker.DotNet configuration ~106 Docker REST methods found in the OpenAPI specification Diffs in POST /build Found additional parameters in the OpenAPI specification: 1 content-type:string:header Found additional parameters in the Docker.DotNet configuration: 7 cgroupparent:string:query cpusetmems:string:query isolation:string:query pullparent:bool:query securityopt:[]string:query session:string:query ulimits:[]*units.ulimit:query Diffs in POST /commit Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 config:*container.config: Diffs in POST /configs/*/update Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 :swarm.configspec:body Diffs in POST /configs/create Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 :swarm.configspec:body Diffs in GET /containers/*/archive Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 nooverwritedirnondir:bool:query Diffs in POST /containers/*/update Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 container.updateconfig:: Diffs in POST /containers/create Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 3 *container.config:`rest:"body"`: hostconfig:*container.hostconfig:body networkingconfig:*network.networkingconfig:body Diffs in GET /containers/json Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 2 before:string:query since:string:query Diffs in GET /images/*/json Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 size:bool:query Diffs in POST /images/*/push Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 fromimage:string:query Diffs in POST /images/*/tag Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 force:bool:query Diffs in GET /images/search Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 x-registry-auth:types.authconfig:headers Diffs in POST /plugins/*/disable Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 force:bool:query Diffs in POST /plugins/*/set Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 :[]string:body Diffs in POST /plugins/*/upgrade Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 :types.pluginprivileges:body Diffs in GET /plugins/privileges Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 x-registry-auth:types.authconfig:headers Diffs in POST /plugins/pull Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 :types.pluginprivileges:body Diffs in POST /services/*/update Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 service:swarm.servicespec:body Diffs in POST /services/create Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 service:swarm.servicespec:body Diffs in POST /swarm/update Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 1 spec:swarm.spec:body Diffs in POST /volumes/create Found additional parameters in the OpenAPI specification: 0 Found additional parameters in the Docker.DotNet configuration: 4 driver:string: driveropts:map[string]string: labels:map[string]string: name:string: GET /_ping not found in the Docker.DotNet configuration. HEAD /_ping not found in the Docker.DotNet configuration. POST /auth not found in the Docker.DotNet configuration. POST /build/prune not found in the Docker.DotNet configuration. GET /configs not found in the Docker.DotNet configuration. GET /configs/* not found in the Docker.DotNet configuration. DELETE /configs/* not found in the Docker.DotNet configuration. HEAD /containers/*/archive not found in the Docker.DotNet configuration. PUT /containers/*/archive not found in the Docker.DotNet configuration. GET /containers/*/attach/ws not found in the Docker.DotNet configuration. GET /containers/*/changes not found in the Docker.DotNet configuration. GET /containers/*/export not found in the Docker.DotNet configuration. POST /containers/*/pause not found in the Docker.DotNet configuration. POST /containers/*/unpause not found in the Docker.DotNet configuration. POST /containers/*/wait not found in the Docker.DotNet configuration. GET /distribution/*/json not found in the Docker.DotNet configuration. GET /exec/*/json not found in the Docker.DotNet configuration. POST /exec/*/resize not found in the Docker.DotNet configuration. GET /images/*/get not found in the Docker.DotNet configuration. GET /images/*/history not found in the Docker.DotNet configuration. GET /images/get not found in the Docker.DotNet configuration. GET /info not found in the Docker.DotNet configuration. GET /networks/* not found in the Docker.DotNet configuration. DELETE /networks/* not found in the Docker.DotNet configuration. POST /networks/*/connect not found in the Docker.DotNet configuration. POST /networks/*/disconnect not found in the Docker.DotNet configuration. POST /networks/create not found in the Docker.DotNet configuration. GET /nodes not found in the Docker.DotNet configuration. GET /nodes/* not found in the Docker.DotNet configuration. DELETE /nodes/* not found in the Docker.DotNet configuration. POST /nodes/*/update not found in the Docker.DotNet configuration. GET /plugins/*/json not found in the Docker.DotNet configuration. POST /plugins/*/push not found in the Docker.DotNet configuration. GET /secrets not found in the Docker.DotNet configuration. GET /secrets/* not found in the Docker.DotNet configuration. DELETE /secrets/* not found in the Docker.DotNet configuration. POST /secrets/*/update not found in the Docker.DotNet configuration. POST /secrets/create not found in the Docker.DotNet configuration. GET /services/* not found in the Docker.DotNet configuration. DELETE /services/* not found in the Docker.DotNet configuration. POST /session not found in the Docker.DotNet configuration. GET /swarm not found in the Docker.DotNet configuration. POST /swarm/init not found in the Docker.DotNet configuration. POST /swarm/join not found in the Docker.DotNet configuration. GET /swarm/unlockkey not found in the Docker.DotNet configuration. GET /system/df not found in the Docker.DotNet configuration. GET /tasks/* not found in the Docker.DotNet configuration. GET /tasks/*/logs not found in the Docker.DotNet configuration. GET /version not found in the Docker.DotNet configuration. GET /volumes/* not found in the Docker.DotNet configuration. DELETE /volumes/* not found in the Docker.DotNet configuration. ```

[galvesribeiro]

I'm not entirely sure the Go files are generated from OpenAPI. I think they generate OpenAPI from the types on those files. Just like Kubernetes APIs do. But nonetheless, we should get rid of it, I agree.

The differences you see in parameters and methods are probably methods that we either haven't implemented because people never used (there are lots of them) or parameters that were added but since they weren't required and people never asked for it, we never added.

So here is the thing. I'll look at the model metadata being read from the OpenAPI you posted and see if we have enough info to generate ourselves the types we need.

This ofc will be a breaking change but I've already warned people on other issues in the past that we would do that eventually.

[HofmeisterAn]

I think they generate OpenAPI from the types on those files. Just like Kubernetes APIs do.

Ok, got it. Looks like I just looked into files like create_response.go or container_top.go. I stumbled across the Code generated... comment, but they are mentioning following in their docs:

Types shared by both the client and server, representing various objects, options, responses, etc. Most are written manually, but some are automatically generated from the Swagger definition. See #27919 for progress on this.

Nevertheless, Microsoft.OpenApi.Readers looks promising. It reads the YML file and provides (I think) all information we need (expect the mapping OC).

So here is the thing. I'll look at the model metadata being read from the OpenAPI you posted and see if we have enough info to generate ourselves the types we need.

Sounds great 👍, let me know if I can help you.

[ACoderLife]

So here is the thing. I'll look at the model metadata being read from the OpenAPI you posted and see if we have enough info to generate ourselves the types we need.

they might also take a update to their swagger if we are missing something.

[mauve]

FWIW NSwagStudio (https://github.com/RicoSuter/NSwag/wiki/NSwagStudio) manages to generate a pretty decent client from the swagger file https://docs.docker.com/engine/api/v1.41.yaml, it is a generated file so it looks a bit annoying, but tbh better than a lot of other generators (especially the openapi-generator which generates some really annoying stuff).

https://gist.github.com/mauve/7403b41ed27b1a78aea07b2c03a3c455