[Gradle] plugin and sample task that generates code as part of the build process

[ber4444]

Description

When I look at the long readme of the project at https://github.com/swagger-api/swagger-codegen, Gradle is not even mentioned on it. You encourage people to wget the .jar file instead, and then go to the command line to generate code with it from swagger.json (optionally via a shell script that of course needs to be tweaked as it's specific to the 'petstore' sample). Way down towards the end of the page there is a reference to a maven plugin as well but no gradle plugin. So Android developers that are pretty much forced by Google to use Gradle tools are now left with going to the command line every time the json spec changes or to learn bash again and tweak your script (which by the way also requires you to go to the command line every time).

Suggest a Fix

We have a working Gradle plugin and Android project using it that generates Api and Dto classes for every new version of the Json spec. Might publish it via a PR.

[wing328]

@ber4444 thanks for the suggestion. We definitely welcome a Gradle plugin with a PR.

[JLLeitschuh]

This is a pretty simple plugin that does exactly what most people need. https://github.com/thebignet/swagger-codegen-gradle-plugin

[ber4444]

@JLLeitschuh would be nice to make that a PR. The first thing that I miss from the current version you have is that it does not allow you to specify your own swagger version. Something like this would be nice:

dependencies {
    swaggerCodegen "io.swagger:swagger-codegen-cli:${swagger_codegen_cli_version}"
    //or even better: swaggerCodegen files('lib/swagger-codegen-cli-patched.jar')
}

Also, it should re-generate stuff when you bump the version of the specs. I.e.

    def versionFile = project.file("build/gen/${version}.built")

    doFirst {
        if (versionFile.exists()) {
            throw new StopExecutionException(
                    'Skipping swagger codegen as code has already been generated')
        }

        // make sure we're generating from scratch
        getOutputDir().deleteDir()
    }

    doLast {
        versionFile.createNewFile()
    }
}

tasks.compileJava.dependsOn 'swaggerCodegen'

[otrosien]

Please also fix the unnecessary swagger-codegen dependency on maven-plugin-tools-api. It should only be a dependency in the swagger-codegen-maven-plugin module. Otherwise maven jars will leak into non-maven builds.

If interested, I could also provide a PR for this separately.

[otrosien]

Created a PR: https://github.com/swagger-api/swagger-codegen/pull/4840

[otrosien]

I'd like to offer my help in creating a gradle plugin for swagger-codegen.

@ber4444 - about your comments. The first requirement should be fairly straight-forward, and is considered best practice for plugin-authors to provide the possibility to have control over tool versions. I don't like the "or even better" syntax pointing to a jar-file. This looks very un-gradleish to me. You can always go via local maven repo if you want to test the integration with snapshots / patched versions.

Secondly, I didn't get the "it should re-generate stuff when you bump the version of the specs. I.e." part. Are you saying, if the user upgrades swagger-codegen version, the gradle plugin should make sure that the code is re-generated? If that's the case, I'm pretty sure there are better ways, so ther is no need for such a marker-file. But I might have misunderstood your requirement.

[ber4444]

Thanks, @otrosien , the first point is more for sharing the patched version in git so that others have the same build without needing to rebuild swagger itself and without having to set up a CI build and artifactory publish thingy.

About the second point, no, I was referring to upgrading the api specs (json file) that serves as an input to swagger. So you would not want to generate the entire api as part of each build as it makes you development slower. But when the spec version changes - once in a while - you do want the plugin to re-run swagger and re-generate the api.

[otrosien]

That's what local maven repo would be useful for, right?

Another problem I see is that gradle plugins are supposed to be built with gradle. Using maven for building isn't officially supported - and if even if it's technically possible, you're missing quite some tooling convinience from gradle.

So I would suggest creating the swagger-codegen plugin for gradle in its own repository using gradle for building and releasing.

[otrosien]

@ber4444 gradle has powerful caching and task avoidance in it, so unlike in maven you wouldn't typically run the "clean" task all the time. In our case gradle would figure out that your spec has changed when its timestamp is newer than the target files.

[otrosien]

@wing328 can you give your thoughts on that topic? can we create a new repo / project for the gradle integration or should we try to call gradle from maven or do some other crazy hacks in order to keep everything inside the swagger-codegen repository?

[JLLeitschuh]

call gradle from maven or do some other crazy hacks in order to keep everything inside the swagger-codegen repository

That sounds really painful and a recipe for a far more complicated build.

[wing328]

should we try to call gradle from maven or do some other crazy hacks in order to keep everything inside the swagger-codegen repository?

If I understand correctly, you want to build Swagger Codegen with gradle instead of maven.

What about adding build.gradle files to this project to begin with so that developers can use gradle or maven to build the project based on their preference?

[otrosien]

@wing328 I don't want to start a flame-war about which build tool to use for swagger-codegen. Of course switching from maven to gradle would resolve the problem, but I'd rather start with a separate repo that uses gradle to build the gradle plugin only.

[wing328]

@otrosien for our Java API client, we generate build files for Maven, Gradle and sbt in the same output. The goal is to let developers to choose whatever they want to build the project (in other words we should not restrict them from using other tools to build the API client)

Your suggestion to add build.gradle files to this project to serve as an alternative is definitely in line with what we want to do and I might be the one who benefits the most as I'm the one who needs to build projects at least several time pretty much every day)

(I see this as a healthy discussion in the Swagger Codegen community)

[otrosien]

@wing328 can you point me to the repo? And I still don't see how that is solving the problem that the gradle plugin wouldn't build with maven. Your proposal only works if all modules can be built with all build tools, or not?

[wing328]

can you point me to the repo?

I believe you're referring to the Java API client. An example can be found in https://github.com/swagger-api/swagger-codegen/blob/master/samples/client/petstore/java/okhttp-gson/

our proposal only works if all modules can be built with all build tools, or not?

I think we can start with just adding gradle to begin with. If there's a demand for sbt or other build tool, we can consider those later.

My understanding is that the current modules in this project can be built with gradle as well but I could be wrong.

[otrosien]

@wing328 well I'm not referring to the modules we build inside swagger-codegen. I'm talking about the main build of swagger-codegen (the project) itself.

[wing328]

The idea is the same. Ideally we want the developers to use whatever build tool they prefer.

Currently, a developer who prefers gradle over maven cannot build this project (swagger codegen) with gradle.

If you can provide the build.gradle files for this project, that would be greatly appreciated.

[JLLeitschuh]

I don't really think it makes sense for this project to support two build tools to create it. This would result in duplicated build logic which could easily be broken in one but not the other. The project's maintainers pick the build tool to build the library. You shouldn't add grunt as a build tool and have a java version of this project just to make the javascript developer happy.

If you want to move from maven to gradle then make the decision to do a total conversion, but don't try to support both simultaneously. That will just lead to more issues and headache than it's worth.

[otrosien]

I second @JLLeitschuh - let's not create maintenance nightmares. I would appreciate if we got our own little repo to concentrate working on the feature.

[wing328]

You shouldn't add grunt as a build tool and have a java version of this project just to make the javascript developer happy.

I agree but I don't think it's fair comparison between grunt (for JS) and gradle (for Java-related languages) in this particular case.

This would result in duplicated build logic which could easily be broken in one but not the other

I also agree potentially there may be issues and we should use CIs to make sure the build tools work as expected.

My guess is that having a build.gradle will attract more Android developers to work on this project , which is a good thing for this project as a whole (this is not to say only Android developers using gradle) and is consistent with what we've been doing - add more features to meet different developer's need and preference as no single solution fits all.

[otrosien]

I can give it a try, maybe to a point that at least gradle build works.

[JLLeitschuh]

There is an easy conversion process that gradle ships with for converting the basic maven to gradle project. https://guides.gradle.org/migrating-from-maven/

From experience writing and re-writing build systems from maven to gradle you may want to give gradle script kotlin a shot.

It allows you to write your entire build in an inferred type language instead of groovies duck typing system.

The strong typing really helps you wrap your head around the dsl and keeps you from spending your time scratching your head thinking "why doesn't my build compile".

I've just spent the past two months or so rewriting my companies build from maven to gradle script kotlin and I'm much happier.

[ePaul]

While I think having a gradle plugin is a good idea, I don't think it implies building this whole project in two different ways. Or is this needed in order to publish the plugin (and its dependencies)?

Then we would have in the end one "swagger-codegen-build-by-maven" and one "swagger-codegen-build-by-gradle" in circulation, with potentially differing features.

(I'm not opposed to a total conversion, even though I have less experience with Gradle.)

[fehguy]

Here's my $0.02. We need the build process to use the most familiar, common tooling, period. If you look back in time, this project used to be a scala project and that created a ton of issues. So once Gradle becomes the defacto build process for Java, we should switch to it. Otherwise, having two paths to build means yet another way for inconsistent results, and frankly the build system should not be the focus of this project. So I vote to leave it alone for now. Or maybe switch to SBT which has a really concise DSL (but there are like 100 people in the world that use it).

The generated clients--however--are completely different. They should have build systems that follow the vernacular of the target language/framework. That may mean a gradle build if we're talking about Android vs. Maven.

[JLLeitschuh]

I think a question that should be asked is if the project's build is currently a pain point for developers/contributors?

• Is the current build slow enough to warrant trying a different tool?
• Is there some feature/task that you wish you could automate that you would need a complicated maven plugin to pull off?
• Are you having problems with inconsistent/confusing dependency resolution?
• Are you tired of having to update 4 pom files every time there is a version bump for the project?

If the answer to these questions is yes then it is worth spending the time to convert to gradle. Gradle provides you with an entire language that you can use to configure your build.

One benefit that I can think of off the top of my head is that you would be able to invoke the gradle builds in your example projects from within the same process.

[fehguy]

That's not my point. There is no such thing as a perfect build system, and there would be pain with Gradle as well. It may be much better than maven, but until it's not the defacto build system for java, my suggestion is to not move that direction.

Nothing against Gradle, but community projects work best when they're as familiar as possible. I personally don't like Maven (I like it better than ant though), but pretty much anyone who comes across this project can build it and contribute. Bumping versions in 4 files is not a good enough reason IMHO, it doesn't happen very often so not really something to optimize for...

[webron]

:ant:

[otrosien]

so.. back to the subject: As stated earlier, when we want to provide a plugin for gradle, we need to develop it with gradle. Maven is not supported in that case. In my eyes we have two options:

• Develop the gradle plugin as its own project using gradle
• Convert swagger-codegen completely from maven to gradle

I would prefer the first alternative.

[otrosien]

FYI: Here's what gradle init --type pom produces. I didn't optimize anything yet, like extracting common version numbers. I just had to fix the maven central repository reference, and a TestNG test that failed on my computer; maybe unrelated... But then there's the "little differences". For example, swagger-generator depends on test-classes from swagger-codegen, which are invisible in gradle when setting a simple project dependency.

I haven't yet looked into what all of the maven plugins do as part of the build, and I wouldn't dare to say the resulting build artifacts are identical to what maven produces.

[JLLeitschuh]

Looks pretty good!

[JLLeitschuh]

I'd be happy to contribute the source code for the swagger codegen gradle plugin I've been using internal to our build as the basis for this plugin. Theres some changes I would make if it were a full blown project but I think the core works pretty well.

[JLLeitschuh]

This is the contents of the plugin as it stands right now: I have a few comments in places I want to fix things up.

The plugin was written in kotlin.

import com.plexxi.gradle.taskHelper
import io.swagger.codegen.config.CodegenConfigurator
import org.gradle.api.Plugin
import org.gradle.api.Project

open class SwaggerCodeGenPlugin : Plugin<Project> {
    companion object TaskNames {
        const val SWAGGER_TASK = "swagger"
    }

    override fun apply(project: Project) {
        // I would probably change this so that it allows for multiple codegen runs to be configured within one project.
        project.extensions.create("swagger", CodegenConfigurator::class.java)
        val swaggerTask = project.taskHelper<SwaggerCodeGenTask>(SWAGGER_TASK)
        project.getTasksByName("compileJava", false).forEach { it.dependsOn(swaggerTask) }
    }
}

import io.swagger.codegen.DefaultGenerator
import io.swagger.codegen.config.CodegenConfigurator
import org.gradle.api.DefaultTask
import org.gradle.api.GradleException
import org.gradle.api.tasks.InputFile
import org.gradle.api.tasks.OutputDirectory
import org.gradle.api.tasks.TaskAction
import java.io.File

abstract class AbstractSwaggerGenTask : DefaultTask() {
    init {
        group = "Swagger"
    }

    /**
     * Ideally this would be marked as an input to this task however I need to fix some things around how it is implemented.
     */
    abstract val configuration: CodegenConfigurator

    @get:InputFile
    val inputFile: File by lazy { project.file(configuration.inputSpec) }

    @get:OutputDirectory
    val outputDir: File by lazy { project.file(configuration.outputDir) }

    /**
     * Guard against deleting the directory being passed.
     * I accidentally did this to the root project directory.
     */
    private fun validateDeleteOutputDir(againstDir: File) {
        if (outputDir == againstDir) {
            throw GradleException("You probably don't want to delete this directory: $againstDir")
        }
    }

    @TaskAction
    fun swaggerCodeGen() {
        validateDeleteOutputDir(project.projectDir)
        validateDeleteOutputDir(project.rootProject.projectDir)

        // If the spec has changed then this file will have have changed.
        outputDir
            .deleteRecursively()
        /*
         * Since the generator sets system properties we need to ensure that two tasks don't try
         * to have system properties set in the same JVM.
         * https://github.com/swagger-api/swagger-codegen/issues/4788
         */
        synchronized(this::class) {
            val config = configuration
            DefaultGenerator()
                .opts(config.toClientOptInput())
                .generate()

            // Clean up the system environment variables that have been set by the code generator.
            // https://github.com/swagger-api/swagger-codegen/issues/4788
            config.systemProperties.keys.forEach { System.clearProperty(it) }
        }
    }
}

import io.swagger.codegen.config.CodegenConfigurator

open class SwaggerCodeGenTask : AbstractSwaggerGenTask() {

    init {
        description = "Generates code from the swagger spec and the CodegenConfigurator."
    }

    override val configuration: CodegenConfigurator by lazy { project.extensions.getByType(CodegenConfigurator::class.java) }

}

When implementing this as a full plugin I'd probably make the plugin extension use the Named Domain Object Container allowing the api users to cusomize multiple CodegenConfigurator under seperate names. Each would create it's own task.

Maybe add a configuration option to automatically add the java output to the current project's source sets. Just a few thoughts.

[otrosien]

@JLLeitschuh good work, we should synchronize efforts. I like the safety-check to not accidently delete project dir. I'm currently working on preparing my company's current plugin to be handed-over to swagger.

Features / Differences:

• written in java7
• support for user-defined swagger-codegen version
• cross-compile testing with different gradle versions and swagger-codegen versions
• 100% support of CodegenConfigurator properties (functional test to check for future versions), all defined as task inputs.
• support of CodegenConfigurator.fromFile() to externalize configuration
• support of multiple swagger codegen tasks
• Adds output dir as java source dir (todo: this should be customizable to support other languages than java)

I'm lacking support for cleaning up system properties, but I have the hope that swagger-codegen does away with that ugly hack.

See my repo for the code: https://github.com/otrosien/swagger-codegen-gradle-plugin/

[otrosien]

BTW: Happy to try out kotlin, if the others are fine with it :)

[otrosien]

.. implemented some of your ideas:

• Add swagger output directory as source dir, but only if java plugin has been applied on the same project -- we could also check for known 'lang' values and map them to respective gradle language plugins (scala for example)
• Added sanity check for task output directory
• Added clean-up of system properties, and synchronized block around the execution
• -Also removing outputDir by default (unless user specifies skipOverwrite)- reverted that change. Swagger generates sub-directories for src/main/java and src/gen/java.. These get registered as source dirs, and if we regularly remove them I fear problems in IDEs. So if the user wants outputDir to be cleaned up, he needs to run cleanSwaggerCodegen

[JLLeitschuh]

The only problem I see with your current solution to using the CodegenConfigurator is that you don't support future versions of the CodegenConfigurator in the build. If additional API's get added to the CodegenConfigrator the task has only been written to the specific version of the CodegenConfigurator.

One way around this is simply exposing the CodegenConfigurator as a configuration object.

If people want to use a different version of the swagger codegenerator than the one that the plugin depends on they can add a line to their buildscript block that forces the version of the swagger codegen dependency to use.

Eg:

buildscript {
    applyFrom("${rootProject.projectDir}/sharedValues.gradle.kts")

    dependencies {
        classpath("io.swagger:swagger-codegen:${extra.get("swaggerCodegenVersion")}")
        // Also depend upon plugin
    }
}

[otrosien]

I do like the minimalism of your solution, and indeed it makes things easier. But as you stated above, this means you don't have full control of input/output declarations, so you end up with sub-optimal up-to-date handling. In the end, releasing a new version of the gradle plugin once there is a new version of swagger-codegen with extended API is not a big deal. And there's a test for that case, so we don't accidently miss support for new properties.

[JLLeitschuh]

The only problem with marking the CodegenConfigurator as an input was that the way that I got it from gradle was through the extention plugin. You can cache the CodegenConfigurator because it implements Serializable. But the one that gradle gives you back with:

project.extensions.getByType(CodegenConfigurator::class.java)

Is actually a decorated proxy version of the object which itself is not cacheable. If we were able to not use the proxied version from gradle the entire object could be cached and, if the user changes the configuration of it,

[otrosien]

@JLLeitschuh this CodegenConfigurator Serializable impl. is new, I just saw that you added it in #4887! Good job. I'll try to use it in a task extension so that CodegenConfigurator can be annotated as task input.

[otrosien]

I got something, but I'm not satisfied... See here (look at spock test): https://github.com/otrosien/swagger-codegen-gradle-plugin/commit/652c66539f44446e3c0b1a5f9377c332e6658160#diff-63998e99513b192529480f3f68db0715L59

The quality of the DSL is IMO degregated. You need to write

swaggerCodegen.config {
inputSpec = 'input.yml'
}

instead of

swaggerCodegen {
inputSpec 'input.yml'
}

[otrosien]

.. and externalizing the swaggerCodegen configuration to a file (swaggerCodegen.fromFile codegenConfig.yml) does not work anymore...

[JLLeitschuh]

I would argue that whatever object swaggerCodegen is should implement NamedDomainObjectContainer so that you can do something like this.

swaggerCodegen {
    "someArbatraryTaskNameYouWantCreated" {
        config {
            inputSpec = 'input.yml'
        }
    }
}

[otrosien]

@JLLeitschuh the missing piece is still that you want gradle to monitor your input files for changes, in order to know if it needs to rebuild. CodegenConfigurator solely works on Strings pointing to files. So, in the end you need to know/develop against the internals of CodegenConfigurator...

[JLLeitschuh]

If you look at my example above I handle that this way:

abstract class AbstractSwaggerGenTask : DefaultTask() {
    init {
        group = "Swagger"
    }

    abstract val configuration: CodegenConfigurator

    @get:InputFile
    val inputFile: File by lazy { project.file(configuration.inputSpec) }

    @get:OutputDirectory
    val outputDir: File by lazy { project.file(configuration.outputDir) }

The only other problem that is the case where your swagger.yaml file references other files.

In my build in the one project that I do do this I use this bit of code:

val swaggerYamlTask = tasks.getByName(SwaggerCodeGenPlugin.SWAGGER_TASK) as SwaggerCodeGenTask
swaggerYamlTask.apply {
    /*
     * Mark the entire directory as an input.
     * We only need to do this for this project because the swagger spec is split across multiple files.
     */
    inputs.dir(specDir)
}

Also, do we want to support the task creating an artifact? If you want to do that I used this bit of code:

val yamlArtifact = mapOf(
    "name" to "swagger-spec",
    "type" to "yaml",
    "extension" to "yaml",
    "file" to swaggerOutputFile,
    "builtBy" to swaggerYamlTask
)

artifacts.add(specConfig.name, yamlArtifact)

This is only really useful if you want to have the spec be an artifact that is used in different projects.

[otrosien]

@JLLeitschuh ok, so whatever input / output you want to manage via CodegenConfig, you would need to duplicate into the task (in your case, you're missing templateDir, for example). CodegenConfig exposing also means you lose the ability to provide convenience methods, which are useful when dealing with map/list properties.

Anyway, I'd like to proceed here, in whatever direction - so @ project admins: can we have a new repo to play with and work on the gradle plugin?

[wing328]

@otrosien What about using a branch (e.g. gradle_plugin) for the time being?

cc @fehguy about creating a new repo

[fehguy]

I suggest using a branch for now

[otrosien]

Could you elaborate on how we should solve the problem that we can't use maven for building?

[wing328]

Could you elaborate on how we should solve the problem that we can't use maven for building?

I believe you mean building the project as a whole using gradle install instead of mvn install. How do you build the project at the moment? Have you already created the build.gradle or convert the existing pom.xml to gradle build files?

For gradle plugin, I've created a new branch: https://github.com/swagger-api/swagger-codegen/tree/gradle_plugin