Support for idiomatic Kotlin

[franckrasolo]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Pulumi's Java API is similar to that of the CDK for Terraform in its use of the Builder pattern.

In other JVM languages such as Kotlin and Scala, it is more idiomatic to use data/case classes with optional parameters instead in most basic cases. See this article on Kotlin's Type-safe builders.

How would someone go about contributing idiomatic support for Kotlin or Scala?

Affected area/feature

The API of the newly launched Pulumi support for Java.

[dixler]

we've considered some stuff and thought it was best to get community feedback first so it's much appreciated

related:

• 151

[franckrasolo]

Thanks @dixler!

I suggest to anyone interested in this topic k8s-kotlin-dsl as arguably the best illustration of a generated Kotlin DSL for Kubernetes out there.

[pawelprazak]

As for the Kotlin's typesafe builder and other Kotlin features, from the top of my head this is what would be required:

• Kotlin codegen to generate providers with contlin builders
• Kotlin SDK wrapper, major improvement that comes to mind is reified types translated to our TypeShape.

Currently I am not aware of any plans to implement this, but Java support started as a community initiative, so it is not impossible. For the reference, Java SDK and codegen took about a man-year of effort so far.

As for Scala, company called VirtusLab is planning to create an idiomatic clean sheet implementation of Pulumi SDK in Scala 3.

[dixler]

An initial approach could yield this DSL design

managedCluster("cluster-1") {
    args {
        identity {
            type = ResourceIdentityType.SystemAssigned
        }
        servicePrincipalProfile {
            clientId = "msi"
        }
        agentPoolProfiles {
            name   = "pool1"
            count  = 1
            osType = OSType.Linux
            osSKU  = OSSKU.Ubuntu
            vmSize = "standard_a2_v2"
            mode   = AgentPoolMode.System
        }
    }
    opts {
        protect = true
    }
}

This would probably be unable to handle Outputs. We could use an overload and change = to += inspiration or other options.

feedback and suggestions are definitely welcome.

[t0yv0]

Just to clarify that I am understanding the comments correctly. It is stated above that from technical perspective, supporting Kotlin:

• would require a forked provider SDK just for Kotlin for every of the 70+ providers since it is not possible to combine Kotlin-style and Java-style resource builders and other facilities in the same provider SDK package

• would suggest a specialized Kotlin SDK that may wrap the Java SDK

Is this accurate?

The largest maintenance cost is forking providers. Is it quite certain there's no compromise in which Kotlin and Java could reasonably reuse the same providers, with potentially making some changes from where we are now?

[pawelprazak]

The above assumptions are just my best guess, since I'm not a Kotlin programmer.

I'm not sure what would be an acceptable compromise, I'll let the Kotlin community decide on that.

IMHO for an idiomatic Kotlin (as in the title of the issue) it would be a shame not to use the full power of Kotlin, to name a few features that look useful:

• type-safe builders (for nice API)
• coroutines (for the runner)
• reified generic types (instead of TypeShape)
• null safety (may help)
• Kotlin.js (for AWS Lambda, etc.) (probably more, please let me know)

I might be wrong, so I encourage Koltin users to voice your opinions, please.

[stepango]

I believe there are two ways to add kotlin support without reimplementing everything :)

Kotlin wrapper for java SDK's - pretty common in Kotlin community, many libraries have it's kotlin counterpart with nicer null-safe API's and kotlin DSL support

Kotlin wrapper for JS/TS SDK - kotlin interracts great with JS and can reuse TS signatures for code generation.

@t0yv0 java SDK could be modified in a way that will support Java and Kotlin style resource builders simulteniously 👍

@dixler I want to suggest alternative approach based on data classes, comparing to DSL style it will provide more type safety and guidance. DSL style may cause issues with names shadowing and multi-assignments.

managedCluster("cluster-1", args = ManagedClusterArgs(
        identity = Identity(
            type = ResourceIdentityType.SystemAssigned
        ),
        servicePrincipalProfile = ServicePrincipalProfile(
            clientId = "msi"
        ),
        agentPoolProfiles  = AgentPoolProfiles(
            name   = "pool1",
            count  = 1,
            osType = OSType.Linux,
            osSKU  = OSSKU.Ubuntu,
            vmSize = "standard_a2_v2",
            mode   = AgentPoolMode.System,
        )
    ),
    opts = Opts(
        protect = true
    )
)

[t0yv0]

@stepango thanks for that input! We're indeed seeking community input and Java/Kotlin expertise to guide the effort.

A bit on our context: the key current cost dimensions for Pulumi are supporting new language across 70+ provider releases, and generating extra classes in those releases that creates package bloat. Ideal designs would allow a good Kotlin experience without requiring a new "aws-kotlin", "azure-kotlin", "google-kotlin" artifact to be managed. If we could modify Java releases or TypeScript releases so they would be consumable from Kotlin, we could support Kotlin much earlier. Of these, Java releases are easier to modify as Java is in Public Preview and more accepting of breaking changes.

If Kotlin requires dedicated releases for every provider, this is still possible but might need to happen later due to prioritization. Upvotes to this issue help gauge interest and direct dev priorities.

So with this context in mind, where does the data class approach you suggest fall? Can Java and Kotlin use the same type definitions for e.g. ManagedClusterArgs or the Kotlin one has to be separate?

[xuanswe]

I want to suggest alternative approach based on data classes,

I prefer Kotlin DSL approach because of the power of putting logic inside each DSL { //... } block. Of course, extracting logic is still needed for clean code. Data classes cannot provide that level of flexibility.

DSL style may cause issues with names shadowing and multi-assignments.

I don't see any issues here. I have experience working with TeamCity Kotlin DSL. It's an amazing experience. In my project (30 microservices on 10 environments), Kotlin DSL is so powerful, and easy to use.

[stepango]

@t0yv0 One of the options which will allow supporting Java and Kotlin APIs simultaneously is to introduce interfaces for classes produced by builders, this way we can have Kotlin implementations where an interface inherited by data class with overridden default parameters, producing API's similar to other supported languages.

@xuan-nguyen-swe putting logic inside { } blocks is the main reason why DSL approach should be avoided in the case of Pulumi, infrastructure definitions should be declarative and easy to understand. Another reason for avoiding that is - keeping consistent API's across all supported languages.

[xuanswe]

putting logic inside { } blocks is the main reason why DSL approach should be avoided in the case of Pulumi

I have experience using the DSL approach with TeamCity and the only bad thing here is because of bad developers, not the DSL. Using it right, your infrastructure definitions should always be declarative and easy to understand. In our project, we split our code into very small logical blocks and reuse them wisely. We don't need much logic inside our code, but the DSL approach keeps our configuration flexible, reusable, and extendable. I reviewed bad DSL usage and prevented them from merging into our project. The best solution is to mentor juniors to write good code, collecting best practices in our team. Data classes or other approaches cannot prevent bad developers from doing things wrong.

Similarly, IMO, Pulumi is created to take advantage of programming languages. With the power of programming languages, Pulumi cannot prevent bad developers from writing non-declarative code. They will try their best to hack the system with workarounds.

Another reason for avoiding that is - keeping consistent API's across all supported languages.

OK, somehow I agree with this reason but feel very unfortunate for the Kotlin community.

[t0yv0]

Continue to be interested in what we can do here! Especially valuable would be more long-form demonstrations of an example resource class with arguments and resource outputs, and consuming code in Java and in Kotlin.

It is especially interesting if we can find a way to get a better Kotlin experience without forking off new artifact builds, so both Kotlin and Java could keep using say com.pulumi:gcp artifact. As of Sep 23 we have 48 providers publishing to https://search.maven.org/search?q=com.pulumi and we are facing some code and jar bloat issues in larger providers like azure-native.

CC @luistrigueiros

[luistrigueiros]

@t0yv0 this is very interressting I did not knew that there where so many java providers alreday available. Kotlin has very good integration and interop with java and it should be possible even in existing providers make then hydrid in the sense that they can be have parts written in java and parts written in Kotlin. This was the Spring has alredy done by adding support for Kotlin in Spring framework.

[t0yv0]

Folks I'm experimenting a bit here this week. I'm trying out swapping out ResourceArgs classes with Kotlin data classes: https://github.com/t0yv0/pulumi-kotlin-experiment/blob/main/src/main/kotlin/appservice/App2.kt#L35

Earlier experiment with Type-safe builders from @dixler : https://gist.github.com/dixler/440f1041fb2a8544e334ae054b85bdca

It seems promising:

• For Kotlin programs, using data classes makes missing required arguments a compile-time error, this is the biggest win

• Java programs can continue to use builders and need only minor updates, while technically breaking this change is relatively small (https://github.com/t0yv0/pulumi-kotlin-experiment/blob/main/src/main/java/appservice/App.java still compiles and works without change)

• Unlike type-safe builder DSLs, which probably push for an ideal Kotlin experience, there is not a ton of net-new code that only serves Kotlin. It does not look that Java can benefit from the Kotlin DSL declarations. In contrast, the data class change replaces some Java code, assuaging my concerns about code/package bloat in our provider ecosystem.

There are some downsides / open questions I'm still working through:

• union types, enums, automatic promotion of T to Output done in builders is not available in the case classes, need to make sure there's a way to use it all

• adding Kotlin toolchain to all provider builds has some cost, look into whether it's possible to produce the desired Kotlin usability effect while keeping a java-only toolchain

• open questions on whether Kotlin bias would introduce any problems with future uses of Pulumi from other languages like Scala, Clojure, or any other JVM-based languages that folks are interested in using

[myhau]

Hello 👋

We (at @virtuslab) recently started doing some experiments with Kotlin and Pulumi.

(I've been using Kotlin as my main language for quite some time + I'm super excited about Pulumi.)

Our plan is to show the PoC to the community (you) soon by publishing artifacts for providers already supported by Pulumi Java (at least GCP, AWS and Azure – classic and native).

The purpose of this is to have a working prototype that could enrich our discussions about the final design of the user interface / DSL. This could also become the final implementation, but there are a couple of roadblocks ahead (see the Implementation section below).

User interface (sneak peek)

See it in action (IntelliJ):

https://user-images.githubusercontent.com/4415632/192609526-51d08fe2-3e5f-4deb-95b1-c643fd6c7ed8.mp4

Or just see the end result (working example copied from our E2E tests – SDK was generated from GCP classic schema):

suspend fun main() {
    Pulumi.run { ctx ->

        // could also be getImage(family = "debian-11", project = "debian-cloud")
        val debianImage = getImage { 
            family("debian-11")
            project("debian-cloud")
        }

        val defaultInstance = instance("default-instance") {
            args {
                machineType("e2-micro")
                zone("europe-central2-a")
                tags("foo", "bar")
                bootDisk {
                    initializeParams {
                        image(debianImage.name)
                    }
                }
                networkInterfaces(
                    {
                        network("default")
                    }
                )
                metadata("foo" to "bar")
                metadataStartupScript("echo hi > /test.txt")
                serviceAccount {
                    scopes("cloud-platform")
                }
            }

            opts {
              protect(true)
            }
        }

        ctx.export("defaultInstanceId", defaultInstance.instanceId)
    }
}

Example features of the API

• Type-safe builders.
  • There is practically no boilerplate.
  • Convenient overloads everywhere.
  • Outputs work. For example, use zone(outputFromOtherResource) or zone("europe-central2-a").
  • metadata(mapOf("foo" to "bar")) can be just metadata("foo" to "bar")
• Coroutines - you don't need to operate on CompletableFuture anymore.
  • See debianImage.name usage above.
• Better nulls (no Optional<T> just idiomatic T?).

Implementation (PoC)

(This will be open-sourced very soon. I'll let you know!)

Our current solution is quite unique, in comparison to other languages.

• The code generation happens in a separate component written in Kotlin (not Go). We didn't touch or fork any repo in pulumi org.
  • Why? There is no better language for Kotlin code generation than Kotlin itself (same goes for other languages). Proper libraries (e.g. KotlinPoet) are there, language expertise is there.
• The generated code delegates the work to Java SDK (we don't interact with any gRPC server directly) and maps all types back and forth between Kotlin and Java worlds.

Biggest problems ahead

• Plugging the code-generation component (in Kotlin) into Pulumi's delivery pipeline or converting to a Go-based implementation.
  • I have a couple of ideas here and would love to start the conversation about this in a separate issue (aka "Pulumi Custom Language Generators"), even if it's very hard to do at this point.
• Code generation for pulumi convert, pulumi import, and docs (we'd have to port HCL model and implement HCL->Kotlin converter, right?).
• Potential JAR bloat.

[t0yv0]

Ah thanks for sharing that @myhau ! This may change our roadmap, if Virtus will support a fully Kotlin native codegen target, beyond a POC. Anyway this is exciting.

What I mostly wonder here is how would we envision integrating it into the Registry and Maven Central, and which party would own and keep up-to-date the Kotlin packages for all the providers. We can keep talking about this and figuring out options! There's been some early conversations about enabling third parties to "bring your own languages" to the Pulumi Registry, but perhaps we can work out some ad-hoc arrangement to get started.

It's good you mention delegates the work to Java SDK - that clarifies my biggest question here. So Kotlin would continue benefiting from continued bugfix/development on the Pulumi Java SDK.

Some of the issues you list are easy to get more information on, for example:

• Potential JAR bloat - build azure-native should give a good idea

Also:

• converting to a Go-based implementation - make sure we've exhausted options for making it "just work" based on the JVM implementation; our key concern is portability but JVM should have a decent track record here, perhaps with some small tweaks Pulumi can support that as is

[t0yv0]

CC @mikhailshilkov @lukehoban

[myhau]

Our team released the PoC: https://github.com/VirtuslabRnD/pulumi-kotlin.

Go through "Code examples" and "Getting started" sections in the README.md to use Pulumi Kotlin SDK in your toy project.

Supported providers

All the supported providers are listed in this table (it's scrollable horizontally). It includes Maven dependency blocks (just authenticate and copy & paste these to your pom.xml) and links to docs (Pulumi registry and Kotlin KDoc HTML generated from the code).

Docs / developer experience

IntelliJ is currently the best way to explore this DSL. KDoc (e.g. google-native) might serve as a reference, but it might be a bit hard to navigate. Integrating this with Pulumi Registry is probably the best way forward (?), but would require a lot of work at this point.

Feedback

We had a lot of fun working on this! Let us know what you think!

[rome-user]

@t0yv0 Regarding Clojure, the current java-sdk seems easy enough to use directly from Clojure. It feels like using Java standard library. With that said, there is one awkward point: the requirement of java.util.function means we cannot simply provide anonymous function to Pulumi/run.

This requires us to write a macro like so

(defmacro ->Consumer [f]
  `(reify java.util.function.Consumer
     (accept [this arg#]
       (~f arg#))))

Then we write (Pulumi/run (->Consumer (fn [ctx] ...))). With that said, I have uploaded stack-readme-java translation to Clojure here: https://github.com/rome-user/pulumi-stack-example

This leads to another issue: in Clojure many people use Leiningen rather than pure Maven. It takes some effort to generate pom.xml that Pulumi understands, but I just succeeded doing it.