search

pulumi / ci-mgmt

Configuration for all things CI
Apache License 2.0
11 stars 4 forks source link

Get Java up-to-date #506

Closed t0yv0 closed 11 months ago

t0yv0 commented 1 year ago

Continuing from https://github.com/pulumi/ci-mgmt/pull/504#issuecomment-1665676166

Need to make sure pulumi-java is up-to-date with the latest version and the SDKs regenerated.

If doing manually see checklist below.

For automation options consider prior art in here https://github.com/pulumi/pulumi-azure-native/blob/master/Makefile#L181-L189

Note also there is a nightly job https://github.com/pulumi/ci-mgmt/blob/master/.github/workflows/update-workflows-ecosystem-providers.yml#L10 that syncs providers to latest ci-mgmt with an automated PR. If this job could take care of re-generating the sdk/java files ("build_sdk") then much manual work could be avoided here in bringing these providers up-to-date.

As another alternative perhaps update-provider tool could bring java versions up-to-date. If we wen that direction we'd need to decouple the java version a provider uses from ci-mgmt first; that is it should be specified in every individual repo not in ci-mgmt repo. In that case update-provider could manage it.

Native Providers

pulumi-command uses the same version of java a pulumi does:

https://github.com/pulumi/pulumi-command/blob/7c5c4bd68e66950c042c5ce1ad3eb6f9bfda835c/Makefile#L88-L97

Bridged Providers

Note: The list of bridged providers is not up to date.

t0yv0 commented 1 year ago

We synced up with Ringo today and also I've noticed https://github.com/pulumi/pulumi-mongodbatlas/pull/261 that looks great, perhaps I'm not up to date on how current automation works. Strong preference to resolve this in automated fashion once and for all for ci-mgmt Pulumi internally, followed by Ringo trying things out on community side and filing issues if something is broken for community use case.

AFAIK it appears that upgrade-provider already can regenerate Java SDK and bump the java-gen version. @aq17 @iwahbe is that accurate, could you help me out here with assessment? Is it true that we could take the desired java version out of ci-mgmt (and Makefiles) and into configuration file per-repository, to decouple the need to make PRs to ci-mgmt; followed by making upgrade tooling automatically bump it? Taking a note for next ideation to clarify.

iwahbe commented 1 year ago

https://github.com/pulumi/pulumi-mongodbatlas/pull/261 was done manually. We could adjust upgrade-provider to do this automatically without much difficulty, but we haven't built that out yet.

guineveresaenger commented 12 months ago

I'm looking into this and having thoughts.

The reason we are struggling to upgrade Java at this point in history is that a change in javaGenVersion in the Makefile is not run in parallel with a codegen step.

First thought: is there ever a situation where we do not want java_gen on the latest version? if so, what is that situation?

If we assume that pulumi-java-gen@latest-release is always good (and update issues like this one seem to indicate this is at least the ideal case) then we could try the following:

  1. It appears as though we are building the release version into the binary name for pulumi-java.
  2. We use pulumictl to fetch the specified Java gen version and it turns out that pulumictl does quite a bit of heavy lifting for us in terms of platform, unzipping, and generally getting the binary and tools into the right place for the provider to use.

I figure this can go a few ways:

  1. update pulumictl to be able to discover and fetch latest
  2. write a script that discovers the latest java gen version and then feed that to the Makefile
  3. write a script to discover latest, download, untar, and handle platform targets, and remove pulumictl from the equation entirely.

Fast options are 1. or 2.

t0yv0 commented 12 months ago

We can't afford any more instability in the builds so a floating reference is a pretty bad idea. The Java reference should remain pinned and evolve via PRs, however the automation must be built out to keep it up to date.

iwahbe commented 12 months ago

I agree that we should avoid floating references here. Imagine the following scenario:

  1. A provider upgrade PR comes in, bumping a provider from v1.0.0 to v1.0.1. The latest version of java is v9.8.0.

  2. The latest java version changes from v9.8.0 to v9.8.1. This java update includes a codegen change.

  3. The PR from (1) is merged. As part of testing the default branch (master/main), CI checks that make build_java is clean. Because v9.8.0 and v9.8.1 build different versions, this fails.

I think the simplest design here is a workflow that:

  1. Downloads the provider repo.
  2. Sets .ci-mgmt.yaml's javaGenVersion to the latest version.
  3. Runs make ci-mgmt, which stamps out the updated java version to the Makefile.
  4. Runs make tfgen build_java to update the java SDK.
  5. Commit changes and open a PR.
guineveresaenger commented 12 months ago

I really don't love the idea of Yet Another Upgrade Workflow. It's better than what we have now but it's significantly different from other upgrades in that it touches .ci-mgmt which we're using as our "lockfile" for the javaGenVersion. I would very much like to be able to do the following:

  1. Set javaGenVersion globally
  2. when this happens, immediately trigger make build_java and commit (ideally add to whichever bridge/sdk upgrade we're doing at the time).

There's some --upgrade-java logic in upgrade-provider but I haven't been able to make it work for me and I don't yet understand the conditions under which this feature is meant to be used and run. I'll investigate there.

guineveresaenger commented 11 months ago

Java is up to date in all providers. Providers have migrated to using a "lockfile" approach that the Makefile will be reading from. CI config will no longer be updating the Makefile, and codegen will always rely on the checked-in java gen version. The upgrade tool will autodetect any new pulumi-java versions, and incorporate it into bridge and upstream updates.

pulumi-bot commented 11 months ago

Cannot close issue:

Please fix these problems and try again.