Slow Rebuilds Using Gradle+Skaffold+Buildpacks

[awesometown]

Expected behavior

When using "skaffold build" image rebuilds with buildpacks (after an initial successful build) should be fast (<10s ideally).

Actual behavior

Image rebuilds take 1m+. When switching to use the Jib builder, builds are consistently sub 10s.

Information

I have a Spring Boot-based application with a Gradle-based build process. The Spring Boot Gradle plug-in includes support for creating buildpacks based images, and these are what we are currently deploying into production. When adopting skaffold, we defaulted to using the buildpacks builder as well for consistency, but have found it to be noticeably slower than other options (specifically Jib, or rebuilding without skaffold).

Comparisons below. These are based on rebuilding the project with a minor code change (eg. changing a string from "Hello World!" to "Hello World!!"):

• running ./gradlew bootBuildImage (which runs the buildpacks builder to create a new image): 9s
• running skaffold build with Jib: 5s
• running skaffold build with buildpacks: 1m 20s

Further details:

• Skaffold version: 1.35.1
• Operating system: Ubuntu 20.04 running on WSL2 (Windows 10)
• Installed via: Homebrew
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta23
kind: Config
metadata:
  name: skaffold-boot

build:
  local:
    push: false
  artifacts:
    # - image: skaffold-boot
    #   context: .
    #   jib: {}
   - image: skaffold-boot
     buildpacks:
       builder: paketobuildpacks/builder:base

deploy:
  kubeContext: minikube
  kustomize:
    paths:
      - k8s/local

portForward:
  - resourceType: deployment
    resourceName: skaffold-boot
    namespace: default
    port: 8080
    localPort: 8080

Steps to reproduce the behavior

Sample project: https://github.com/awesometown/skaffold-boot

Logs for the various run types are in the linked repository.

The skaffold.yaml in the project contains two builder configs, one using buildpacks, one for jib. You can uncomment the appropriate one and re-run skaffold build after making a trivial change to the source code.

You can also see the performance of creating a buldpacks build without skaffold by running ./gradlew bootBuildImage

[briandealwis]

@awesometown would it be possible to include some logs to compare a 1st and 2nd run with skaffold build --timestamps?

[awesometown]

I added three logs to the linked repository:

• firstbuild.log -> fresh build (to the best of my knowledge; if there's more I should clean out prior to running, let me know)
• secondbuild.log -> just re-ran the build right after the first one, no code changes
• thirdbuild.log -> results of the build after making a small code change

[briandealwis]

@awesometown I don't see 1 minute or more:

• secondbuild.org: 10:36:46 to 10:37:08 = 22s
• thirdbuildorg: 10:38:14 to 10:38:36 = 22s

[awesometown]

Hmm... you're right. Sorry; I'd run them in the background and not done the mental math myself.

I just re-ran two more builds, each after having made a single string change in the application. The logs are here and here and are more representative of what I was seeing before. I'm not actually sure how I achieved those 22s times now :S

The bulk of the slowdown is all in the gradle run (~45s).

Running the (I think?) same gradle command outside the Skaffold/buildpacks build (gradlew --no-daemon assemble) takes about ~8s.

[tejal29]

related to #3665 mabye?