search

GoogleCloudPlatform / appengine-plugins

A client Java library to manage App Engine Java applications for any project that performs App Engine Java application management. For example, the Maven, Gradle and Eclipse App Engine plugins, custom user tools, etc.
Apache License 2.0
41 stars 27 forks source link

Gradle parallel build support #1001

Open czp3009 opened 2 years ago

czp3009 commented 2 years ago

When building project with multiple sub-projects, gradle parallel builds are key to improving CI speed.

In this case, gradle will try to build all sub-projects at the same time.

If there are multiple sub-projects to deploy (multiple services), these sub-projects will all have the task downloadCloudSdk.

This is a file download task, if gradle start downloading at the same time, the task will fail.

fail message:

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':default-service:downloadCloudSdk'.
> java.nio.file.FileAlreadyExistsException: /root/.cache/google-cloud-tools-java/managed-cloud-sdk/downloads/google-cloud-sdk.tar.gz

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':example-service:downloadCloudSdk'.
> java.nio.file.FileAlreadyExistsException: /root/.cache/google-cloud-tools-java/managed-cloud-sdk/downloads/google-cloud-sdk.tar.gz

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

There are several workaround, but none of them are perfect:

  1. Not using parallel builds. This results in very slow builds, especially in large projects.
  2. Use an image with gcloud already installed as a CI build image. This may result in the gcloud not being up to date, and the build image containing the compiler needed for the build, which may not been created by the CI user.

For tasks that require requests to external resources, such as the file system, gradle provides an API to avoid execute multi time: https://docs.gradle.org/current/userguide/build_services.html

I don't know much about gradle plugins, can you please see if you can use build service to solve this problem.

elefeint commented 2 years ago

Thanks! We'll keep it open as an enhancement request.