We should run generated quickstarts

dbolduc commented 2 years ago

We compile generated quickstarts, but we never run them. https://github.com/googleapis/google-cloud-cpp/blob/ac535f71e6c1baa3a81caa1fb565d5d0e29e1133/ci/etc/quickstart-config.sh#L25-L32

It will also take some effort to figure out what arguments should be used, and whether any setup is necessary to run a given quickstart.

Cleanup

[x] Change GOOGLE_CLOUD_CPP_TEST_REGION from GOOGLE_CLOUD_REGION. The former matches our other test variable names. The latter was me not thinking and copying GOOGLE_CLOUD_PROJECT which we use because it is used in other libraries and tools.

After #8254 the following libraries will need work:

Cannot run in our CI Builds

[x] google/cloud/assuredworkloads/:
[x] google/cloud/servicecontrol/
[x] google/cloud/accesscontextmanager/:
[x] google/cloud/policytroubleshooter/:
[x] google/cloud/channel/: requires a partner account id

Requires Custom Resources

[x] google/cloud/debugger/
[x] google/cloud/appengine/
[x] google/cloud/gameservices/
[x] google/cloud/iot/
[x] google/cloud/recommender/
[x] google/cloud/resourcemanager/
[x] google/cloud/talent/

Requires Permissions unavailable to a Service Account

[x] google/cloud/shell/
[x] google/cloud/oslogin/ (with my credentials using $USER@google.com worked)

Requires X-User-Project (#8245)

[x] google/cloud/securitycenter/
[x] google/cloud/accessapproval/
[x] google/cloud/automl/
[x] google/cloud/dlp/
[x] google/cloud/language/
[x] google/cloud/orgpolicy/
[x] google/cloud/retail/
[x] google/cloud/texttospeech/
[x] google/cloud/translate/
[x] google/cloud/videointelligence/
[x] google/cloud/vision/
[x] google/cloud/webrisk/
[x] google/cloud/asset/

Hangs

[x] google/cloud/cloudbuild/

Fails

[x] google/cloud/pubsublite/
[x] google/cloud/workflows/

coryan commented 2 years ago

Proposal

I will change the generator to create test targets for the quickstart programs. These test targets will:

Contain the integration-test tag so they are excluded from builds or build steps only running unit tests.
Contain the quickstart tag so they can all be executed in parallel by the build system (CMake or Bazel).
The test targets arguments will be described as environment variable names
A driver script will convert these arguments to the value of these variables configured for the CI build
Using this extra indirection allows us to run locally in our own projects

Overview

A build target for CMake may look like this:

include(CTest)
if (BUILD_TESTING)
    add_executable(filestore_quickstart "quickstart/quickstart.cc")
    target_link_libraries(filestore_quickstart
                          PRIVATE google-cloud-cpp::experimental-filestore)
    google_cloud_cpp_add_common_options(filestore_quickstart)
    add_test(
        NAME filestore_quickstart
        COMMAND cmake -P "${PROJECT_SOURCE_DIR}/cmake/quickstart-runner.cmake"
                $<TARGET_FILE:filestore_quickstart> GOOGLE_CLOUD_PROJECT)
    set_tests_properties(filestore_quickstart
                         PROPERTIES LABELS "integration-test;quickstart")
endif ()

For Bazel it may look like this:

sh_test(
    name = "quickstart_test",
    srcs = ["//bazel:quickstart-runner.sh"],
    args = [
        "filestore",
        "GOOGLE_CLOUD_PROJECT",
    ],
    data = ["//google/cloud/filestore/quickstart:quickstart"],
    tags = [
        "integration-test",
        "quickstart",
    ],
)

The generator will include this code in the scaffold, we only need to edit the names of the environment variables.

Details

I expect I will use the generator to update most libraries, I will run the generator with the --scaffold option, then use a loop to revert the changes to all files except for ${library}/{CMakeLists.txt,BUILD.bazel,quickstart/BUILD.bazel}. Then I will revert any unwanted changes. I anticipate I will need to enable a number of services and update our project configuration before sending the PR.

Differences from the current quickstart tests

We run the quickstart programs for different reasons:

When we change the quickstart program or the library we want to verify it still compiles and runs
We want to verify that using the CMake, Make, and Bazel files in the quickstart directory, as described in the README.md produces a working program.
When we install the libraries we want to verify the installed artifacts are usable.

I propose that we only solve the first problem with this PR. We have existing builds to verify the quickstart programs compile when the library is installed, e.g., https://github.com/googleapis/google-cloud-cpp/blob/62cb8319e40f97858bd821169ec36d31a66445bb/ci/verify_quickstart/CMakeLists.txt#L20.

I do not think we need to verify the quickstart programs run when compiled as part of the project and also run when compiled as standalone programs.

Future Work

We need to refactor the builds to use a similar scheme for hand-crafted libraries.

coryan commented 2 years ago

This is now done. We could improve some of the quickstarts tests (but the quickstarts themselves are Okay) by having a test organization and test folders, but they are good enough as-is.

googleapis / google-cloud-cpp