googleapis / gapic-generator-python

Generate Python API client libraries from Protocol Buffers.
Apache License 2.0
119 stars 67 forks source link

Add presubmit to check the docs build on the golden files #1713

Open parthea opened 1 year ago

parthea commented 1 year ago

Right now we have no docs testing for golden files in this repository. It would be great to have a way to test the docs via bazel rules, similar to how we run integration tests via bazel.

parthea commented 1 year ago

Ideally the docs build will run as part of the integration tests here:

https://github.com/googleapis/gapic-generator-python/blob/e4db9941078fe417e0d7b30bcd937e6c4dc0e6ba/.github/workflows/tests.yaml#L299-L300

The proposed solution is that simply running the command below should run docs tests as well

bazel test //tests/integration/... --test_output=errors

Running this command now, I see

INFO: Elapsed time: 24.905s, Critical Path: 23.92s
INFO: 50 processes: 15 internal, 35 linux-sandbox.
INFO: Build completed successfully, 50 total actions
//tests/integration:asset                                       (cached) PASSED in 0.1s
//tests/integration:credentials                                 (cached) PASSED in 0.0s
//tests/integration:eventarc                                    (cached) PASSED in 0.0s
//tests/integration:logging                                     (cached) PASSED in 0.1s
//tests/integration:redis                                       (cached) PASSED in 0.0s
//tests/integration:credentials_py_gapic_test                            PASSED in 3.1s
//tests/integration:eventarc_py_gapic_test                               PASSED in 3.5s
//tests/integration:logging_py_gapic_test                                PASSED in 6.7s
//tests/integration:redis_py_gapic_test                                  PASSED in 3.9s

There are 2 approaches that come to mind to achieve this:

  1. One approach could be to leverage the existing py_test rule that runs unit tests. We would add a folder in https://github.com/googleapis/gapic-generator-python/tree/main/gapic/templates/tests which includes code to run the docs build, such as the code below.

https://github.com/googleapis/gapic-generator-python/blob/e4db9941078fe417e0d7b30bcd937e6c4dc0e6ba/gapic/templates/noxfile.py.j2#L132-L144

Option 1 is prototyped here

  1. Another approach could be to add new bazel rules specifically for docs testing, like //tests/integration:*_py_gapicdocs_test , in addition the to the existing //tests/integration:*_py_gapic_test.

in order to do this, the bazel file below needs to be updated with py_test() targets which run the docs build.

https://github.com/googleapis/gapic-generator-python/blob/e4db9941078fe417e0d7b30bcd937e6c4dc0e6ba/tests/integration/BUILD.bazel#L77-L87

Option 2 is prototyped in this branch .

In my opinion, the preferred approach is option 1 because it leverages the existing py_test rule so we don't have to maintain multiple rules. Both approaches are working, so we can choose either one.