CI is back to being slow again. Since removing a bunch of parallelism in
https://github.com/joneshf/terraform-provider-openwrt/pull/126, things
take a long time to run. Some of this is expected when there are large
changes. However, when adding a new Data Source/Resource, it seems like
the entire test suite runs again from scratch.
It seems like there should be some kind of caching in the tests that
isn't showing up, and it looks like it's due to the change in build
flags/environment variables used to run integration tests. Because we
use the same cache for all the tests, we end up doing something like:
Run the unit tests with one set of build flags/environment variables.
It creates a cached test run.
Run the integration tests a different set of build flags/environment
variables. This causes the cached tests from the unit tests to be
invalidated.
It creates a new cached test run.
In a future commit, run the unit tests with the original set of build
flags/environment variables. This causes the cached tests from the
integration tests to be invalidated.
Because we're always alternating between unit tests and integration
tests, the cached tests are never being used.
In order to address that, we're splitting the test jobs between
integration test and unit tests. We make a different cache for each job,
so they can reuse their cached tests when possible. This should allow
the tests to only run what has changed between commits, instead of
needing to re-run everything, all the time.
CI is back to being slow again. Since removing a bunch of parallelism in https://github.com/joneshf/terraform-provider-openwrt/pull/126, things take a long time to run. Some of this is expected when there are large changes. However, when adding a new Data Source/Resource, it seems like the entire test suite runs again from scratch.
It seems like there should be some kind of caching in the tests that isn't showing up, and it looks like it's due to the change in build flags/environment variables used to run integration tests. Because we use the same cache for all the tests, we end up doing something like:
Because we're always alternating between unit tests and integration tests, the cached tests are never being used.
In order to address that, we're splitting the test jobs between integration test and unit tests. We make a different cache for each job, so they can reuse their cached tests when possible. This should allow the tests to only run what has changed between commits, instead of needing to re-run everything, all the time.