search

PrefectHQ / terraform-provider-prefect

Terraform Provider for Prefect Cloud
https://registry.terraform.io/providers/PrefectHQ/prefect/latest/docs
Apache License 2.0
29 stars 13 forks source link

Run acceptance tests against Prefect server in Docker Compose #192

Open mitchnielsen opened 3 weeks ago

mitchnielsen commented 3 weeks ago

Summary

In https://github.com/PrefectHQ/terraform-provider-prefect/issues/186, we added support for running Prefect in Docker Compose and being able to test terraform plan/apply commands against it.

This is helpful for local development and testing, but we may also be able to use this in CI.

We could replace the PREFECT_API_URL value in the CI file to point to something like http://localhost:4200, and tell CI to spin up Prefect using the Docker Compose configuration.

This means we'd be testing in an ephemeral instance of Prefect instead of against dev/staging environments, which can cause transient errors due to environment pollution.

One caveat mentioned by @parkedwards:

just a caveat - one wrinkle is that our OSS server isn't an exact match/port of our SaaS (which the current acceptance tests are mostly oriented towards supporting) - the OSS server has a subset of features and endpoints. so there may be some extra potholes we hit as we try to marry the tests<>this local prefect server instance. but i like where this is going

Acceptance criteria

parkedwards commented 3 weeks ago

something to think about - not sure if we can do this with golang tests, but in pytest you can create test groups and invoke them separately. i really do like this local testing facility, but if it only supports non-Cloud-specific features/resource, maybe we should group our tests to be OSS vs. Cloud and run them with separate make commands. Cloud would be the full thing, but OSS would be a subset - the benefit would be that we're testing provider compatibility with OSS, but also shorter feedback loop. and then the Cloud tests can run in CI/CD

mitchnielsen commented 3 weeks ago

Yeah good points, we'll have to delineate which ones are Cloud-only for sure.

I do think the OSS tests could run in CI/CD too if we wanted, since all we need is that prefecthq/prefect image running.

mitchnielsen commented 3 weeks ago

As for grouping them in Golang, I think there's usually a few options:

mitchnielsen commented 3 weeks ago

https://pkg.go.dev/testing#hdr-Skipping

parkedwards commented 3 weeks ago

tags sound like the way to go 🙌

mitchnielsen commented 1 week ago

After some discussion, it may be more valuable to prioritize https://github.com/PrefectHQ/terraform-provider-prefect/issues/218 for now as that would increase the reliability of tests in Prefect Cloud (which can run all tests, rather than just a subset that could run when testing against OSS).