Open spkane opened 1 year ago
I have published the provider and found a bug in the code, which made things work for now, but I still think there is an underlying problem here.
The code had a search-and-replace error in the update test for the resource, which became very apparent once I published the provider and got around the terraform init
registry issue.
diff --git a/inventory/item_resource_test.go b/inventory/item_resource_test.go
index 43d4b41..3ccc54a 100644
--- a/inventory/item_resource_test.go
+++ b/inventory/item_resource_test.go
@@ -40,10 +40,10 @@ resource "inventory_item" "test" {
}
`,
Check: resource.ComposeAggregateTestCheckFunc(
- resource.TestCheckResourceAttr("item_item.test", "name", "jet"),
- resource.TestCheckResourceAttr("item_item.test", "tag", "SR-71 Blackbird"),
+ resource.TestCheckResourceAttr("inventory_item.test", "name", "jet"),
+ resource.TestCheckResourceAttr("inventory_item.test", "tag", "SR-71 Blackbird"),
// Verify dynamic values have any value set in the state.
- resource.TestCheckResourceAttrSet("item_item.test", "id"),
+ resource.TestCheckResourceAttrSet("inventory_item.test", "id"),
),
},
// Delete testing automatically occurs in TestCase
However, I am pretty sure that this was just a secondary issue and is not related to the failure that was occurring when the provider was unpublished. Now that it is published, I can not reproduce the error when dev_overrides
is set. Without the dev_overrides
set it still fails as before.
Also, changing the dev_overrides
line between these two settings makes no difference (at least now that it is published).
"registry.terraform.io/superorbital/inventory" = "/Users/spkane/dev/go/path/bin/"
versus"superorbital/inventory" = "/Users/spkane/dev/go/path/bin/"
I'm running into the same issue like OP. For a use such as 'Create and Read testing' or 'Update and Read testing' one can get away setting resource.TestStep.ImportState
to false. However, for use case 'ImportState testing' , resource.TestStep.ImportState
needs to be set true and then my test fails with the same error mentioned by OP.
Hi @spkane 👋 Thank you for raising this issue and sorry you ran into trouble here.
From reading through this report, it appears that this issue is limited to functionality for acceptance testing providers, which in your case is being managed by the github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource
package. I do not believe there is anything to be done in the terraform-plugin-framework Go module for this report, so I'm going to use GitHub's issue transfer functionality to move this issue to the https://github.com/hashicorp/terraform-plugin-sdk repository.
Now that this was transferred, some additional context --
The Terraform CLI dev_overrides
functionality directly interferes with how Terraform Providers are typically orchestrated by the testing code in helper/resource
. Declaring a dev_overrides
configuration is considered an escape hatch for provider developers wishing to manually test their provider code with real Terraform commands. The testing code meanwhile expects to fully manage the lifecycle of Terraform Provider binaries by starting and stopping the provider server via the TestCase
/TestStep
type ProtoV5ProviderFactories
/ProtoV6ProviderFactories
field definitions (similarly the Providers
and ProviderFactories
fields represent older ways to declare providers without explicitly writing the terraform-plugin-sdk provider server code; it was just implicitly handled). When managing the provider server lifecycle, Terraform CLI is executed with a special TF_REATTACH_PROVIDERS
environment variable that enables the running provider server to be discovered within Terraform's internals without its own provider server management. This enables the provider code-under-test to be instrumented with tooling such as debuggers, etc.
This leaves the testing code in an awkward position here. Detecting whether a provider is being managed via dev_overrides
is difficult as Terraform CLI does not implement commands or an interface to retrieve this type of configuration. Reimplementing some or all of that internal Terraform logic to detect the expected configuration file (based off Terraform's loading preferences), parse it, then see if this may be an issue for the test seems problematic.
The best recommendation for now I think would be to disable the usage of dev_overrides
when performing automated testing, either by commenting out the entry, moving the file, or pointing the TF_CLI_CONFIG_FILE
environment variable elsewhere. Installing the provider server binary and using dev_overrides
is not necessary for successfully testing unreleased provider code.
So, this is good to know.
However, I am not sure that this completely resolves the issue. In the description, I also mention that With dev_overerides disabled. Neither test works.
Re-working my workflow to avoid using dev_overrides
, except when actually testing using real terraform
commands is probably fine, but I was working on a provider that had yet to be published on the Hashicorp plugin registry and it looked like something in the test was trying to reach out to the registry during the init phase.
$ make testacc
************ Running Acceptance Tests ************
TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...
? github.com/superorbital/terraform-provider-inventory [no test files]
=== RUN TestAccItemDataSource
item_data_source_test.go:11: Step 1/1 error: Error running pre-apply refresh:
Error: Inconsistent dependency lock file
The following dependency selections recorded in the lock file are
inconsistent with the current configuration:
- provider registry.terraform.io/superorbital/inventory: required by this configuration but no version is selected
To make the initial dependency selections that will initialize the dependency
lock file, run:
terraform init
--- FAIL: TestAccItemDataSource (0.23s)
=== RUN TestAccItemResource
item_resource_test.go:10: Step 1/3 error: Error running pre-apply refresh:
Error: Inconsistent dependency lock file
The following dependency selections recorded in the lock file are
inconsistent with the current configuration:
- provider registry.terraform.io/superorbital/inventory: required by this configuration but no version is selected
To make the initial dependency selections that will initialize the dependency
lock file, run:
terraform init
--- FAIL: TestAccItemResource (0.21s)
FAIL
FAIL github.com/superorbital/terraform-provider-inventory/inventory 0.451s
FAIL
make: *** [testacc] Error 1
Maybe this is caused by something else in the environment?
Apologies if I misunderstood something about your report. One potential quirk with having an explicit:
terraform {
required_providers {
inventory = {
source = "superorbital/inventory"
}
}
}
In the test configuration, is that the testing code may require an additional TF_ACC_PROVIDER_NAMESPACE=superorbital
environment variable set. This configuration is not normally necessary for acceptance testing (as it will get implicitly added), but the explicit required_providers
configuration may be causing it to be required.
I believe you can verify whether this is the case by enabling TF_LOG=TRACE
in your test execution and looking for a TRACE entry with Setting Terraform CLI reattach configuration
-- if the reattach configuration has registry.terraform.io/hashicorp/inventory
then Terraform would raise this error since there would be a mismatch of provider addresses between the reattach configuration and the one defined explicitly in the configuration. Setting TF_ACC_PROVIDER_NAMESPACE
should swap out hashicorp
in that reattach configuration.
I am working on a currently unpublished Terraform provider. The source code is here, but it is not in the registry at the moment.
The provider works, but I have
dev_overrides
set in my.terraformrc
file this causes the acceptance tests for only the resource to fail, as the tests seem to try and run aterraform init
, which does not work whendev_overrides
are set.Module version
Relevant provider source code
main.go
provider_test.go
import ( "github.com/hashicorp/terraform-plugin-framework/providerserver" "github.com/hashicorp/terraform-plugin-go/tfprotov6" )
const ( // providerConfig is a shared configuration to combine with the actual // test configuration so the Inventory client is properly configured. providerConfig = `terraform { required_providers { inventory = { source = "superorbital/inventory" } } }
Configure the connection details for the Inventory service
provider "inventory" { } ` )
var ( // testAccProtoV6ProviderFactories are used to instantiate a provider during // acceptance testing. The factory function will be invoked for every Terraform // CLI command executed to create a provider server to which the CLI can // reattach. testAccProtoV6ProviderFactories = map[string]func() (tfprotov6.ProviderServer, error){ "inventory": providerserver.NewProtocol6WithError(New()), } )
Terraform Configuration Files
~/.terraformrc
Makefile
generate: $(info **** Generating Provider Documentation ****) go generate ./...
install: $(info **** Building and Installing Binary ****) go install .
test: $(info **** Running Unit Tests ****) go test -v -count=1 -parallel=4 ./...
testacc: $(info **** Running Acceptance Tests ****) TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...
dev_overerides
disabled. Neither test works.Steps to Reproduce
TF_ACC=1 go test -count=1 -parallel=4 -timeout 10m -v ./...
References
N/A