Acceptance Testing

[oogali]

(Apologies for not using the issue template, but I did not feel it that it applied here)

I've pulled the provider source code down to my system and I'm attempting to run the acceptance tests against my personal NS1 account using the API key. However, I find that a large number of the acceptance tests are failing.

Is this a known issue? Or a sign that my local environment isn't properly configured?

$ NS1_APIKEY=... make testacc
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v  -timeout 120m
?       github.com/terraform-providers/terraform-provider-ns1   [no test files]
=== RUN   TestAccDataSourceDNSSEC
    testing.go:640: Step 0 error: errors during apply:

        Error: GET https://api.nsone.net/v1/zones/terraform-test-0mjvjo61navuszt.io/dnssec: 404 record not found

          on /var/folders/1w/v79253615qq39vgm1wq58y4m0000gn/T/tf-test121672980/main.tf line 6:
          (source code not available)

    testing.go:701: Error destroying resource! WARNING: Dangling resources
        may exist. The full state and error is shown below.

        Error: errors during refresh: GET https://api.nsone.net/v1/zones/terraform-test-0mjvjo61navuszt.io/dnssec: 404 record not found

        State: <nil>
--- FAIL: TestAccDataSourceDNSSEC (1.93s)
=== RUN   TestAccDataSourceRecord
--- PASS: TestAccDataSourceRecord (2.20s)
=== RUN   TestAccDataSourceZone_basic
--- PASS: TestAccDataSourceZone_basic (1.91s)
=== RUN   TestAccDataSourceZone_primary
    test_funcs.go:30: 
                Error Trace:    test_funcs.go:30
                                                        testing.go:954
                                                        testing_config.go:109
                                                        testing_config.go:27
                                                        testing.go:614
                Error:          lengths don't match: 1 != 0
                Test:           TestAccDataSourceZone_primary
    test_funcs.go:30: 
                Error Trace:    test_funcs.go:30
                                                        testing.go:954
                                                        testing_config.go:109
                                                        testing_config.go:27
                                                        testing.go:614
                Error:          lengths don't match: 1 != 0
                Test:           TestAccDataSourceZone_primary
--- FAIL: TestAccDataSourceZone_primary (1.48s)
=== RUN   TestAccDataSourceZone_dnssec
--- PASS: TestAccDataSourceZone_dnssec (3.07s)
=== RUN   TestProvider
--- PASS: TestProvider (0.00s)
=== RUN   TestProvider_impl
--- PASS: TestProvider_impl (0.00s)
=== RUN   TestAccAPIKey_basic
    testing.go:640: Step 0 error: After applying this step, the plan was not empty:

        DIFF:

        UPDATE: ns1_apikey.it
          account_manage_account_settings:  "false" => "false"
          account_manage_apikeys:           "false" => "false"
          account_manage_ip_whitelist:      "false" => "false"
          account_manage_payment_methods:   "false" => "false"
          account_manage_plan:              "false" => "false"
          account_manage_teams:             "false" => "false"
          account_manage_users:             "false" => "false"
          account_view_activity_log:        "false" => "false"
          account_view_invoices:            "false" => "false"
          data_manage_datafeeds:            "false" => "false"
          data_manage_datasources:          "false" => "false"
          data_push_to_datafeeds:           "false" => "false"
          dhcp_manage_dhcp:                 "true" => "false"
          dhcp_view_dhcp:                   "true" => "false"
          dns_manage_zones:                 "false" => "false"
          dns_records_allow.#:              "0" => "0"
          dns_records_deny.#:               "0" => "0"
          dns_view_zones:                   "false" => "false"
          dns_zones_allow_by_default:       "false" => "false"
          id:                               "627065b8de90150085434f3e" => "627065b8de90150085434f3e"
          ip_whitelist.#:                   "2" => "2"
          ip_whitelist.0:                   "1.1.1.1" => "1.1.1.1"
          ip_whitelist.1:                   "2.2.2.2" => "2.2.2.2"
          ip_whitelist_strict:              "true" => "true"
          ipam_manage_ipam:                 "true" => "false"
          ipam_view_ipam:                   "true" => "false"
          key:                              "CSGP8UXQPT5fj1dtcAGo" => "CSGP8UXQPT5fj1dtcAGo"
          monitoring_manage_jobs:           "false" => "false"
          monitoring_manage_lists:          "false" => "false"
          monitoring_view_jobs:             "false" => "false"
          name:                             "jnihyf4r1jauahg" => "jnihyf4r1jauahg"
          security_manage_active_directory: "true" => "false"
          security_manage_global_2fa:       "false" => "false"

        STATE:

        ns1_apikey.it:
          ID = 627065b8de90150085434f3e
          provider = provider.ns1
          account_manage_account_settings = false
          account_manage_apikeys = false
          account_manage_ip_whitelist = false
          account_manage_payment_methods = false
          account_manage_plan = false
          account_manage_teams = false
          account_manage_users = false
          account_view_activity_log = false
          account_view_invoices = false
          data_manage_datafeeds = false
          data_manage_datasources = false
          data_push_to_datafeeds = false
          dhcp_manage_dhcp = true
          dhcp_view_dhcp = true
          dns_manage_zones = false
          dns_view_zones = false
          dns_zones_allow_by_default = false
          ip_whitelist.# = 2
          ip_whitelist.0 = 1.1.1.1
          ip_whitelist.1 = 2.2.2.2
          ip_whitelist_strict = true
          ipam_manage_ipam = true
          ipam_view_ipam = true
          key = CSGP8UXQPT5fj1dtcAGo
          monitoring_manage_jobs = false
          monitoring_manage_lists = false
          monitoring_view_jobs = false
          name = jnihyf4r1jauahg
          security_manage_active_directory = true
          security_manage_global_2fa = false
--- FAIL: TestAccAPIKey_basic (0.46s)
...

[zahiar]

I have the same issue, for me the following resources having failing acceptance tests:

• DNS SEC
• API Key
• Application
• Pulsar Job
• Record
• Team
• User
• Zone
• NotifyList - though I have a fix for this one, pull request has been raised: #214

[Zach-Johnson]

Yes, the acceptance tests are currently failing, I believe due to some API changes that were not reflected in this provider. The exception may be Pulsar jobs - that may be due to lack of feature enablement on the account you are using to run the acceptance tests. I'm working on getting fixes for this prioritized internally, thanks for raising this.

[zahiar]

@Zach-Johnson Would it be possible to also have these acceptance tests run automatically when a pull request is opened? Similar to how the current test-pr workflow that runs unit tests, that could be expanded to run acceptance tests as well.

[Zach-Johnson]

@zahiar yeah this is definitely doable, we just don't have a lot of bandwidth for improvements on this provider right now.

[eravin-ns1]

We fixed most of the acceptance tests in v1.13.0, which was released today. The remaining test failures are actual bugs like issue 228 and we hope to fix them soon.

[eravin-ns1]

Note that some of the tests require having an account enabled for Pulsar, specific dedicated networks, or other paid features. I'm thinking we could adjust the names of the tests to flag the ones not likely to succeed for the "average" end user.

[zahiar]

@eravin-ns1 Thanks 🙏

One option could be to hide tests that rely on paid features behind a flag. So by default all free tests are run, and if you pass in a flag, then all free & paid tests are run. Just a thought to make it easier, given that most users won't have those paid features on their accounts.