"chef-automate diagnostics run" fails with IAMv2

[trickyearlobe]

Describe the bug

Reported by customer and verified with A2 build 20200127203438

With a clean install of A2 with IAMv1 enabled A2 diagnostics pass (with 2 skips related to IAMv2) as expected.

After migrating to IAMv2, A2 diagnostics fail.

To Reproduce

• Perform a fresh install of A2
• Run chef-automate diagnostics run and observe tests pass, or are skipped as expected.
• Run chef-automate iam upgrade-to-v2
• Run chef-automate diagnostics run and observe some tests fail (causing whole diagnostic to fail)

Versions (please complete the following information):

• OS: CentOS 7.6
• Automate Build Number 20200127203438

Diagnostics output with IAMv2 enabled

chef-automate diagnostics run
[✗] Generating data for auth-policies
      Failed to POST https://localhost/api/v0/auth/policies. Received unexpected status code 400
      Response Body:
       {"error":"authz-service set to v2","code":9,"message":"authz-service set to v2","details":[{"@type":"type.googleapis.com/chef.automate.domain.authz.common.ErrorShouldUseV2"}]}
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/integration.(*mustJSONDecodeSuccess).WithValue
        /src/components/automate-cli/pkg/diagnostics/integration/helpers.go:59
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/integration.CreatePolicyOnToken
        /src/components/automate-cli/pkg/diagnostics/integration/api.go:97
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/integration.CreateAuthPoliciesDiagnostic.func1
        /src/components/automate-cli/pkg/diagnostics/integration/auth_policies.go:30
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/runner.(*Runner).Run
        /src/components/automate-cli/pkg/diagnostics/runner/runner.go:181
      main.runDiagnosticsRunCmd
        /src/components/automate-cli/cmd/chef-automate/diagnostics.go:129
      github.com/spf13/cobra.(*Command).execute
        /src/vendor/github.com/spf13/cobra/command.go:762
      github.com/spf13/cobra.(*Command).ExecuteC
        /src/vendor/github.com/spf13/cobra/command.go:852
      github.com/spf13/cobra.(*Command).Execute
        /src/vendor/github.com/spf13/cobra/command.go:800
      main.Execute
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:87
      main.EarlyUpdate.func1
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:75
      main.nextCLIOrElse
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:199
      main.EarlyUpdate
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:73
      main.main
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:66
      runtime.main
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/proc.go:203
      runtime.goexit
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/asm_amd64.s:1357
      Could not create policy
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/integration.CreatePolicyOnToken
        /src/components/automate-cli/pkg/diagnostics/integration/api.go:100
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/integration.CreateAuthPoliciesDiagnostic.func1
        /src/components/automate-cli/pkg/diagnostics/integration/auth_policies.go:30
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/runner.(*Runner).Run
        /src/components/automate-cli/pkg/diagnostics/runner/runner.go:181
      main.runDiagnosticsRunCmd
        /src/components/automate-cli/cmd/chef-automate/diagnostics.go:129
      github.com/spf13/cobra.(*Command).execute
        /src/vendor/github.com/spf13/cobra/command.go:762
      github.com/spf13/cobra.(*Command).ExecuteC
        /src/vendor/github.com/spf13/cobra/command.go:852
      github.com/spf13/cobra.(*Command).Execute
        /src/vendor/github.com/spf13/cobra/command.go:800
      main.Execute
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:87
      main.EarlyUpdate.func1
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:75
      main.nextCLIOrElse
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:199
      main.EarlyUpdate
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:73
      main.main
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:66
      runtime.main
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/proc.go:203
      runtime.goexit
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/asm_amd64.s:1357
[✓] Generating data for auth-teams
[✓] Generating data for auth-users
[✓] Generating data for cfgmgmt-actions
[✓] Generating data for cfgmgmt-ccr
[✓] Generating data for cfgmgmt-jobs-config
[✓] Generating data for cfgmgmt-liveness
[✓] Generating data for compliance-node
[✓] Generating data for compliance-profile
[✓] Generating data for compliance-report
[✓] Generating data for compliance-scanning
[✓] Generating data for compliance-scan-job
[✓] Generating data for compliance-secret
[✓] Generating data for deployment
[✓] Generating data for iam-v2
[✓] Generating data for license
[✓] Generating data for notification-rules
[✓] Generating data for ingest-purge
[✓] Generating data for compliance-purge
[✓] Generating data for event-feed-purge
[✗] Cleaning up auth-policies
      Key not found
      github.com/chef/automate/components/automate-cli/pkg/diagnostics.init
        /src/components/automate-cli/pkg/diagnostics/context.go:20
      runtime.doInit
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/proc.go:5222
      runtime.doInit
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/proc.go:5217
      runtime.main
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/proc.go:190
      runtime.goexit
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/asm_amd64.s:1357
      Could not load generated context
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/integration.CreateAuthPoliciesDiagnostic.func3
        /src/components/automate-cli/pkg/diagnostics/integration/auth_policies.go:57
      github.com/chef/automate/components/automate-cli/pkg/diagnostics/runner.(*Runner).Run
        /src/components/automate-cli/pkg/diagnostics/runner/runner.go:240
      main.runDiagnosticsRunCmd
        /src/components/automate-cli/cmd/chef-automate/diagnostics.go:129
      github.com/spf13/cobra.(*Command).execute
        /src/vendor/github.com/spf13/cobra/command.go:762
      github.com/spf13/cobra.(*Command).ExecuteC
        /src/vendor/github.com/spf13/cobra/command.go:852
      github.com/spf13/cobra.(*Command).Execute
        /src/vendor/github.com/spf13/cobra/command.go:800
      main.Execute
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:87
      main.EarlyUpdate.func1
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:75
      main.nextCLIOrElse
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:199
      main.EarlyUpdate
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:73
      main.main
        /src/components/automate-cli/cmd/chef-automate/chef-automate.go:66
      runtime.main
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/proc.go:203
      runtime.goexit
        /hab/pkgs/core/go/1.13.1/20191001233339/src/runtime/asm_amd64.s:1357
[✓] Cleaning up auth-teams
[✓] Cleaning up auth-users
[✓] Cleaning up cfgmgmt-ccr
[✓] Cleaning up cfgmgmt-jobs-config
[✓] Cleaning up cfgmgmt-liveness
[✓] Cleaning up compliance-node
[✓] Cleaning up compliance-profile
[✓] Cleaning up compliance-scanning
[✓] Cleaning up compliance-scan-job
[✓] Cleaning up compliance-secret
[✓] Cleaning up deployment
[✓] Cleaning up iam-v2
[✓] Cleaning up license
[✓] Cleaning up notification-rules
[✓] Cleaning up ingest-purge
[✓] Cleaning up compliance-purge
[✓] Cleaning up event-feed-purge
DiagnosticsError: One or more diagnostics checks failed: Could not create policy: Failed to POST https://localhost/api/v0/auth/policies. Received unexpected status code 400
Response Body:
 {"error":"authz-service set to v2","code":9,"message":"authz-service set to v2","details":[{"@type":"type.googleapis.com/chef.automate.domain.authz.common.ErrorShouldUseV2"}]}; Could not load generated context: Key not found

[bcmdarroch]

Sorry you hit this! This should be resolved by the time of our force-upgrade in a few weeks. We're probably going to port those auth_ generators to the IAM v2 specific diagnostic.

[pag-r]

The core issue here is that there are no way to add a node and credentials from the Automate UI. This is where I've noticed it first. While trying to add node or credentials it returns HTTP Code 400.

[susanev]

@pag-r did you mean to add this comment to a different issue?

[pag-r]

@susanev no, this is what I noticed when I saw this error in my environment. At least in my case adding creds and nodes is impossible no more. Only way to make it work again is to chef-automate reinstall

[bcmdarroch]

@pag-r do you mind opening a new issue that describes how to reproduce the error you're seeing in your environment? It sounds like the error you're describing may be a separate problem we should investigate

We might also be able to help if you post in https://community-slack.chef.io/

[pag-r]

@susanev and @bcmdarroch, I was wrong, there is no connection between diagnostics run and HTTP 400. I just checked it when diagnostics failed after switching to v2 but forgot that I've applied policyfile. tl;dr: please ignore my comment from Feb 10th.

[susanev]

work in progress: https://github.com/chef/automate/pull/2842

[gsreynolds]

2842 was merged, the updated diagnostics should land in the next Automate release after the current 20200220011437

[susanev]

we confirmed this is fixed in acceptance, closing.