Error: Unable to decode resource from state

brandon-fryslie commented 3 years ago

Description

I've installed driftctl via homebrew. I try to run it against our infrastructure via command:

driftctl scan --from tfstate+s3://our-s3-bucket/

I get the error:

ERRO[0007] Unable to decode resource from state name=my-instance type=aws_instance
Scanned resources:    (0)
attribute "secondary_private_ips" is required

Note: Our s3 bucket contains the terraform state for 10-20 modules, some of which use the terraform workspace feature.

Environment

OS: MacOS 10.15.7
driftctl version: 0.7.1

How to reproduce

Run command against our infrastructure.

Possible Solution

Possibly this could be a documentation issue? I cannot find any information about the error in the docs (or existing github issues). I would like to try driftctl out but there does not seem to be any information on getting it to run against my own existing account. I could start a new account just to test driftctl, but for it to be useful I need to be able to use it with my existing resources.

I think the solution here would be one or more of:

Fix the bug in driftctl (if there is a bug)
Improve the error message, possibly linking to documentation on how to resolve
Provide online documentation explaining what the error means and how to resolve it
Provide online documentation with requirements for running this against an existing account

I currently do not have an idea of how to proceed. I would love to be able to use the application. Thank you for your help!

eliecharra commented 3 years ago

@brandon-fryslie Thanks for your report, which version of AWS provider and terraform are you using ? driftctl only support provider from version 3.x and above https://docs.driftctl.com/0.7.1/limitations#terraform--providers-support

Tocyuki commented 3 years ago

Same problem is occured in my environment.

Environment

exec via docker
driftctl version is v0.7.1
AWS provider version is v3.38.0

Command

docker-compose run --rm driftctl scan --from tfstate+s3://${tfstate_bucket}/

docker-compose.yml

version: '3.8'

services:
  driftctl:
    image: cloudskiff/driftctl:v0.7.1
    working_dir: /terraform
    volumes:
      - ./terraform:/terraform

Messages

ERRO[0013] Unable to decode resource from state name=app type=aws_codedeploy_app

attribute "unique_id" is required

wbeuil commented 3 years ago

Hello @brandon-fryslie, can you give the AWS provider with which you created your state. It could potentially be two things: either you used a version 2.X and we're not supporting it as per our doc or terraform integrated a breaking change in a minor version of the 3.X provider for your resource.

wbeuil commented 3 years ago

Hello @Tocyuki, the problem here is that terraform integrated a breaking change in the schema for the aws_codedeploy_app resource between v3.19 and v3.38. You can see the schema for v3.19 (the provider version that we use for now in driftctl) here where it states that the schema needs a unique_id attribute and this one for v3.38 (the one you created your state with) here where the schema doesn't need the unique_id anymore.

You would need for your state to work to wait for driftctl v0.9.0. Thanks !

brandon-fryslie commented 3 years ago

Thanks for your report, which version of AWS provider and terraform are you using ? driftctl only support provider from version 3.x and above https://docs.driftctl.com/0.7.1/limitations#terraform--providers-support

can you give the AWS provider with which you created your state. It could potentially be two things: either you used a version 2.X and we're not supporting it as per our doc or terraform integrated a breaking change in a minor version of the 3.X provider for your resource.

I do not know which state is breaking driftctl, so I don't know which version of the provider is the problem. We had some using >= 3.3.0 and some using 2.69.0. I went and updated all of our providers to >= 3.3.0 and reran terraform apply. There were no changes to the resources caused by changing the provider version.

I am still seeing the same error (identical). Is it possible for driftctl itself to keep track of which provider versions are required, and check the state file for that information? That would help provide a more better error message to the user.

I'm now trying to scan a subfolder in the bucket rather than the entire state and driftctl has not crashed yet. However I'm not sure it will properly recognize all my resources since I'm only reading a small part of the overall terraform state (the scan is still running). I'll go down this route until I can find the individual state file that is breaking driftctl which should help debug the issue.

driftctl scan --from tfstate+s3://my-terraform-state/security-groups
Scanning resources: ⣷ (4566)

eliecharra commented 3 years ago

Thanks a lot for your time @brandon-fryslie 🙏🏻 We are currently working on a core refactoring of driftctl to support any terraform version (tracking issue is #347). It may fix all theses kinds of problems. Every error with attribute "something" is required is caused by a version incompatibility between terraform provider and driftctl. We hope to publish release a soon as possible but it's a lot of work, almost all our bandwidth is actually dedicated to this issue.

sjourdan commented 3 years ago

all providers versions are now supported (v2x and v3x) starting with driftctl 0.10.0 https://github.com/cloudskiff/driftctl/releases/tag/v0.10.0

Please reopen or comment on the issue if needed!

Tocyuki commented 3 years ago

v.0.10.0 did not solve the problem.

Environment

exec via docker
driftctl version is v0.10.0
AWS provider version is v3.44.0

Command

docker-compose run --rm driftctl scan --from tfstate+s3://${tfstate_bucket}/terraform.tfstate

docker-compose.yml

version: '3.8'

services:
  driftctl:
    image: cloudskiff/driftctl:v0.10.0
    working_dir: /terraform
    volumes:
      - ./terraform:/terraform

Messages

ERRO[0007] Unable to decode resource from state name=app type=aws_codedeploy_app
Scanned states (1)      
attribute "unique_id" is required

eliecharra commented 3 years ago

@sjourdan I'm closing this one and opened a new issue there https://github.com/cloudskiff/driftctl/issues/690 for @Tocyuki's issue as this is not really the same thing. This issue talks about error on an supported resource, but @Tocyuki scan fails on codedeploy which is currently not supported, this should not happen.

There is two topics there @Tocyuki, I think that you do not use the right version of terraform provider to scan you state. First you have to find which version of the provider was used to generate this state, this can be found in your terraform.lock.hcl :

Then you have to run driftctl with the good provider version, for you should be something like that :

version: '3.8'

services:
  driftctl:
    image: cloudskiff/driftctl:v0.10.0
    working_dir: /terraform
    environment:
      DCTL_TF_PROVIDER_VERSION: '3.44.0' # Put your provider version here
    volumes:
      - ./terraform:/terraform

It will probably fix your issue, but anyway this shouldn't have failed so we are gonna fix this in #690.

Keep us informed and thanks for this report 🙏🏻

snyk / driftctl

Error: Unable to decode resource from state #488