turbot / steampipe-plugin-terraform

Use SQL to instantly query resources, data sources and more from Terraform code. Open source CLI. No DB required.
https://hub.steampipe.io/plugins/turbot/terraform
Apache License 2.0
28 stars 3 forks source link

Support recursive search for tfstate in s3 bucket? #105

Closed keyolk closed 2 months ago

keyolk commented 4 months ago

I am testing steampipe against tfstates. I am managing multiple tfstate files in one s3 bucket. After testing, it seems that formats such as "$bucketname/*/.tfstate" are not supported. In practice, we manage a lot of tfstates, and it would be very helpful if recursive search is possible.

Found some test cases which looks related to this https://github.com/turbot/steampipe-plugin-sdk/blob/develop/plugin/get_source_files_test.go#L165 But it never work as I expected

ParthaI commented 3 months ago

Hi @keyolk,

I apologize for the delayed response.

Here are a few key points for you to review:

connection "tf_bucket" {
  plugin = "terraform"

  configuration_file_paths = [
    "s3::https://test-delete90.s3.us-east-2.amazonaws.com//**/*.tf?aws_profile=default"
  ]
}

Query Result:


> select * from tf_bucket.terraform_resource
+---------------------+----------------------+--------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
| name                | type                 | mode   | address                                 | arguments                                                                                                                                                                                                               >
+---------------------+----------------------+--------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
| example             | tls_private_key      | <null> | tls_private_key.example                 | {"algorithm":"RSA"}                                                                                                                                                                                                     >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
| named_test_resource | aws_acm_certificate  | <null> | aws_acm_certificate.named_test_resource | {"certificate_body":"${tls_self_signed_cert.example.cert_pem}","private_key":"${tls_private_key.example.private_key_pem}","tags":{"name":"turbot-test-20200125-create-update"}}                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
| example             | tls_self_signed_cert | <null> | tls_self_signed_cert.example            | {"allowed_uses":["key_encipherment","digital_signature","server_auth"],"private_key_pem":"${tls_private_key.example.private_key_pem}","subject":{"common_name":"turbot.com","organization":"Turbot HQ Pvt. Ltd."},"valid>
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
|                     |                      |        |                                         |                                                                                                                                                                                                                         >
+---------------------+----------------------+--------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->

I haven’t yet tried it with a tfstate file in an S3 bucket. Could you please give it a try and let us know if you encounter any further issues?

Thank you!

keyolk commented 2 months ago

@ParthaI Hi! basically, it works. But extreamely slow and most of case, it returns error like below

steampipe query "select name, type, mode, address, arguments, attributes, path from tfstate.terraform_resource where mode = 'managed' limit 10" --output json
Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000)

Any idea?

keyolk commented 2 months ago

And seems it just follow AWS_REGION env even I set aws_region in its file_path which make much confusing

ParthaI commented 2 months ago

Hello @keyolk, thank you for testing these things out!

steampipe query "select name, type, mode, address, arguments, attributes, path from tfstate.terraform_resource where mode = 'managed' limit 10" --output json Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000)

It seems the error occurs while parsing the content of a specific state file. It would be great if you could figure out which state file content is failing.

The information you provide will greatly assist me in replicating the issue you're encountering.

Thank you!

keyolk commented 2 months ago

@ParthaI it took about 70sec on my m1 macbook

connection "tfstate" {
  plugin = "terraform"

  state_file_paths = [
    "s3::https://<mybucket>.s3.<my_region>.amazonaws.com//<my_object_prefix>/**/terraform.tfstate?aws_profile=<my_profile>&aws_region=<my_region>"
  ]
}
$ time steampipe query "select name, type, mode, address, arguments, attributes, path from tfstate.terraform_resource where mode = 'managed' limit 10" --output json
...
real    1m14.606s
user    0m4.186s
sys     0m4.547s

$ steampipe --version
Steampipe v0.23.5

$ file $(which steampipe)                                                                                                                                                                               
/opt/homebrew/bin/steampipe: Mach-O 64-bit executable arm64

$ steampipe plugin list | grep terraform
| hub.steampipe.io/plugins/turbot/terraform@latest       | 0.12.0  | tfstate |

$ du -chs state.json
256K    state.json
256K    total

for single file like the above it can get result but takes too much time about 80 sec

$ find . | grep tfstate | wc -l                                                                                                                                                                                
      45
$ du -chs *
...
4.2M    total

And the above case it ahs 45 tfstate with size of 4M it got error

Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000)

Also some error logs from plugin.log

2024-09-10 08:26:05.015 UTC [WARN]  initializePlugin failed: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x140005f36c0)
2024-09-10 08:26:05.019 UTC [WARN]  PluginManager Get failed for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000d10c00)
2024-09-10 08:27:11.809 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956798220: readAndStreamAsync failed to read previous rows from cache: context canceled publisher tfstate-1725956798220 (tfstate-1725956798220)
2024-09-10 08:27:11.809 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956798220: QueryCache AbortSet - aborting request  with error set request has no subscribers (0 subscribers) (tfstate-1725956798220)
2024-09-10 08:27:11.809 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956798220: queryData.streamRows returned error: context canceled
2024-09-10 08:27:11.809 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956798220: executeForConnection tfstate returned error context canceled, writing to CHAN
2024-09-10 08:28:19.991 UTC [WARN]  failed to set connection config for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference
2024-09-10 08:28:19.991 UTC [WARN]  initializePlugin failed: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000bb6100)
2024-09-10 08:28:19.995 UTC [WARN]  PluginManager Get failed for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000ba6108)
2024-09-10 08:28:58.695 UTC [WARN]  failed to set connection config for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference
2024-09-10 08:28:58.695 UTC [WARN]  initializePlugin failed: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14003566cc0)
2024-09-10 08:28:58.697 UTC [WARN]  PluginManager Get failed for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000f240a8)
2024-09-10 08:29:31.244 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956938334: readAndStreamAsync failed to read previous rows from cache: context canceled publisher tfstate-1725956938334 (tfstate-1725956938334)
2024-09-10 08:29:31.244 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956938334: QueryCache AbortSet - aborting request  with error set request has no subscribers (0 subscribers) (tfstate-1725956938334)
2024-09-10 08:29:31.244 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956938334: queryData.streamRows returned error: context canceled
2024-09-10 08:29:31.244 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725956938334: executeForConnection tfstate returned error context canceled, writing to CHAN
2024-09-10 08:30:18.276 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  failed to delete the temp directory /var/folders/7g/1dw9yqgj4mzcrfgcsqj9vs6w0000gq/T/steampipe-plugin-terraform: unlinkat /var/folders/7g/1dw9yqgj4mzcrfgcsqj9vs6w0000gq/T/steampipe-plugin-terraform/tfstate/2024-09-10T08:30:13Z/aws/chat-dev/apne1: directory not empty
2024-09-10 08:31:33.674 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.674 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: streamRows execution has failed: tfstate-1725957057522 - calling queryCache.AbortSet (tfstate: runtime error: index out of range [0] with length 0)
2024-09-10 08:31:33.674 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryCache AbortSet - aborting request  with error tfstate: runtime error: index out of range [0] with length 0 (1 subscriber) (tfstate-1725957057522)
2024-09-10 08:31:33.674 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: queryData.streamRows returned error: tfstate: runtime error: index out of range [0] with length 0
2024-09-10 08:31:33.674 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: executeForConnection tfstate returned error tfstate: runtime error: index out of range [0] with length 0, writing to CHAN
2024-09-10 08:31:33.674 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: error channel received tfstate: runtime error: index out of range [0] with length 0
2024-09-10 08:31:33.691 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.700 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.708 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.729 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.729 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: readAndStreamAsync failed to read previous rows from cache: tfstate: runtime error: index out of range [0] with length 0 publisher tfstate-1725957057522 (tfstate-1725957057522)
2024-09-10 08:31:33.772 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.845 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.878 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:34.003 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:34.051 UTC [WARN]  steampipe-plugin-terraform.plugin: [WARN]  1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:38.355 UTC [WARN]  refreshConnections failed with err
        failed to create transaction to perform update query: failed to connect to `user=root database=steampipe`: 127.0.0.1:9193 (127.0.0.1): server error: FATAL: the database system is shutting down (SQLSTATE 57P03)
2024-09-10 08:31:38.391 UTC [WARN]  setAllConnectionStateToError failed to acquire connection from pool: failed to connect to `user=root database=steampipe`: 127.0.0.1:9193 (127.0.0.1): tls error: read tcp 127.0.0.1:51540->127.0.0.1:9193: read: connection reset by peer
2024-09-10 08:31:38.392 UTC [WARN]  failed to send error notification, error

And about the env AWS_REGION it take the env first while I set the aws_region in the terraform.spc Lets say I have "s3::https://<mybucket>.s3.<my_region>.amazonaws.com//<my_object_prefix>/**/terraform.tfstate?aws_profile=<my_profile>&aws_region=us-east-1" and AWS_REGION=us-west-2 it would take bucket from us-west-2

And also there are no any log from user side, if I set the wrong path or region or bucket, ... May better make user to check if the given spc config is valid when it take 0 bucket or object with the connection

ParthaI commented 2 months ago

Hello @keyolk,

Thank you for providing such detailed information!

I was able to replicate the error: Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000). To address this, I’ve submitted a PR that I believe should resolve the issue.

In my test environment, the tfstate files in my bucket were around 80KB, and the query executed successfully in 1.4s. However, I haven't been able to test with larger file sizes.

Could you please try the PR branch issue-105 in your environment and let us know how it performs? Your feedback would be greatly appreciated, and I hope this resolves the issue you're experiencing.

Thank you!

keyolk commented 2 months ago

I was able to take states in 90 sec for the 4M states. And also 90 sec for the 256K as well. Seems it is not related to the size of state files, and not directly related to the terraform plugin. Thanks! I'll see more why the latency happens