Closed keyolk closed 2 months ago
Hi @keyolk,
I apologize for the delayed response.
Here are a few key points for you to review:
.tf
file resources stored in an S3 bucket, and I got the expected results using the following configuration in the terraform.spc
file:connection "tf_bucket" {
plugin = "terraform"
configuration_file_paths = [
"s3::https://test-delete90.s3.us-east-2.amazonaws.com//**/*.tf?aws_profile=default"
]
}
Query Result:
> select * from tf_bucket.terraform_resource
+---------------------+----------------------+--------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
| name | type | mode | address | arguments >
+---------------------+----------------------+--------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
| example | tls_private_key | <null> | tls_private_key.example | {"algorithm":"RSA"} >
| | | | | >
| | | | | >
| named_test_resource | aws_acm_certificate | <null> | aws_acm_certificate.named_test_resource | {"certificate_body":"${tls_self_signed_cert.example.cert_pem}","private_key":"${tls_private_key.example.private_key_pem}","tags":{"name":"turbot-test-20200125-create-update"}} >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| example | tls_self_signed_cert | <null> | tls_self_signed_cert.example | {"allowed_uses":["key_encipherment","digital_signature","server_auth"],"private_key_pem":"${tls_private_key.example.private_key_pem}","subject":{"common_name":"turbot.com","organization":"Turbot HQ Pvt. Ltd."},"valid>
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
| | | | | >
+---------------------+----------------------+--------+-----------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
I haven’t yet tried it with a tfstate
file in an S3 bucket. Could you please give it a try and let us know if you encounter any further issues?
Thank you!
@ParthaI Hi! basically, it works. But extreamely slow and most of case, it returns error like below
steampipe query "select name, type, mode, address, arguments, attributes, path from tfstate.terraform_resource where mode = 'managed' limit 10" --output json
Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000)
Any idea?
And seems it just follow AWS_REGION env even I set aws_region in its file_path which make much confusing
Hello @keyolk, thank you for testing these things out!
steampipe query "select name, type, mode, address, arguments, attributes, path from tfstate.terraform_resource where mode = 'managed' limit 10" --output json Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000)
It seems the error occurs while parsing the content of a specific state file. It would be great if you could figure out which state file content is failing.
terraform.spc
file?The information you provide will greatly assist me in replicating the issue you're encountering.
Thank you!
@ParthaI it took about 70sec on my m1 macbook
connection "tfstate" {
plugin = "terraform"
state_file_paths = [
"s3::https://<mybucket>.s3.<my_region>.amazonaws.com//<my_object_prefix>/**/terraform.tfstate?aws_profile=<my_profile>&aws_region=<my_region>"
]
}
$ time steampipe query "select name, type, mode, address, arguments, attributes, path from tfstate.terraform_resource where mode = 'managed' limit 10" --output json
...
real 1m14.606s
user 0m4.186s
sys 0m4.547s
$ steampipe --version
Steampipe v0.23.5
$ file $(which steampipe)
/opt/homebrew/bin/steampipe: Mach-O 64-bit executable arm64
$ steampipe plugin list | grep terraform
| hub.steampipe.io/plugins/turbot/terraform@latest | 0.12.0 | tfstate |
$ du -chs state.json
256K state.json
256K total
for single file like the above it can get result but takes too much time about 80 sec
$ find . | grep tfstate | wc -l
45
$ du -chs *
...
4.2M total
And the above case it ahs 45 tfstate with size of 4M it got error
Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000)
Also some error logs from plugin.log
2024-09-10 08:26:05.015 UTC [WARN] initializePlugin failed: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x140005f36c0)
2024-09-10 08:26:05.019 UTC [WARN] PluginManager Get failed for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000d10c00)
2024-09-10 08:27:11.809 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956798220: readAndStreamAsync failed to read previous rows from cache: context canceled publisher tfstate-1725956798220 (tfstate-1725956798220)
2024-09-10 08:27:11.809 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956798220: QueryCache AbortSet - aborting request with error set request has no subscribers (0 subscribers) (tfstate-1725956798220)
2024-09-10 08:27:11.809 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956798220: queryData.streamRows returned error: context canceled
2024-09-10 08:27:11.809 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956798220: executeForConnection tfstate returned error context canceled, writing to CHAN
2024-09-10 08:28:19.991 UTC [WARN] failed to set connection config for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference
2024-09-10 08:28:19.991 UTC [WARN] initializePlugin failed: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000bb6100)
2024-09-10 08:28:19.995 UTC [WARN] PluginManager Get failed for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000ba6108)
2024-09-10 08:28:58.695 UTC [WARN] failed to set connection config for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference
2024-09-10 08:28:58.695 UTC [WARN] initializePlugin failed: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14003566cc0)
2024-09-10 08:28:58.697 UTC [WARN] PluginManager Get failed for hub.steampipe.io/plugins/turbot/aws@latest: setAllConnectionConfigs failed: runtime error: invalid memory address or nil pointer dereference (0x14000f240a8)
2024-09-10 08:29:31.244 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956938334: readAndStreamAsync failed to read previous rows from cache: context canceled publisher tfstate-1725956938334 (tfstate-1725956938334)
2024-09-10 08:29:31.244 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956938334: QueryCache AbortSet - aborting request with error set request has no subscribers (0 subscribers) (tfstate-1725956938334)
2024-09-10 08:29:31.244 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956938334: queryData.streamRows returned error: context canceled
2024-09-10 08:29:31.244 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725956938334: executeForConnection tfstate returned error context canceled, writing to CHAN
2024-09-10 08:30:18.276 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] failed to delete the temp directory /var/folders/7g/1dw9yqgj4mzcrfgcsqj9vs6w0000gq/T/steampipe-plugin-terraform: unlinkat /var/folders/7g/1dw9yqgj4mzcrfgcsqj9vs6w0000gq/T/steampipe-plugin-terraform/tfstate/2024-09-10T08:30:13Z/aws/chat-dev/apne1: directory not empty
2024-09-10 08:31:33.674 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.674 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: streamRows execution has failed: tfstate-1725957057522 - calling queryCache.AbortSet (tfstate: runtime error: index out of range [0] with length 0)
2024-09-10 08:31:33.674 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryCache AbortSet - aborting request with error tfstate: runtime error: index out of range [0] with length 0 (1 subscriber) (tfstate-1725957057522)
2024-09-10 08:31:33.674 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: queryData.streamRows returned error: tfstate: runtime error: index out of range [0] with length 0
2024-09-10 08:31:33.674 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: executeForConnection tfstate returned error tfstate: runtime error: index out of range [0] with length 0, writing to CHAN
2024-09-10 08:31:33.674 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: error channel received tfstate: runtime error: index out of range [0] with length 0
2024-09-10 08:31:33.691 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.700 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.708 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.729 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.729 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: readAndStreamAsync failed to read previous rows from cache: tfstate: runtime error: index out of range [0] with length 0 publisher tfstate-1725957057522 (tfstate-1725957057522)
2024-09-10 08:31:33.772 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.845 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:33.878 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:34.003 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:34.051 UTC [WARN] steampipe-plugin-terraform.plugin: [WARN] 1725957057522: QueryData StreamError runtime error: index out of range [0] with length 0 (tfstate-1725957057522)
2024-09-10 08:31:38.355 UTC [WARN] refreshConnections failed with err
failed to create transaction to perform update query: failed to connect to `user=root database=steampipe`: 127.0.0.1:9193 (127.0.0.1): server error: FATAL: the database system is shutting down (SQLSTATE 57P03)
2024-09-10 08:31:38.391 UTC [WARN] setAllConnectionStateToError failed to acquire connection from pool: failed to connect to `user=root database=steampipe`: 127.0.0.1:9193 (127.0.0.1): tls error: read tcp 127.0.0.1:51540->127.0.0.1:9193: read: connection reset by peer
2024-09-10 08:31:38.392 UTC [WARN] failed to send error notification, error
And about the env AWS_REGION
it take the env first while I set the aws_region in the terraform.spc
Lets say I have
"s3::https://<mybucket>.s3.<my_region>.amazonaws.com//<my_object_prefix>/**/terraform.tfstate?aws_profile=<my_profile>&aws_region=us-east-1"
and AWS_REGION=us-west-2
it would take bucket from us-west-2
And also there are no any log from user side, if I set the wrong path or region or bucket, ... May better make user to check if the given spc config is valid when it take 0 bucket or object with the connection
Hello @keyolk,
Thank you for providing such detailed information!
I was able to replicate the error: Error: tfstate: runtime error: index out of range [0] with length 0 (SQLSTATE HV000)
. To address this, I’ve submitted a PR that I believe should resolve the issue.
In my test environment, the tfstate
files in my bucket were around 80KB
, and the query executed successfully in 1.4s
. However, I haven't been able to test with larger file sizes.
Could you please try the PR branch issue-105
in your environment and let us know how it performs? Your feedback would be greatly appreciated, and I hope this resolves the issue you're experiencing.
Thank you!
I was able to take states in 90 sec for the 4M states. And also 90 sec for the 256K as well. Seems it is not related to the size of state files, and not directly related to the terraform plugin. Thanks! I'll see more why the latency happens
I am testing steampipe against tfstates. I am managing multiple tfstate files in one s3 bucket. After testing, it seems that formats such as "$bucketname/*/.tfstate" are not supported. In practice, we manage a lot of tfstates, and it would be very helpful if recursive search is possible.
Found some test cases which looks related to this https://github.com/turbot/steampipe-plugin-sdk/blob/develop/plugin/get_source_files_test.go#L165 But it never work as I expected