search

turbot / steampipe

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
https://steampipe.io
GNU Affero General Public License v3.0
6.79k stars 263 forks source link

Queries fail to complete when trying to retrieve a large number of rows and caching is enabled #4133

Closed cbruno10 closed 1 month ago

cbruno10 commented 6 months ago

Describe the bug When querying an AWS aggregator connection with 10 accounts and regions = ["*"] with select * from aws_sqs_queue, I'd expect 80k+ rows to be returned. Each queue currently invokes 5 hydrate calls as well.

But, it's been stalling at various row counts, e.g., 27k, 48k, with errors like:

2024-02-15 14:29:00.415 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 0 key aws_pipeling_scale_testing_011_aws_sqs_queue_title,queue_arn,akas,region,account_id,fifo_queue,visibility_timeout_seconds,redrive_policy,kms_master_key_id,message_retention_seconds,sqs_managed_sse_enabled,queue_url,fifo_throughput_limit,delay_seconds,max_message_size,deduplication_scope,tags,partition,receive_wait_time_seconds,policy,policy_std,content_based_deduplication,_ctx_-1-0 after 10 retries: Entry not found (aws_pipeling_scale_testing_011-1708006776706)
2024-02-15 14:29:00.417 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_011-1708006776706 (aws_pipeling_scale_testing_011-1708006776706)
2024-02-15 14:29:29.043 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 0 key aws_pipeling_scale_testing_019_aws_sqs_queue_queue_url,fifo_queue,delay_seconds,message_retention_seconds,redrive_policy,deduplication_scope,max_message_size,receive_wait_time_seconds,kms_master_key_id,partition,account_id,title,policy,content_based_deduplication,akas,tags,queue_arn,fifo_throughput_limit,sqs_managed_sse_enabled,visibility_timeout_seconds,policy_std,region,_ctx_-1-0 after 10 retries: Entry not found (aws_pipeling_scale_testing_019-1708006776706)
2024-02-15 14:29:29.044 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_019-1708006776706 (aws_pipeling_scale_testing_019-1708006776706)
2024-02-15 14:29:45.359 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 0 key aws_pipeling_scale_testing_016_aws_sqs_queue_partition,region,fifo_queue,delay_seconds,visibility_timeout_seconds,policy,akas,sqs_managed_sse_enabled,policy_std,redrive_policy,deduplication_scope,fifo_throughput_limit,max_message_size,message_retention_seconds,receive_wait_time_seconds,tags,content_based_deduplication,kms_master_key_id,queue_url,title,account_id,queue_arn,_ctx_-1-0 after 10 retries: Entry not found (aws_pipeling_scale_testing_016-1708006776706)
2024-02-15 14:29:45.359 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_016-1708006776706 (aws_pipeling_scale_testing_016-1708006776706)
2024-02-15 14:29:55.217 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 0 key aws_pipeling_scale_testing_018_aws_sqs_queue_max_message_size,content_based_deduplication,akas,account_id,queue_arn,fifo_throughput_limit,policy_std,kms_master_key_id,tags,partition,title,fifo_queue,message_retention_seconds,queue_url,receive_wait_time_seconds,sqs_managed_sse_enabled,visibility_timeout_seconds,policy,redrive_policy,deduplication_scope,region,delay_seconds,_ctx_-1-0 after 10 retries: Entry not found (aws_pipeling_scale_testing_018-1708006776706)
2024-02-15 14:29:55.218 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_018-1708006776706 (aws_pipeling_scale_testing_018-1708006776706)
2024-02-15 14:30:57.463 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 1 key aws_pipeling_scale_testing_012_aws_sqs_queue_kms_master_key_id,akas,fifo_throughput_limit,max_message_size,receive_wait_time_seconds,deduplication_scope,sqs_managed_sse_enabled,policy,policy_std,partition,queue_url,queue_arn,fifo_queue,delay_seconds,message_retention_seconds,visibility_timeout_seconds,region,account_id,title,redrive_policy,content_based_deduplication,tags,_ctx_-1-1 after 10 retries: Entry not found (aws_pipeling_scale_testing_012-1708006776706)
2024-02-15 14:30:57.463 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_012-1708006776706 (aws_pipeling_scale_testing_012-1708006776706)
2024-02-15 14:30:57.463 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: waitUntilDone - setRequestSubscriber received an error from setRequest aws_pipeling_scale_testing_012-1708006776706: Entry not found (aws_pipeling_scale_testing_012-1708006776706)
2024-02-15 14:31:01.557 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 1 key aws_pipeling_scale_testing_017_aws_sqs_queue_tags,message_retention_seconds,fifo_queue,fifo_throughput_limit,receive_wait_time_seconds,visibility_timeout_seconds,policy_std,content_based_deduplication,kms_master_key_id,queue_arn,region,account_id,queue_url,delay_seconds,max_message_size,sqs_managed_sse_enabled,policy,redrive_policy,deduplication_scope,akas,title,partition,_ctx_-1-1 after 10 retries: Entry not found (aws_pipeling_scale_testing_017-1708006776706)
2024-02-15 14:31:01.557 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_017-1708006776706 (aws_pipeling_scale_testing_017-1708006776706)
2024-02-15 14:31:01.557 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: waitUntilDone - setRequestSubscriber received an error from setRequest aws_pipeling_scale_testing_017-1708006776706: Entry not found (aws_pipeling_scale_testing_017-1708006776706)
2024-02-15 14:31:01.747 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 2 key aws_pipeling_scale_testing_010_aws_sqs_queue_akas,queue_arn,max_message_size,visibility_timeout_seconds,policy,redrive_policy,content_based_deduplication,kms_master_key_id,tags,partition,title,fifo_throughput_limit,delay_seconds,sqs_managed_sse_enabled,deduplication_scope,fifo_queue,message_retention_seconds,account_id,queue_url,receive_wait_time_seconds,policy_std,region,_ctx_-1-2 after 10 retries: Entry not found (aws_pipeling_scale_testing_010-1708006776706)
2024-02-15 14:31:01.747 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_010-1708006776706 (aws_pipeling_scale_testing_010-1708006776706)
2024-02-15 14:31:01.747 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: waitUntilDone - setRequestSubscriber received an error from setRequest aws_pipeling_scale_testing_010-1708006776706: Entry not found (aws_pipeling_scale_testing_010-1708006776706)
2024-02-15 14:31:01.853 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: getRowsSince failed to read page 2 key aws_pipeling_scale_testing_015_aws_sqs_queue_queue_arn,policy_std,region,queue_url,fifo_queue,max_message_size,receive_wait_time_seconds,redrive_policy,content_based_deduplication,akas,partition,fifo_throughput_limit,message_retention_seconds,sqs_managed_sse_enabled,visibility_timeout_seconds,policy,deduplication_scope,title,delay_seconds,kms_master_key_id,tags,account_id,_ctx_-1-2 after 10 retries: Entry not found (aws_pipeling_scale_testing_015-1708006776706)
2024-02-15 14:31:01.853 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: readAndStreamAsync failed to read previous rows from cache: Entry not found publisher aws_pipeling_scale_testing_015-1708006776706 (aws_pipeling_scale_testing_015-1708006776706)
2024-02-15 14:31:01.853 UTC [WARN]  steampipe-plugin-aws.plugin: [WARN]  1708006776706: waitUntilDone - setRequestSubscriber received an error from setRequest aws_pipeling_scale_testing_015-1708006776706: Entry not found (aws_pipeling_scale_testing_015-1708006776706)

However, if caching is disabled, my query always completes OK

Steampipe version (steampipe -v) v0.21.6

To reproduce See above

Expected behavior Query should complete like it does when caching is disabled

Additional context Add any other context about the problem here.

github-actions[bot] commented 4 months ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

github-actions[bot] commented 2 months ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

github-actions[bot] commented 1 month ago

This issue was closed because it has been stalled for 90 days with no activity.