AWS ELBV2 Table empty

[ashwinnirantar]

Describe the Bug

aws_elbv2_load_balancers this table is empty on our end.

Expected Behavior

This table should be populated with appropriate data on elbv2's.

CloudQuery Version

cloud-query linux binary version: v0.15.9

cloudquery plugin version: v0.6.1 (includes cq provider)

[roneli]

Hi @ashwinnirantar, thank you for reporting the issue.

Can you please give me more details on your end if possible, such config.hcl / execution log ?

If possible, try to run the your provider only with the elbv2 resource, see config example below:

cloudquery {
  connection {
    dsn = "host=localhost user=postgres password=pass database=postgres port=5432 sslmode=disable"
  }
  provider "aws" {
    source  = "cloudquery"
    version = "v0.6.1"
  }
}

provider "aws" {
  configuration {}
  resources            = ["elbv2.load_balancers"]
  enable_partial_fetch = true
}

[ashwinnirantar]

Hi @ashwinnirantar, thank you for reporting the issue.

Can you please give me more details on your end if possible, such config.hcl / execution log ?

If possible, try to run the your provider only with the elbv2 resource, see config example below:

cloudquery {
  connection {
    dsn = "host=localhost user=postgres password=pass database=postgres port=5432 sslmode=disable"
  }
  provider "aws" {
    source  = "cloudquery"
    version = "v0.6.1"
  }
}

provider "aws" {
  configuration {}
  resources            = ["elbv2.load_balancers"]
  enable_partial_fetch = true
}

Hi,

Our config.hcl is very similar to the one you provided, except for just elbv2.load_balancers Not sure if I can run that locally, but I tried to get logs for you.

Maybe this will help. Following is a screenshot of logs we get in Datadog. This is after filtering for the table aws_elbv2_load_balancers

Specifically, operation error Elastic Load Balancing v2: DescribeLoadBalancers, https response error StatusCode: 0, RequestID: , canceled, context canceled This is repeated every time for this resource.

I am not sure if you still need our config.hcl. But I can provide you if needed.

[roneli]

Hi @ashwinnirantar sorry for the late reply, thanks for the info.

Seems like the context got cancelled for some reason from a prior error.

is this flag enable_partial_fetch = true configured in your hcl?. If this flag isn't enabled, if any resource fails with a critical error the provider stops the fetch and context is cancelled.

To eliminate any possibilities, can you run the fetch with just the elbv2 resource in the resource list?

Feel free to hop on our discord and ping me, will be more than happy to help from there as well.

[ashwinnirantar]

Thanks for your reply, let me try that..Oh discord is perfect, let me ping you there once I try this solution out..

[roneli]

Hi @ashwinnirantar, hope you were able to resolve your issue.

If not please try the latest version v0.8.2 we found an issue that might have caused the deletion of your data in aws_elbv2_load_balancers.

As always we are also available on discord if you require further assistance.

[ashwinnirantar]

Hi @roneli, I tried with enable_partial_fetch set to be true, but I am still getting the same empty table. We can try upgrading the version. Let me get back to you on that.

[roneli]

Hi @ashwinnirantar, did the upgrade solve the issue? I replied to another issue #414 are they related?

[ashwinnirantar]

Hi @ashwinnirantar, did the upgrade solve the issue? I replied to another issue #414 are they related?

Hi yes, this issue is fixed. We are able to see data in the table, but like I mentioned in the other table it is just data about network load balancers (application load balancers missing)

[ashwinnirantar]

I can confirm that enable_partial_fetch configuration worked as the solution. This was not enough since a further configuration for max_retries had to be modified to get complete data. For our use case, it had to be changed to 10.