AWS performance is frustrating

[pburkholder]

As the compliance lead for cloud.gov, I'd like to use InSpec to validate that resources we're creating in AWS on behalf of our customers are correctly configured, but I may need to use a different tool given current performance.

Describe the problem

I've written an example control that evaluates all the IAM policies that our broker creates whenprovisioning S3 buckets.

Evaluating ~622 policies takes 11 minutes. I know there are a lot API calls, but that seems like a long time.

So:

• Are there better ways to structure tests so they run faster? And are they documented and I just didn't find them?
• Is this the sort of report you need to help tackle inspec/inspec#5003
• Is InSpec just not the right tool for this particular job?

Thanks!

Peter

[james-stocks]

One possible slow point is fetching all the policies here then subsequently fetching them one at time here. If you have all the policy data in a hash from line 7, possibly you can have InSpec assert on your hash instead of going back to AWS. You could also describe aws_iam_policies.where and have the where clause exclude anything that is compliant, then test its('entries') { should be_empty } i.e. anything left in the hash is non-compliant.

If it's the case that the AWS resources are simply taking a long time, the project to discuss that in is https://github.com/inspec/inspec-aws There's also https://community-slack.chef.io/ which has an #inspec-aws channel that gets lots of eyes onto your code.

[tbugfinder]

Hi @pburkholder I'd be interested in replaying that. Not sure if API throttling would kick in. Do you have terraform code or cloudformation template available to easily deploy the test case?

[pburkholder]

Thanks for the feedback. I have to fetch the policies one-at-a-time because aws_iam_policies.where(policy_name: /#{s3_iam_pattern}/) doesn't fetch the actual policies, just the following:

[:policy_name, :policy_id, :arn, :path, :default_version_id, :attachment_count, :permissions_boundary_usage_count, :is_attachable, :create_date, :update_date]

But I'll try the Slack and other resources you suggested.

Meanwhile, can you move this issue to the inspec-aws repository? I forgot that was a separate thing. Thanks

[aaronlippold]

Hi Peter

Happy to help if I can

Thanks

On Tue, Sep 29, 2020, 5:08 PM Peter Burkholder (@pburkholder) < notifications@github.com> wrote:

Thanks for the feedback. I have to fetch the policies one-at-a-time because aws_iam_policies.where(policy_name: /#{s3_iam_pattern}/) doesn't fetch the actual policies, just the following:

[:policy_name, :policy_id, :arn, :path, :default_version_id, :attachment_count, :permissions_boundary_usage_count, :is_attachable, :create_date, :update_date]

But I'll try the Slack and other resources you suggested.

Meanwhile, can you move this issue to the inspec-aws repository? I forgot that was a separate thing. Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/inspec/inspec/issues/5267#issuecomment-700990567, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALK42A4ACTV3KOIJWWJTTDSIJEEZANCNFSM4R56L27A .

[tohch4]

So it sounds like a few things are going on here, @pburkholder.

1. Do we need the ability to pull with inspec-aws the full policy object and not just FilterTable results re the previous comment?

Thanks for the feedback. I have to fetch the policies one-at-a-time because aws_iam_policies.where(policy_name: /#{s3_iam_pattern}/) doesn't fetch the actual policies

2. Does the performance hit here relate to train and the inherent lack of concurrency that is not implemented in inspec core, specifically inspec/inspec#3010? It sounds like a lot of API calls being done sequentially and in order, whether or not concurrent calls for different resources is possible?