search

jckuester / awsls

A list command for AWS resources
MIT License
838 stars 54 forks source link

Concurrency control? #38

Open iainelder opened 3 years ago

iainelder commented 3 years ago

I felt ambitious today and tried to list 17 accounts in 17 regions.

awsls failed before listing anything.

$ awsls cloudtrail --regions "${regions}" --profiles "${profiles}"

Error: failed to launch provider (/home/isme/.awsls/terraform-provider-aws_v3.31.0_x5): pipe2: too many open files

I'm not really sure what limit I'm hitting here, whether it's imposed by awsls itself or by my operating system.

When I halved the number of regions to 8, the behavior was unpredictable and unstable.

Once it froze for a while before printing:

Finished (killed)

Another time it printed something more esoteric:

panic: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded"

goroutine 254 [running]:
github.com/hashicorp/terraform/plugin.(*GRPCProvider).getSchema(0xc0002a7500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/runner/go/pkg/mod/github.com/hashicorp/terraform@v0.12.28/plugin/grpc_provider.go:80 +0x289
github.com/hashicorp/terraform/plugin.(*GRPCProvider).Configure(0xc0002a7500, 0x0, 0x0, 0x8270f98, 0xc002107060, 0x6e01b40, 0xc00209d770, 0x414494, 0xc003bb9d60, 0x6e013c0)
    /home/runner/go/pkg/mod/github.com/hashicorp/terraform@v0.12.28/plugin/grpc_provider.go:277 +0x6a
github.com/jckuester/terradozer/pkg/provider.TerraformProvider.Configure(0x7fdaa9527390, 0xc0002a7500, 0x2540be400, 0x8270f98, 0xc002107060, 0x6e01b40, 0xc00209d770, 0x0, 0x0)
    /home/runner/go/pkg/mod/github.com/jckuester/terradozer@v0.1.4-0.20210411111420-d43446e05adb/pkg/provider/provider.go:101 +0xbc
github.com/jckuester/awstools-lib/terraform.NewProviderPool.func1(0xc0002e4750, 0xc0003d0aa5, 0x3, 0xc0003d0aaa, 0x6, 0xc0003d0a80, 0x33, 0x2540be400, 0xc000200300, 0xc00005bd90, ...)
    /home/runner/go/pkg/mod/github.com/jckuester/awstools-lib@v0.0.0-20210411200154-5b8a784826d0/terraform/provider_pool.go:90 +0x1233
created by github.com/jckuester/awstools-lib/terraform.NewProviderPool
    /home/runner/go/pkg/mod/github.com/jckuester/awstools-lib@v0.0.0-20210411200154-5b8a784826d0/terraform/provider_pool.go:53 +0x657

I suppose there is a certain stable limit for the number of accounts and regions I can query.

Is there some way to control the concurrency of awsls without splitting the calls into batches?

jckuester commented 3 years ago

Hi @iainelder, I started working on the first issues that you are describing. All provider plugins (17x17=289) are launched in the beginning and lead to too many open files. I'll try to launch/close the provider each time when needed (this is also what Terraform does and the overhead of launch/close is very small): https://github.com/jckuester/awsls/pull/39