search

cycloidio / terracognita

Reads from existing public and private cloud providers (reverse Terraform) and generates your infrastructure as code on Terraform configuration
MIT License
2.17k stars 163 forks source link

Gracefully handle throttling #47

Open jkallay1 opened 5 years ago

jkallay1 commented 5 years ago

Received the error:

could not import from AWS: Error retrieving EC2 tags for CloudFront Distribution [omitted] Throttling: Rate exceeded

and got a stack trace, with no output going to the output file.

There should be some way to see all of the complete resources that have been fetched so that even with throttling all resources can be fetched over time.

xescugc commented 5 years ago

Sorry for the delay I was on vacations :sun_with_face:

That's a good idea :).

I wanted to do that with fully support by passing a "half" state/hcl and ignoring the data already fetched for example.

The other thing is that I've improved the Throttling in a pending PR which I think I'll split to just add the better support for it directly without waiting for the full PR :smile:

Right now we give a small delay when a Throttling, a handmade retry basically to be able to maybe avoid it in a while haha, but sometimes it still has it after a while of retrying.

rquadling commented 4 years ago

The cause of the retry is not displayed until after all retries. So, I have to wait for 90s until I get the error. Not great.

Also, some errors are not errors (I'm raising a separate bug report for that).