search

nccgroup / opinel

Python code shared by Scout2 and AWS-Recipes
GNU General Public License v2.0
24 stars 18 forks source link

opinel 1.0x seems to have introduced a 75-second delay. #12

Closed grettir closed 7 years ago

grettir commented 7 years ago

opinel 1.0x seems to have introduced a (roughly) 75-second delay to any anything making calls to the AWS SDK, whether running one of the AWS-recipes Python scripts or running Scout2.

For instance, when running aws_recipes_assume_role.py, there's a 75-second delay before results are returned and the credentials are cached in the ~/.aws/cli/cache directory.

When running aws_recipes_init_sts_session.py, there's a 75-second delay before it prompts for the MFA code.

When running aws_recipes_create_ip_ranges.py it says "Fetching public IP information for the 'xxxxxxx' environment..." and then pauses for 75 seconds before it starts iterating through the various regions and displaying "...in us-east-1: EC2 instances", etc.

After that initial 75-second delay, everything runs as expected. (For instance, there's not an additional 75-second delay for each step of the Scout2 report generation process, only at the very beginning.)

And running the scripts or Scout2 with the --debug option returns no additional information. The scripts are succeeding; they just take 75 seconds longer than expected.

l01cd3v commented 7 years ago

I haven't seen this behavior when testing. I have a feeling this may be due to a long timeout when attempting to read EC2 instance's credentials. If you're not running this code from an EC2 instance, could you try to get the source, comment lines 515-517 in opinel/utils.py, and install the module (sudo python setup.py install) ?

If the delay goes away, I'll probably modify the code to look for EC2 credentials in the last resort. If not, I'll probably need more info such as the OS, version of Python, and how complex your credentials/config files are?

grettir commented 7 years ago

Commenting out lines 515-516 resolved the problem.

The problem occurs with even the most basic of credentials/config files. In fact, starting from scratch...

~ aws configure --profile wubba
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Default region name [None]: us-east-1
Default output format [None]: json
~ aws_recipes_configure_iam.py --profile wubba
Found long-lived credentials for the profile 'wubba'. Do you want to use those when configuring mfa (y/n)?
y
Trying to read the MFA serial number associated with this IAM user...
Your long-lived credentials will now be available via the wubba-nomfa profile.

Results in the following bare-bones config/credentials files:

~/.aws/credentials

[wubba]
aws_access_key_id = AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[wubba-nomfa]
aws_access_key_id = AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
aws_mfa_serial = arn:aws:iam::xxxxxxxxxxxx:mfa/xxxxxxxxxx

~/.aws/config

[profile wubba]
output=json
region=us-east-1

Then, if we run aws_recipes_init_sts_session.py with some surrounding code to calculate the execution time…

~ start_time=`date +%s`; aws_recipes_init_sts_session.py --profile wubba --debug; end_time=`date +%s`; echo execution time was `expr $end_time - $start_time` s.
Enter your MFA code (or 'q' to abort):
xxxxxx
execution time was 85 s.

...it takes 85 seconds, ~10 seconds of which was me fumbling to enter the MFA code once the prompt appeared after ~75 seconds.

This is on macOS Sierra, running the following:

...and up-to-date AWS-recipes scripts.

And I don't think it's a privileges/rights issue, since I've seen the behavior in multiple AWS accounts, including those in which I have read-only privileges and those in which I have admin privileges.

Hopefully that helps.

l01cd3v commented 7 years ago

Thanks, this is what I thought; it means that attempting to fetch the credentials at 169.254.169.254 takes a while. Out of curiosity, does running this command take a while too?

wget http://169.254.169.254/latest/meta-data/iam/security-credentials

l01cd3v commented 7 years ago

And last request: could you uncomment the previous lines and change line 630 in opinel/utils.py to look as follow:

    requests.get('http://169.254.169.254/latest/meta-data/iam/security-credentials', timeout = 1)

I'm guessing the default timeout in your environment was set to 75 seconds. If this works, I'll verify that this timeout value works on EC2 instances and I'll push the code change.

grettir commented 7 years ago

Yep, making that change to line 627 (rather than 630) fixes it.

And it looks like the standard default timeout for HTTP requests (including Python's Requests library) is, indeed, 75 seconds. Good call.

l01cd3v commented 7 years ago

Great, thanks for confirming ! I just pushed a fix and released opinel 1.0.6 that includes this change. I won't force change the minimum version of opinel in Scout2 and AWS-recipes so just make sure you update to the latest version.

l01cd3v commented 7 years ago

Of course I didn't run tests before releasing and it was broken... v1.0.7 should work !