search

willbengtson / trailblazer-aws

Blazing CloudTrail since 2018
134 stars 23 forks source link

Automate parameter generation #5

Open 0xdabbad00 opened 5 years ago

0xdabbad00 commented 5 years ago

This does a pretty good of automatically generating parameters for the functions. It still needs some work (the code is ugly in places, and does not work for all of the APIs) so I'll continue working on this PR, but I wanted to make these changes public so you can see the path I'm going down.

This currently gets all the way to sagemaker.render_ui_template before it throws an exception, so it almost is able to run through everything. I still need to do a full run and collect all of the output, but it looks like this should call something like 60% of the functions, and hopefully most of the other errors can be quickly fixed.

The general concept for this code is explained in #2 . I look at the shape of the function to identify the required parameters, then look at the shape of those and try to automatically create strings that match the required patterns by using the exrex library. There are a number of special cases I needed to fill due mostly to limitations in the specification botocore gives for the actual shape needed for the parameters in the calls.

0xdabbad00 commented 5 years ago

Currently 4767 functions are attempted to be called. I think botocore currently has 5299 functions (found by running cat botocore/botocore/data/*/*/service-*.json | jq -cr '. as $root | .operations|keys[]|[$root.metadata.serviceId, .] | @tsv' | sort | uniq | wc). The code skips 34 services due to them not existing in us-east-1, or needing to be added to a beta as no regions exist for it, and s3control seems broken, so I assume that accounts for the skipped functions.

The skipped services are:

Skipping amplify - No regions exist for this service
Skipping appmesh - No regions exist for this service
Skipping cloudsearchdomain - No regions exist for this service
Skipping comprehendmedical - No regions exist for this service
Skipping connect - No regions exist for this service
Skipping datasync - No regions exist for this service
Skipping devicefarm - Only available in [u'us-west-2']
Skipping discovery - Only available in [u'us-west-2']
Skipping dlm - No regions exist for this service
Skipping eks - No regions exist for this service
Skipping fsx - No regions exist for this service
Skipping globalaccelerator - No regions exist for this service
Skipping iot-jobs-data - No regions exist for this service
Skipping iot1click-devices - No regions exist for this service
Skipping iot1click-projects - No regions exist for this service
Skipping kafka - No regions exist for this service
Skipping license-manager - No regions exist for this service
Skipping macie - No regions exist for this service
Skipping mediaconnect - No regions exist for this service
Skipping mediastore-data - No regions exist for this service
Skipping mgh - Only available in [u'us-west-2']
Skipping mobile - No regions exist for this service
Skipping mq - No regions exist for this service
Skipping pi - No regions exist for this service
Skipping quicksight - No regions exist for this service
Skipping ram - No regions exist for this service
Skipping rds-data - No regions exist for this service
Skipping robomaker - No regions exist for this service
Skipping route53resolver - No regions exist for this service
Skipping s3control - End-points do not seem to be working
Skipping securityhub - No regions exist for this service
Skipping signer - No regions exist for this service
Skipping sms-voice - No regions exist for this service
Skipping transcribe - No regions exist for this service

444 functions could not be found in the service.json file. This is due to the capitalization rules changing between services and functions, such as my code looking for GetIpSet, when it should be looking for GetIPSet.

There are 32 "KeyError Exception" which I think again might be capitalization issues, ex. error: KeyError Exception: chime.reset_personal_pin - u'accountId'

So I think I can fix those 444 + 32.

Then there are 870 remaining errors. The counts for these exceptions are:

   1 AWS.SimpleQueueService.NonExistentQueue
   1 ApplicationNameRequiredException
   1 AutomationDefinitionNotFoundException
   1 DeploymentGroupNameRequiredException
   1 GitHubAccountTokenNameRequiredException
   1 InternalError
   1 InvalidAMIID.Malformed
   1 InvalidCustomerGatewayId.Malformed
   1 InvalidDhcpOptionsId.Malformed
   1 InvalidDocument
   1 InvalidInternetGatewayId.Malformed
   1 InvalidQueryExpression
   1 InvalidRequest
   1 InvalidResourceException
   1 InvalidRouteTableId.Malformed
   1 NoSuchChange
   1 RepositoryNameRequiredException
   1 UnknownResourceFault
   2 404
   2 InvalidAuthenticationTokenException
   2 InvalidCiphertextException
   2 InvalidNetworkAclId.Malformed
   2 InvalidParameterCombination
   2 InvalidRoleException
   2 InvalidVpcId.Malformed
   2 InvalidVpcPeeringConnectionId.Malformed
   2 MalformedManifestException
   2 NoSuchQueryLoggingConfig
   2 SubscriptionRequiredException
   2 UnrecognizedClientException
   3 403
   3 InvalidARNFault
   3 InvalidToken
   3 NoSuchDelegationSet
   3 NoSuchTrafficPolicyInstance
   4 BadRequestException
   4 InvalidCommentIdException
   4 InvalidID
   5 InvalidDeploymentIdException
   6 DeploymentIdRequiredException
   6 InvalidInstanceID.Malformed
   7 InternalFailure
   7 NoSuchTrafficPolicy
   8 NoSuchHealthCheck
   8 NoSuchHostedZone
   9 AuthorizationFailure
   9 BadRequest
   9 ResourceNotFoundException
  10 AuthorizationError
  10 InvalidArgumentException
  13 InvalidArn
  14 InvalidParameterException
  17 InvalidAddress
  17 ServiceException
  18 InvalidInputException
  20 InvalidParameter
  20 InvalidParameterValueException
  25 InvalidParameterValue
  25 InvalidRequestException
  30 UnauthorizedResourceAccessException
  30 ValidationError
  32 NotFoundException
  51 MissingParameter
  60 InvalidArnException
  64 WAFNonexistentItemException
  83 NoSuchBucket
 186 ValidationException

I'm guessing at best half of those can be resolved, so we'll be left with somewhere under 800 functions that can't be resolved, as they are due to things like needing real, currently existing ARNs in the account, to reference.

0xdabbad00 commented 5 years ago

Error types were found by cat'ing the output through | grep "error: " | grep -v "error: Could not find" | grep -v "KeyError Exception" | sed 's/error: An error occurred (//' | sed 's/).*//' | sort | uniq -c | sort -n

0xdabbad00 commented 5 years ago

Currently at 4767 functions called, 243 problems with not finding the function that I should be able to resolve, and 821 remaining errors (some of which are related to not finding the function though). My goal is by tomorrow evening to fix all of the problems with not finding the functions. These look like:

error: Could not find UpdateWebAcl in service json botocore/botocore/data/waf-regional/2016-11-28/service-2.json

due to it needing to look for UpdateWebACL and this subsequently results in:

error: An error occurred (ValidationException) when calling the UpdateWebACL operation: 2 validation errors detected: Value null at 'changeToken' failed to satisfy constraint: Member must not be null; Value null at 'webACLId' failed to satisfy constraint: Member must not be null
0xdabbad00 commented 5 years ago

The only remaining errors for Could not find a function name are:

error: Could not find GenerateDBAuthToken in service json botocore/botocore/data/rds/2014-10-31/service-2.json
error: Could not find Copy in service json botocore/botocore/data/s3/2006-03-01/service-2.json
error: Could not find DownloadFile in service json botocore/botocore/data/s3/2006-03-01/service-2.json
error: Could not find DownloadFileobj in service json botocore/botocore/data/s3/2006-03-01/service-2.json
error: Could not find UploadFile in service json botocore/botocore/data/s3/2006-03-01/service-2.json
error: Could not find UploadFileobj in service json botocore/botocore/data/s3/2006-03-01/service-2.json

The GenerateDBAuthToken doesn't seem to exist in RDS, so I don't know where that is being found, but the call actually seems to work. For the S3 issues, I think there is some sort of renaming happening. Example:

error: Could not find DownloadFileobj in service json botocore/botocore/data/s3/2006-03-01/service-2.json
Calling s3.download_fileobj with params {} in us-east-1
error: An error occurred (403) when calling the HeadObject operation: Forbidden

Notice, the error says that HeadObject failed, when the call was for DownloadFileobj.

There are 822 errors in total. I'm going to start looking at what actually showed up in CloudTrail from all this.

0xdabbad00 commented 5 years ago

I reviewed what was recorded in CloudTrail, and looks like 2687 events of 4767 calls made (56%). I need to dig into what was recorded and what wasn't.

0xdabbad00 commented 5 years ago

:rage: I'm looking at some of the differences between what was called and what was recorded and it doesn't look like there are any indications that I can use to resolve these issues. For example, for the ec2 service, here are the calls that were not recorded:

cancelimporttask
copysnapshot
createflowlogs
deleteflowlogs
describeflowlogs
describeimportimagetasks
describeimportsnapshottasks
describemovingaddresses
importimage
importsnapshot
moveaddresstovpc
restoreaddresstoclassic

Looking at the logs for create_flow_logs and with a code change to show the error, this is all it shows:

Calling ec2.create_flow_logs with params {u'ResourceType': u'VPC', u'ResourceIds': [u'xiqssmmtsgmkjpdxikrqdrh'], u'TrafficType': u'ACCEPT'} in us-east-1
error: An error occurred (UnauthorizedOperation) when calling the CreateFlowLogs operation: You are not authorized to perform this operation.

There are lots of other UnauthorizedOperation errors that are recorded, so that is not the problem. It could be that the unauthorized operation is only recorded if the ResourceIds are valid.

Given this situation, I believe this is as far as this concept can be taken unfortunately. :( With work, maybe a few hundred more functions could be recorded by fixing the ValidationException and similar, but I think there are many functions, such as CreateFlowLogs, that will only be recorded if they are given real resources to work on.