Amazon S3 Task Rework

[m-q-t]

Hi team,

Please find in this PR a complete re-work of the S3 Tasks. The majority of the tasks are new while some older ones have been re-written to support the new changes made.

lib/entities/aws_s3_bucket.rb

• This was modified to validate a stricter regex that only includes the bucket name rather than a URI which contains s3.amazonaws.com. S3 bucket names have strict naming rules in which the regex supports.

lib/tasks/helpers/aws.rb

• New helper for AWS tasks; specifically right now this only has methods that are S3 related. The current methods include returning an S3 client (which verifies whether the keys used to initialize the client are valid). Also a method which extracts the bucket name from a URI which supports both S3 virtual hosted style & path style URI's.

lib/tasks/enrich/aws_s3_bucket.rb

• This task was modified to only confirm whether the bucket actually exists and if the bucket belongs to the API Key which was provided in the task config. The reason for verifying whether the bucket belongs to the API Key will be explained in the next two tasks.

lib/tasks/aws_s3_find_listable_objects.rb

• This task searches an S3 bucket for any 'listable' objects. Please note that just because an object is listable doesn't mean it's readable. However we've had an existing issue for this (severity 4) and in some scenarios it would aid an attacker to see what files necessarily exist on a bucket even if they're not readable.

• The task supports two additional options:

  • bruteforce_found_objects -> If this option is enabled (which it is by default) this task calls start_task at the end with the S3 Bruteforce Objects task passing in the found objects as a wordlist.

  • use_authentication -> This stems from the enrichment task and this is done because buckets can be configured in a way which allows any authenticated S3 user to list objects using the API. By any authenticated user it means literally any AWS user and not necessarily ones that relate to the account which owns the bucket. The task however has an additional check to verify whether the AWS Keys that are provided in the task config actually own the bucket. This is done specifically because this can result in a false positive (in scenarios where the key can read the files due to having permissions) and we're specifically checking whether other authenticated users are able to list the contents of the bucket. If the bucket owns the API key or there are no API keys set in the task config, the task will use an unauthenticated technique to list the contents of the bucket (aka just sending an HTTP request and checking to see if bucket listing is enabled).

lib/tasks/aws_s3_bruteforce_objects.rb

• This tasks searches an S3 bucket for any readable objects. This task works in conjunction with the task above and if a readable object is found an existing issue of Severity 2 is created.

• The task supports two additional options:

  • objects_list -> Pass in a wordlist of objects to check if they are readable. This is where the objects from the aws_s3_find_listable_objects task is passed in. If no wordlist is passed in, the task reads from data/s3_common_objects.list which contains a wordlist of the top 100 common files (curated using different wordlists from SecLists)

• use_authentication - Follows the same principle as the task above. TL;DR - Based on whether API keys are provided, the task will use different techniques (authenticated & unauthenticated) to confirm if the object is readable.

lib/tasks/aws_s3_gather_buckets.rb

• This task uses the API Keys to gather the list of buckets that are owned by the user. Entities are created from these buckets and thus are sent to be enriched.

lib/tasks/aws_s3_put_file.rb

• This task was rewritten to verify whether any authenticated AWS user is able to write an S3 bucket. There are no 'unauthenticated' ways to write to a file to the bucket.

lib/tasks/aws_s3_bruteforce_buckets.rb

• This task was originally called s3_brute and was re-written to reflect its behavior. The task takes in a list of keywords and creates a wordlist by generating permutations using the keywords and common prefixes/suffixes (such as dev, testing, eng, etc.). The wordlist is then iterated upon to check whether the specific bucket exists or not.

• The task accepts the following additional options:

  • additional_permutation_wordlist -> Any additional prefixes/suffixes to use to generate permutations with.
  • additional_keywords -> Any additional keywords to use with permutations to generate bucket names with.

The following workflow has been updated to utilize the new tasks:

• lib/workflows/profile_organization_external_light_active.yml

Finally, the following existing tasks which created S3 Entities were modified to support the new changes: Note: This was super seamless due to the extract_bucket_name_from_uri helper method.

• lib/tasks/search_grayhat_warfare.rb
• lib/tasks/uri_browser_analysis.rb
• lib/tasks/uri_extract_linked_hosts.rb

Best regards, Maxim

[m-q-t]

A few screenshots:

[jcran]

re: breaking hosted service - check the hosted-only workflow files for any mention of these tasks. That the only interaction i'm aware of that could cause issues.

[m-q-t]

@shpendk

1. I see you deleted the task search_wayback_machine.rb. Is that on purpose and if so, why?

Sorry that was an accident on my end. I accidentally started working on another set of tasks without realizing I was in the wrong branch. I'll add back the original search_waybackm_macine.rb task so it doesn't actually delete anything. Thanks for catching that.

2. Why did you not include aws_s3_bruteforce_objects in the list of tasks for AwsS3Bucket entities in the workflow file?

The reason for this is because the aws_s3_find_listable_objects task calls the aws_s3_bruteforce_objects task at the end if any listable objects are found; passing in those objects. I figured this would probably be a better idea than having it call an additional time from the workflow using the wordlist of the "100 Common Objects". If you feel that it should also call it an additional time, please let me know and I'll change it.

re: breaking hosted service - check the hosted-only workflow files for any mention of these tasks. That the only interaction i'm aware of that could cause issues.

I'll go through the hosted-only workflows and update them as well however they are not in this repository so that will be a separate PR.

Thanks for reviewing.

[shpendk]

@m-q-t regarding running aws_s3_bruteforce_objects, yes lets run it for every AWS s3 bucket. If listable_objects doesn't find anything, we still want to bruteforce files.

I think after this final change we're good to merge. Really nice work man.

[m-q-t]

Appreciate the kind words @shpendk

The workflow has been updated.