Cloud providers like Azure, Google, and AWS have changed the way organizations build out their infrastructure. By nature, the cloud tends to be more "open" due to the fact that it is operating on shared hardware owned by someone else (the cloud provider). This can present risks, though. One of the risks that has been widely publicized is that of overly permissive S3 buckets in AWS. There are plenty of news stories about data being leaked via an S3 bucket that was publicly shared unintentionally. These misconfigurations can lead to lawsuits, bad press, and other negative outcomes for businesses. There is a need for more detailed baselines for S3 buckets in order to minimize sensitive data leakage.
The S3 Baseline Project aims to fix that problem of misconfigured S3 buckets leaking sensitive data by providing actionable baselines which security professionals and cloud administrators can use to secure their S3 buckets. The goals of this project are:
Organizations of all sizes would be able to benefit from having detailed baselines like those proposed above, in order to assist them in securing S3 buckets. Smaller organizations with less in-house cloud expertise would likely benefit most from these baselines, though. Having a proven baseline to guide the setup and configuration of S3 buckets would make it easier for organizations of any size to safely utilize cloud resources like S3 buckets.
| Risk name (value) | Impact | Likelihood | Description |
|---|---|---|---|
| Legality of Scanning S3 Buckets (63) | 9 | 7 | Scanning S3 buckets as a part of AWS infrstructure may create an issue in legality. We may only be able to go so far with testing on our sandbox buckets without losing the ability to do so. |
| Creating Realisitc AWS Infrstructure (35) | 7 | 5 | Creating a realistic infrastructure is needed in order to properly test our guidlines. If we are unable to create this it may lead to gaps in guidlines and recommendations. |
| Scripting Limitaitons in AWS (30) | 6 | 5 | When automating the implemntation we may run into issues with how much can be done. It may not be possible to fully automate through AWS CLI. |
| Limitations of Pen Testing (24) | 6 | 4 | We may not have the necessary tools or knowledge to fully pen test the sandbox buckets creating gaps in guidlines. |
| Data Acquisition (18) | 9 | 2 | When creating S3 buckets we must fill them with data in order to test. We may be unable to find usefull data that has a mix of both sensetive test data and benign data. |
| Resource | Dr. Hale needed? | Investigating Team member | Description |
|---|---|---|---|
| Sample AWS Account | No | Matt | Need to spin up a test AWS account, complete with S3 buckets. |
| Pentesting Tool | No | Grace | A pentesting solution will be needed to detect vulnerabilities in publicly accessible S3 buckets. |
| IaC Scripting Knowledge | Maybe | Mark | Some Infrastructure as Code (IaC) knowledge will be needed to help create the AWS configs. This may be a book, tutorials, or other training resources to help sharpen the team's knowledge of IaC. |
| Baseline Storage/Repository | No | Matt | There needs to be a working directory for the team to store baseline drafts and collaborate. This will likely be GitHub, but could be Google Docs, etc. |
| DLP Tool | No | Grace | In order to identify sensitive data types shared publicly, a DLP (Data Loss Prevention) tool may help classify and detect data. |
In order to get a sense of what work had already been done in the S3 bucket space a thorough literature review was started and is in progress. The literature review as it pertains to this project is broken up into two pieces:
A cross-section of research papers related to S3 bucket security are listed in the table below.
| Article | Date Published | Description/Summary |
|---|---|---|
| There's a Hole in that Bucket!: A Large-scale Analysis of Misconfigured S3 Buckets | 2018 | Introduction: Cloud storage services, such as Amazon S3 (Simple Storage Service), offer cost-effectiveness and ease of use, appealing to hobbyists and businesses alike. The proliferation of cloud storage, accessed through user-friendly APIs, is reducing barriers for developers by eliminating the need for complex in-house storage setups. Amazon S3 provides a reliable and scalable infrastructure, accessible via web console or programmatically, facilitating seamless integration into applications. Custom domain referencing is enabled by setting up CNAME resource records. Methods: This study investigates the prevalence and impact of misconfigurations in Amazon S3 buckets. We automatically identify and validate publicly listable buckets, employing various discovery methods. Candidate bucket names are filtered and verified for public readability or writability. They also examine publicly accessible websites referencing writable buckets to assess the potential for resource infection attacks. Results: The tool, implemented in Python, effectively analyzes buckets in parallel. Using a sample set size of k = 30, they determine bucket readability. While the exposure of ACL permissions to anonymous users is identified, they emphasize that this is not inherently a security breach. Analysis reveals 3,415 buckets with writable ACLs for anonymous users and 3,446 buckets with writable ACLs for authenticated users. Discussion: The study prompts ethical considerations regarding large-scale scans and underscores the necessity of precautions to mitigate potential risks. The study emphasizes that the experiments focused solely on identifying misconfigurations without accessing user data. The methodology relies on HTTP requests and file extension analysis to assess sensitive exposure. Conclusion: This research highlights security implications associated with Amazon S3 usage, aiming to raise awareness and caution users about real-world security risks. The automated tool facilitates the discovery and verification of public S3 buckets, revealing significant proportions vulnerable to unauthorized access. Over 200 readable buckets were found to leak sensitive data. |
| Swinhoe, D. What are Amazon Zelkova and Tiros? AWS looks to reduce S3 configuration errors | 2018 | Introduction: According to Skyhigh Networks, a notable percentage of Amazon S3 buckets have unrestricted public access, while a significant portion remains unencrypted. To address AWS S3 configuration errors, Amazon is developing two new tools – Zelkova and Tiros – aimed at enhancing visibility into data and resource access. Despite previous security initiatives by AWS, organizations such as Verizon, Booz Allen Hamilton, and others have experienced data exposure due to configuration errors. Methods: Amazon's efforts to improve AWS security include the launch of Macie, a machine learning tool for sensitive data discovery, and features like default encryption and permission checks for S3. Nevertheless, challenges persist due to a lack of user education, complex deployments, and the rapid expansion of cloud services, often deployed without the oversight of security teams. Results: AWS's new tools aim to mitigate human error and minimize the risk of data leaks by providing enhanced security verification capabilities before infrastructure changes are implemented. However, while these tools offer benefits, there are also potential downsides, as highlighted by some experts. Discussion: The introduction of Zelkova and Tiros reflects AWS's commitment to bolstering security measures. However, the effectiveness of these tools may depend on user education, the complexity of deployments, and the evolving nature of cloud environments. Balancing the benefits and potential drawbacks of these tools is crucial for organizations aiming to enhance their AWS security posture. Conclusion: As AWS continues to innovate in cloud security, the introduction of Zelkova and Tiros signifies a proactive approach to addressing configuration errors and minimizing data leakage risks. While these tools offer promising solutions, their successful implementation will require careful consideration of various factors, including user education and the dynamic nature of cloud environments. |
| Cloud based automated encryption approach to prevent S3 bucket leakage using AWS Lambda | 2022 | Introduction: The cloud, composed of hardware, software, databases, and associated operations, offers scalable and cost-effective computing services. Utilized widely in remote workplaces, distributed storage services leverage APIs to streamline interactions between system administrators and developers. AWS cloud services, along with other major providers like Google Cloud and Microsoft Azure, offer various benefits, including data compliance, scalability, and cost-effectiveness. Objectives: This report aims to explore methods for enhancing data privacy in cloud platforms, particularly focusing on protecting data within AWS S3 buckets through automated encryption. Methods: The report discusses the challenges of data leakage in private S3 buckets, highlighting the vulnerability of bucket names in revealing sensitive information. A methodology for restricting the flow of sensitive data from AWS S3 buckets is presented, utilizing tools like S3Scanner and AWS CloudFormation for secure bucket configuration. Results: Implementation and evaluation of the proposed methodology demonstrate the effectiveness of automated encryption for S3 buckets. A template designed for CloudFormation streamlines resource provisioning and administration, embedding AWS Lambda encryption functions and IAM roles for enhanced security. Conclusion: Automated encryption techniques, such as those demonstrated in this report, offer a robust solution for protecting data stored in AWS S3 buckets. Leveraging AWS CloudFormation and Lambda can significantly enhance cloud security and privacy, ensuring timely action against potential threats. As security and privacy are intertwined, adopting such automated approaches is crucial for safeguarding sensitive information in cloud environments. |
| Security best practices for Amazon S3 | NA | Introduction: The article "Security Best Practices for Amazon S3" by AWS offers detailed guidance on securing data stored in Amazon S3 (Simple Storage Service). It focues on the importance of implementing robust security measures to protect sensitive information from unauthorized access and data breaches. Key Points: Access Control: The article emphasizes the significance of properly configuring access control settings using IAM policies, bucket policies, and Access Control Lists (ACLs) to restrict unauthorized access to S3 buckets and objects. Encryption: It discusses various options for server-side encryption, including SSE-S3, SSE-KMS, and SSE-C, to ensure the confidentiality and integrity of data stored in S3. Monitoring and Logging: Proper monitoring and logging are essential for detecting and responding to security incidents. The article explains how to enable logging and set up notifications for S3 events using Amazon CloudWatch and AWS CloudTrail. Amazon S3 Block Public Access: To prevent accidental exposure of data to the public internet, the article recommends enabling Amazon S3 Block Public Access settings at the account level. Data Classification and Access Management: The importance of classifying data based on its sensitivity and implementing appropriate access controls is emphasized to ensure compliance and mitigate security risks. Conclusion: Overall, the article provides comprehensive guidance on implementing security best practices to safeguard data stored in Amazon S3, covering access control, encryption, monitoring, and compliance considerations. Implementing these measures is crucial for maintaining the security and integrity of data in cloud environments. |
| AWS security cookbook : practical solutions for managing security policies, monitoring, auditing, and compliance with AWS | 2020 | Introduction: The book "AWS Security Cookbook" offers practical solutions for securing Amazon Web Services (AWS) infrastructure by implementing security policies, monitoring tools, and compliance measures. It emphasizes the importance of following cloud security best practices and explores various AWS services and features designed to enhance security. Book Description: The book is aimed at security consultants and professionals seeking to secure AWS infrastructure by implementing policies and following best practices. It covers a wide range of topics, including IAM and S3 policies, data security, application security, monitoring, and compliance. The discussion extends to AWS security services like Config, GuardDuty, Macie, and Inspector, along with cloud security best practices. Audience: The book targets IT security professionals, cloud security architects, and cloud application developers working on security-related roles who are interested in utilizing AWS infrastructure for secure application deployment. Key Features: Implementing Security Solutions: The book provides useful recipes for implementing robust cloud security solutions on AWS, with topics like permission policies, key management, and network security. Monitoring and Auditing: How to monitor AWS infrastructure and workloads effectively using tools like CloudWatch, CloudTrail, Config, GuardDuty, and Macie. Preparation for Certification: It helps prepare for the AWS Certified Security-Specialty exam by exploring different security models, compliance offerings, and best practices. |
| Securing Weak Points in Serverless Architectures. | 2020 | Introduction: Serverless technology has transformed enterprise computing. It enables dynamic, scalable operations while freeing organizations from the burden of server management. However, despite the security advantages that serverless models offer, security is still a shared responsibility between CSPs and users. Serverless Architectures: In serverless computing, CSPs like AWS handle infrastructure management, enabling users to focus solely on deploying code. This abstraction layer ensures secure infrastructure, but users must still secure their data and applications. Connected Services in a Serverless Architecture: Critical AWS services, including Lambda, API Gateway, and IAM, are integral to serverless architectures. However, users must carefully manage permissions and configurations to maintain security and prevent unauthorized access. Misconfigurations and Unsecure Practices: Common misconfigurations in services like S3 and Lambda, along with unsecure coding practices, can lead to data exposure and vulnerabilities. Implementing least-privilege principles and adhering to best practices are essential for mitigating these risks. Possible Compromise and Attack Scenarios: Malicious actors exploit common errors and misconfigurations to execute attacks such as credential theft, privilege escalation, and financial exploitation. Vigilance and proactive security measures are necessary to prevent such incidents. Security Measures and Recommendations: Developers should prioritize code review, and employ application security solutions to detect and prevent attacks. Regular rotation of IAM access keys and diligent configuration management are crucial for maintaining security. Conclusion: While CSPs handle infrastructure security, users must actively secure their code, data, and access controls. Adherence to security best practices and vigilance are paramount for mitigating risks and ensuring the security of serverless deployments. |
Some tools were found to be potentially helpful as well. Those tools will be considered when crafting the baseline configurations. A large amount of documentation is available from AWS for the below tools.
This project aims to bolster S3 bucket security. In order to do that, the project has been split into five key sections, each complete with subtasks needed to accomplish the overall goal, all listed below.
Starting with the AWS S3 best practice documentation, a set of baseline configurations and security controls will be developed for securing S3 buckets. These baselines will include specific recommendations that correspond to settings in the AWS console. There will be multiple baselines developed, each of which will be tailored to a different set of security requirements. For example, one baseline may be appropriate for a healthcare organization storing information that must be compliant with HIPAA. Another baseline might be applicable to an ecommerce company looking to store public images and customer reviews.
Some of the settings that should be addressed by the baselines include:
A sandbox environment will be built out in AWS. In reality, multiple sandbox environments may be needed, or may be more convenient. These sandboxes will be able to mirror the settings prescribed in the baselines defined in Step 1. The sandboxes will be built in AWS using real-world practices so that we are able to mimick closely the real-world environment that organizations would be running.
The sandbox would be used to test implementing the baselines to start. Some questions that should be addreessed by this initial round of testing might include:
Additional network configuration settings, including VPC (Virtual Private Cloud) configurations, subnets, security groups, and routing tables, will be configured to match those found in a production environment to make sure that the test environment operates within a similar network and security boundaries as real life.
Prescribed security tools such as AWS Config, AWS CloudTrail, and AWS Security Hub will be set up according to the baseline configurations.
After setting up the sandbox, penetration testing will be performed against the various sandbox setups, secured using the baseline configurations. Penetration testing should utilize available tooling such as Kali Linux and various other penetration testing tools and replicate the types of attacks that an attacker would be expected to carry out against an S3 bucket found in the real world. A mixture of automated tools and manual testing will be implemented to ensure test coverage. Custom scripts may be developed to support the testing as needed.
Penetration test results from wide open buckets or buckets configured to some basic level of hardening will be used as a "control" to judge the effectiveness of the security baselines. For example, penetration testing of a wide open bucket should show a lot of vulnerabilities easily accessible by a hacker. In order to prove that the provided S3 baseline configurations actually have a marked impact on the security of S3 buckets, test results after should show a reduction in the number of vulnerabilities.
Results from initial baselines will be noted and shared. If there are glaring vulnerabilities that are not acceptable according to the baseline's level of intended security, modifications may need to be made to the baseline and the tests re-run.
Time is a luxury for most IT departments, and especially so for the smaller organizations that can stand to benefit from S3 baselines. It would be beneficial to also provide a means of automating the implementation of the developed baselines. This would mean that a system administrator or cloud administrator could implement the baseline recommendations by running a single script rather than hunting down potentially dozens of settings in the AWS console.
The AWS CLI can be used to run scripts, and AWS Cloudformation can also be used to automate the construction of security policies and settings. Both tools will be incorporated into this step and the most thorough, most convenient methods will be selected to put forth as an option for system administrators to use to implement the baselines.
In order to do anything on AWS, you need to sign up for an AWS account. There are multiple ways to do this. If you work for or are part of an organization there may already be an organizational account that you can take advantage of. If not, you can sign up for an AWS account using your email address and a credit card (used for billing services used).
The purpose of these baselines is to enhance the security of S3 buckets. And, as was stated above, some level of testing is needed to verify the baselines are in fact contributing to the security of the buckets. So, in order to test this, three (or more) buckets are needed, one corresponding to each level of baseline provided. Each bucket will have settings adjusted according to the prescribed baseline, and can then be tested against the others.
In order to test and verify the security provided by the baselines, some sort of penetration testing box is needed. In the case of this setup, a Kali Linux instance is recommended. A Kali Linux EC2 can be spun up in the same AWS region from within the AWS console. The AWS Marketplace offers a Kali Linux AMI which is the easiest way to spin up this VM. Choosing the official Kali Linux AMI allows you to then choose what size of EC2 instance and which tier of EBS you need. A T2.micro was used for the purposes of our setup and testing, with a magnetic EBS option in order to save on AWS usage costs.