Research Solutions for how to Set up SFTP through AWS S3

[jaylong255]

AWS Transfer Family

AWS Transfer Family is a managed service that enables secure transfer of files into and out of AWS storage services. It supports protocols such as Secure File Transfer Protocol (SFTP), File Transfer Protocol Secure (FTPS), and File Transfer Protocol (FTP).

It looks like AWS Transfer Family may be the preferred way to handle this for both S3 and EFS. I'm leaving some of the Docker stuff below in case it turns out to be useful but pretty much disregard it for now.

Here's an overview of how it works:

Key Components

1. Endpoints: AWS Transfer Family creates managed endpoints that can be accessed via SFTP, FTPS, or FTP. These endpoints serve as the entry points for file transfers.

2. Users: You can create users who are authorized to connect to the endpoints. Each user can have their own specific configurations and permissions.

3. Authentication: Users can be authenticated using service-managed or custom identity providers. Service-managed users are stored within the AWS Transfer Family, whereas custom identity providers can integrate with your existing identity management systems.

4. Data Storage: Files transferred via AWS Transfer Family are stored in Amazon S3 buckets or EFS (Elastic File System). You can specify the target storage for each user.

5. Logging and Monitoring: AWS Transfer Family integrates with AWS CloudTrail and Amazon CloudWatch, allowing you to monitor and log file transfer activities.

Steps to Set Up AWS Transfer Family

1. Create a Server: In the AWS Management Console, create a new server, specifying the protocol (SFTP, FTPS, or FTP) you wish to use.

# Create an AWS Transfer Family server
resource "aws_transfer_server" "example" {
  identity_provider_type = "SERVICE_MANAGED"
  protocols             = ["SFTP"]
  endpoint_type         = "PUBLIC"

  tags = {
    Name = "example-transfer-server"
  }
}

2. Configure Access: Define the endpoint type (public or VPC), the identity provider, and other settings such as domain and endpoint details.

3. Create Users: Add users who can access the server, specifying their authentication method, home directory (in S3 or EFS), and any IAM roles needed for access control.

# Create a user for the Transfer server
resource "aws_transfer_user" "example_user" {
  server_id = aws_transfer_server.example.id
  user_name = "example-user"
  role      = aws_iam_role.transfer_server_role.arn

  home_directory = "/example-bucket"
}

4. Specify Permissions: Configure the IAM policies to ensure users have the appropriate permissions to read, write, or manage the files in the specified S3 bucket or EFS.

# Create an IAM role for the Transfer server
resource "aws_iam_role" "transfer_server_role" {
  name = "transfer-server-iam-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "transfer.amazonaws.com"
        }
      }
    ]
  })
}

# Attach an IAM policy to the role
resource "aws_iam_role_policy_attachment" "transfer_server_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSTransferLoggingAccess"
  role       = aws_iam_role.transfer_server_role.name
}

5. Transfer Files: Users can now connect to the server using their preferred file transfer client and protocol to upload or doacwnload files.

Features and Benefits

• Managed Service: Reduces the overhead of managing your own file transfer infrastructure.
• Security: Supports encryption in transit and at rest, and integrates with AWS IAM for fine-grained access control.
• Scalability: Automatically scales to accommodate varying file transfer volumes.
• Compatibility: Supports standard file transfer protocols, making it easy to integrate with existing systems.
• Cost-Effective: Pay-as-you-go pricing, where you only pay for the endpoints you use and the data transferred.

Use Cases

• Data Migration: Securely transferring large datasets to AWS for processing and analysis.
• Backup and Recovery: Automating backups from on-premises systems to AWS.
• File Sharing: Providing a secure and managed way for partners and customers to exchange files.

Overall, AWS Transfer Family simplifies and secures the process of transferring files to and from AWS, making it a versatile solution for a variety of file transfer needs.

[jaylong255]

AWS Transfer Family

[jaylong255]

AWS Transfer Family Terraform Module

[jaylong255]

Here's an example of how to create an AWS Transfer Family server using Terraform:

# Configure the AWS provider
provider "aws" {
  region = "us-west-2"
}

# Create an AWS Transfer Family server
resource "aws_transfer_server" "example" {
  identity_provider_type = "SERVICE_MANAGED"
  protocols             = ["SFTP"]
  endpoint_type         = "PUBLIC"

  tags = {
    Name = "example-transfer-server"
  }
}

# Create an IAM role for the Transfer server
resource "aws_iam_role" "transfer_server_role" {
  name = "transfer-server-iam-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "transfer.amazonaws.com"
        }
      }
    ]
  })
}

# Attach an IAM policy to the role
resource "aws_iam_role_policy_attachment" "transfer_server_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSTransferLoggingAccess"
  role       = aws_iam_role.transfer_server_role.name
}

# Create a user for the Transfer server
resource "aws_transfer_user" "example_user" {
  server_id = aws_transfer_server.example.id
  user_name = "example-user"
  role      = aws_iam_role.transfer_server_role.arn

  home_directory = "/example-bucket"
}

# Create an SSH key for the user
resource "aws_transfer_ssh_key" "example_ssh_key" {
  server_id = aws_transfer_server.example.id
  user_name = aws_transfer_user.example_user.user_name
  body      = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD3F6tyPEFEzV0LX3X8BsXdMsQz1x2cEikKDEY0aIj41qgxMCP/iteneqXSIFZBp5vizPvaoIR3Um9xK7PGoW8giupGn+EPuxIA4cDM4vzOqOkiMPhz5XK0whEjkVzTo4+S0puvDZuwIsdiW9mxhJc7tgBNL0cYlWSYVkz4G/fslNfRPW5mYAM49f4fhtxPb5ok4Q2Lg9dPKVHO/Bgeu5woMc7RY0p1ej6D4CKFE6lymSDJpW0YHX/wqE9+cfEauh7xZcG0q9t2ta6F6fmX0agvpFyZo8aFbXeUBr7osSCJNgvavWbM/06niWrOvYX2xwWdhXmXSrbX8ZbabVohBK41 example@example.com"
}

# Output the server endpoint
output "sftp_endpoint" {
  value = aws_transfer_server.example.endpoint
}

Would you like me to explain or break down this code?

[jaylong255]

[jaylong255]

https://docs.aws.amazon.com/transfer/latest/userguide/transfer-file.html#filezilla

[jaylong255]

Dockerfile for Testing FTP Server

# Use an official Ubuntu as a parent image
FROM ubuntu:latest

# Set environment variables
ENV FTP_USER=user
ENV FTP_PASS=pass

# Install vsftpd
RUN apt-get update && \
    apt-get install -y vsftpd && \
    rm -rf /var/lib/apt/lists/*

# Create FTP user with password
RUN useradd -m $FTP_USER && echo "$FTP_USER:$FTP_PASS" | chpasswd

# Set up vsftpd configuration
RUN echo "listen=YES" >> /etc/vsftpd.conf && \
    echo "anonymous_enable=NO" >> /etc/vsftpd.conf && \
    echo "local_enable=YES" >> /etc/vsftpd.conf && \
    echo "write_enable=YES" >> /etc/vsftpd.conf && \
    echo "chroot_local_user=YES" >> /etc/vsftpd.conf && \
    echo "allow_writeable_chroot=YES" >> /etc/vsftpd.conf && \
    echo "pasv_min_port=40000" >> /etc/vsftpd.conf && \
    echo "pasv_max_port=40010" >> /etc/vsftpd.conf && \
    echo "pasv_address=127.0.0.1" >> /etc/vsftpd.conf

# Expose ports
EXPOSE 20 21 40000-40010

# Add startup script
COPY start_vsftpd.sh /start_vsftpd.sh
RUN chmod +x /start_vsftpd.sh

# Run vsftpd
CMD ["/start_vsftpd.sh"]

Startup Script for FTP Server

#!/bin/bash

# Start vsftpd
/usr/sbin/vsftpd /etc/vsftpd.conf

[jaylong255]

Build Command

docker build -t ftp-server .

Run Command

docker run -d -p 21:21 -p 20:20 -p 40000-40010:40000-40010 ftp-server

[jaylong255]

Prerequisites

1. FileZilla Pro: Ensure you have the Pro version of FileZilla, as the free version does not support S3 interfaces.
2. FTP Enabled on Wasabi: Make sure FTP is enabled on your Wasabi account.

Configuring FileZilla Pro for Wasabi

1. Add Wasabi as a Provider:

   • Open FileZilla Pro.
   • Go to Settings -> Transfers -> S3: Providers.
   • Add a new provider for Wasabi.
   • Enter the appropriate endpoints for the Wasabi region you are using (e.g., s3.us-east-1.wasabisys.com for the us-east-1 region).

2. Set Up the Wasabi Connection:

   • Open the Site Manager in FileZilla Pro.
   • Create a new site and select the S3 - Amazon Simple Storage Service protocol.
   • Enter your Wasabi Access Key and Secret Key.

Additional Notes

• Multi-Factor Authentication (MFA):
  • If MFA is enabled, combine your password with the MFA code.
  • Format: Password/mfa=nnnnnn where nnnnnn is the 6-digit code from your Authenticator App.
  • Example: If your password is Password1 and the MFA code is 123456, enter Password1/mfa=123456 in the password field.