search

OpenDroneMap / opendronemap-ecs

Serverless API to get opendronemap tasks running on AWS Elastic Container Service (ECS)
GNU General Public License v3.0
5 stars 6 forks source link

readme

opendronemap-ecs

Serverless API to get opendronemap tasks running on AWS Elastic Container, with input files copied in from and output files copied back out to Simple Storage Service (S3).

Note

STILL IN DEVELOPMENT

Requirements

  1. Install serverless and the AWS command line tool:

  2. Create the cluster by running aws ecs create-cluster --cluster-name "odm". This is just a one-off step (unless you delete the cluster) that just registers the cluster name, we separately manage add (and remove) EC2 instances to the cluster through an autoscaling group as described later in Step 4.

  3. Configure an autoscaling launch configuration through the AWS EC2 console. Go to your desired region and scroll the left nav down to find 'AUTO SCALING'>'Launch Configurations' then select 'Create Launch Configuration'

    • Choose AMI - Under the AWS Marketplace tab, search for "ECS" to find the Amazon ECS-Optimized Amazon Linux AMI. A list of current AMI IDs for ECS AMIs by region can be found here.
    • Choose Instance Type - I recommend r4.4xlarge and the memory setting in odm-task-definition.json is geared for that.
    • Configure Details - Enter a name of your choice, review other settings, make sure the IAM role has both access to your s3 bucket (example policy you can attach) and AmazonEC2ContainerServiceforEC2Role permissions, and under 'Advanced Details' for 'User data' enter user-data.yml.
    • Add storage - make sure the root volume is big enough to (temporarily) hold your input files for an individual task (assuming one task / instance). No need to add an extra EBS volume.
    • Configure security group - Select or create a security group (no inbound ports strictly required although for debugging in the EC2 instance you must allow ssh in). If launching in EC2-classic (if your account supports that) then the security group must be an EC2-classic security group.

  4. Launch an autoscaling cluster using the new configuration. At the end of Step 3 select 'Create an Auto Scaling group using this launch configuration' or go to your desired region and scroll the left nav down to find 'AUTO SCALING'>'Auto Scaling Groups' then select 'Create Auto Scaling group' then choose 'Create an Auto Scaling group from an existing launch configuration' and select the launch configuration created in Step 3.

    • : Configure Auto Scaling group details
      • Group name -
      • Group size -
      • Network - For network select a VPC (default unless you want to have a separate isolated VPC). If your account is old enough that you have the choice of EC2-classic or EC2-VPC, I recommend EC2-VPC, although it will work in either. Note previous comment about security groups in Step 1.E.
      • Subnet - For higher availability I recommend using subnets across all availability zones.
    • : Configure scaling policies - I recommend one based on average CPU to start / terminate instances based on demand, as ODM is a CPU intensive task.
      • Scale between X and Y instances - Here set a minimum of 1, and set a maximum equal to the number of image processing tasks (one task per instance, however note issue discussed here), one image task corresponding to one project folder per the /run API discussed below. Note you may wish to get your AWS account limit on number of EC2 instances adjusted (see EC2 limits page).
      • Target value: - I recommend 30% so scaling happens early in an ODM task, a bit low to allow for more capacity for new tasks while the first task is starting up, but not so low to trigger other minor background processes on the EC2 instances in the cluster. Note that the autoscaling group can be deleted, which you may wish to do once your tasks are finished. You can launch a new one later for further processing by repeating this step. There's no need to delete the cluster, as it can remain around as a logical entity with no instances registered, and if you do delete it, you would have to re-run Step 2.

  5. Edit serverless.yml with your custom settings as needed, you can then deploy the service by running:

    serverless deploy

    This will return the URLs that you can then use to interact with the API you then set up custom domain with TLS in AWS web console for API Gateway or using the AWS CLI tool. For example:

    Serverless: Stack update finished...
Service Information
service: odm
stage: prod
region: us-east-1
stack: odm-prod
api keys:
None
endpoints:
POST - https://abc123.execute-api.us-east-1.amazonaws.com/prod/run
POST - https://abc123.execute-api.us-east-1.amazonaws.com/prod/register
functions:
run: odm-prod-run
register: odm-prod-register

API

Debugging

Web