This is a module for Terraform that deploys Dagster in AWS.
Average cost of minimal setup with RDS: ~60$/month
The Dagster setup provided with this module is intended to be used to manage your runs/schedules/etc... If you want Dagster to have access to services like AWS EMR, AWS Glue, ..., use the output role and give it permissions to these services through IAM.
module "dagster" {
source = "datarootsio/ecs-dagster/aws"
resource_prefix = "my-awesome-company"
resource_suffix = "env"
vpc_id = "vpc-123456"
public_subnet_ids = ["subnet-456789", "subnet-098765"]
rds_password = "super-secret-pass"
}
To add new pipelines to Dagster:
workspace.yml
file the new pipeline file name and its path in the mounted volume of the ECS instance,
load_from:
- python_file:
relative_path: new_pipeline.py
working_directory: /path/to/mounted/volume
workspace.yml
file.This module supports HTTPS however RBAC is not supported (yet?) by Dagster. Therefore, this module results in Dagit being publicly available. It is important to note that the user should implement and manage authentication themselves, for example by implementing SSO.
Name | Version |
---|---|
terraform | >= 0.14 |
Name | Version |
---|---|
aws | n/a |
Name | Description | Type | Default | Required |
---|---|---|---|---|
aws_availability_zone | The availability zone of the resource. | string |
"eu-west-1a" |
no |
aws_region | The region of the aws account | string |
"eu-west-1" |
no |
certificate_arn | The arn of the certificate that will be used. | string |
"" |
no |
dagster-container-home | n/a | string |
"/opt" |
no |
dagster_config_bucket | Dagster bucket containing the config files. | string |
"dagster-bucket" |
no |
dagster_file | The config file needed to use database and daemon with dagit. | string |
"dagster.yaml" |
no |
dns_name | The dns name that will be used to expose Dagster. It will be auto generated if not provided. | string |
"" |
no |
ecs_cpu | The amount of cpu to give to the ECS instance. | number |
1024 |
no |
ecs_memory | The amount of ecs memory to give to the ECS instance. | number |
2048 |
no |
ip_allow_list | A list of ip ranges that are allowed to access the airflow webserver, default: full access | list(string) |
[ |
no |
log_retention | The number of days that the logs shoud live. | number |
7 |
no |
private_subnet | The private subnets where the RDS and ECS reside. | list(string) |
[] |
no |
public_subnet | The public subnet where the load balancer should reside. Moreover, the ecs and rds will use these if no private subnets are defined. At least two should be provided. | list(string) |
[] |
no |
rds_deletion_protection | n/a | bool |
false |
no |
rds_instance_class | The type of instance class for the RDS. | string |
"db.t2.micro" |
no |
rds_password | The password to access the RDS instance. | string |
"" |
no |
rds_skip_final_snapshot | Whether or not to skip the final snapshot before deleting (mainly for tests) | bool |
true |
no |
rds_username | The username to access the RDS instance. | string |
"" |
no |
resource_prefix | The prefix of the resource to be created | string |
"ps" |
no |
resource_suffix | The suffix of the resource to be created | string |
"sp" |
no |
route53_zone_name | The name of the route53 zone that will be used for the certificate validation. | string |
"" |
no |
tags | Tags to add to the created resources. | map(string) |
{ |
no |
use_https | Expose traffic using HTTPS or not | bool |
false |
no |
vpc | The id of the virtual private cloud. | string |
"" |
no |
workspace_file | The config file needed to run dagit. | string |
"workspace.yaml" |
no |
Name | Description |
---|---|
dagster_alb_dns | The DNS name of the ALB, with this you can access the dagster webserver |
dagster_connection_sg | The security group with which you can connect other instance to dagster, for example EMR Livy |
dagster_dns_record | The created DNS record (only if "use_https" = true) |
dagster_task_iam_role | The IAM role of the dagster task, use this to give dagster more permissions |
Available targets:
tools Pull Go and Terraform dependencies
fmt Format Go and Terraform code
lint/lint-tf/lint-go Lint Go and Terraform code
test/testverbose Run tests
Contributions to this repository are very welcome! Found a bug or do you have a suggestion? Please open an issue. Do you know how to fix it? Pull requests are welcome as well! To get you started faster, a Makefile is provided.
Make sure to install Terraform, Go (for automated testing) and Make (optional, if you want to use the Makefile) on your computer. Install tflint to be able to run the linting.
make tools
make fmt
make lint
make test
(or go test -timeout 2h ./...
without Make)Make sure you branch from the 'open-pr-here' branch, and submit a PR back to the 'open-pr-here' branch.
MIT license. Please see LICENSE for details.