search

dagster-io / dagster-cloud

Unify all your data tools into a productive, enterprise-grade platform.
https://dagster.io/cloud
21 stars 13 forks source link

ECS Fargate Public IP allocation #18

Open RoyMaunders opened 3 months ago

RoyMaunders commented 3 months ago

We have a problem with ECS Fargate tasks being started with public IPs when it isn't necessary.

https://github.com/dagster-io/dagster-cloud/blob/f76340913ca05479c96216f7e1eb702170e14b72/dagster-cloud/dagster_cloud/workspace/ecs/client.py#L720

The code makes a check for NAT gateways to choose if to assign public IP. While this works for the simple case its not comprehensive and so maybe we could introduce a simple configuration value to switch on/off public IP allocation.

We have a hub and spoke style network with VPCs connected via the transit gateway. All outbound internet access is sent via a central egress vpc/subnet. So no NAT gateway in this VPC but its still a 'private' subnet.

mlarose commented 1 month ago

@RoyMaunders sorry for the delay in response time.

I wonder if instead of the configuration solution, if changing the logic from detecting NAT gateways to detecting if the subnets used are routing to an Internet Gateway. Presumably, there is no such IGW in your network architecture where the Dagster Agent is deployed?

RoyMaunders commented 1 month ago

@RoyMaunders sorry for the delay in response time.

I wonder if instead of the configuration solution, if changing the logic from detecting NAT gateways to detecting if the subnets used are routing to an Internet Gateway. Presumably, there is no such IGW in your network architecture where the Dagster Agent is deployed?

Hello @mlarose, Thanks for taking a look. That would certainly fix our usage. You are correct in assuming there is no IGW where our dagster agent is deployed. I'd suggested a configuration setting so you wouldn't have to potentially solve the problem for other edge cases.