search

oceanmodeling / ondemand-storm-workflow

Other
2 stars 1 forks source link

Running flows as ECS tasks #16

Open SorooshMani-NOAA opened 1 year ago

SorooshMani-NOAA commented 1 year ago

Prefect 2 is different in how the flow runs are passed to agents. To make ensemble work, first attempt was using Fargate task, to do so the odssm-flowrun-task needs to be modified to require FARGATE for requires_compatibilities and switched the network mode to awsvpc. The main issue was that this would fail with an error like:

Submission failed. prefect_aws.ecs.TaskFailedToStart: ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "fs-3cb78988.efs.us-east-1.amazonaws.com" - check that your file system ID is correct, and ensure that the VPC has an EFS mount target for this file system ID. See https://docs.aws.amazon.com/console/efs/mount-dns-name for more detail. Attempting to lookup mount target ip address using botocore. Failed to import necessary dependency botocore, please install botocore first. : unsuccessful EFS utils command execution; code: 1

So I switched back to EC2, but the problem there is adding the placement constraint. Right now using the second EC2 works, as long as there's only a single main end-to-end run. Also #14 affects the run here.

SorooshMani-NOAA commented 1 year ago

The vpc_id was missing from the Prefect ECSTask infrastructure definition. I got the EC2 launch type (without explicit placement) to work, next I can try FARGATE but this time with vpc_id added to the ECSTask definition.