ECSTask block `publish_as_work_pool` does not set `network_configuration`

[mattiamatrix]

Hello, I am evaluating the migration from Agent to WorkPool. I have a few ECSTask blocks that I can easily transform into WorkPools with publish_as_work_pool.

It partially works but, these ECSTask blocks have the Task Customizations set in the form of:

[
  {
    "op": "replace",
    "path": "/networkConfiguration/awsvpcConfiguration/assignPublicIp",
    "value": "DISABLED"
  },
  {
    "op": "add",
    "path": "/networkConfiguration/awsvpcConfiguration/subnets",
    "value": ["subnet-***"]
  },
  {
    "op": "add",
    "path": "/networkConfiguration/awsvpcConfiguration/securityGroups",
    "value": ["sg-***"]
  }
]

These settings need to be replicated in the worker network_configuration as:

{
  "subnets": ["subnet-****"],
  "assignPublicIp": "DISABLED",
  "securityGroups": ["sg-****"]
}

I can do this manually from the UI, but it is not done automatically when using publish_as_work_pool.

Is this a limitation or is there a workaround I can use to update the ECSWorker?

I looked at the code but the creation of ECSWorker is a bit magical. I would like to create an ECSWorker the same way as I create ECSTask(...).save("...") in Python code but I couldn't find any documentation to do it. Looks like ECSWorker can only be created from UI or CLI.

Expectation / Proposal

• Have documentation on how to create ECSWorker in Python
• Create ECSWorker workers with a patter similar to that used to create ECSTask blocks
• Set network_configuration in ECSWorker manually or via publish_as_work_pool

Traceback / Example

• [ ] I would like to help contribute a pull request to resolve this!

[desertaxle]

Thanks for the issue @mattiamatrix! I'm very excited to see that you're starting a migration to the ECS worker!

I had some challenges when implementing the network configuration portion of .publish_as_work_pool and it hasn't been fully implemented yet. We can take another run at including the network configuration when created a work pool from an ECSTask block. The current workaround is to edit the base job template via either the UI or the CLI to include your network configuration. We don't currently have a Pythonic way to work with work pool base job templates, but if that's something that you would like to see, feel free to submit an enhancement request on the core Prefect repo!

[mattiamatrix]

Thank you @desertaxle for the quick answer!! 🙏

I created https://github.com/PrefectHQ/prefect/issues/12091, hope it makes sense!

In our setup, we use python to define secrets, blocks, notifications, etc. It would be great to have the same for work pools and workers.