search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.95k stars 1.57k forks source link

`prefect worker start --type ecs --pool batch --install-policy if-not-present` no longer works in prefect 2.x #15232

Closed altso closed 3 weeks ago

altso commented 3 weeks ago

Bug summary

prefect worker start --type ecs --pool batch --install-policy if-not-present no longer works in Prefect 2 as it tries to install the latest prefect-aws (0.5.0) which references prefect>=3.0.0rc1. All versions as of 9/4/2024.

This could be easily reproduced in docker:

> docker run --rm -it prefecthq/prefect:2-latest prefect worker start --type ecs --pool batch --install-policy if-not-present
/usr/local/lib/python3.10/contextlib.py:142: SAWarning: Skipped unsupported reflection of expression-based index ix_flow_run__coalesce_start_time_expected_start_time_desc
  next(self.gen)
/usr/local/lib/python3.10/contextlib.py:142: SAWarning: Skipped unsupported reflection of expression-based index ix_flow_run__coalesce_start_time_expected_start_time_asc
  next(self.gen)
Installing prefect-aws...
Collecting prefect-aws
  Downloading prefect_aws-0.5.0-py3-none-any.whl.metadata (2.9 kB)
Collecting boto3>=1.24.53 (from prefect-aws)
  Downloading boto3-1.35.12-py3-none-any.whl.metadata (6.6 kB)
Collecting botocore>=1.27.53 (from prefect-aws)
  Downloading botocore-1.35.12-py3-none-any.whl.metadata (5.7 kB)
Collecting mypy-boto3-s3>=1.24.94 (from prefect-aws)
  Downloading mypy_boto3_s3-1.35.2-py3-none-any.whl.metadata (17 kB)
Collecting mypy-boto3-secretsmanager>=1.26.49 (from prefect-aws)
  Downloading mypy_boto3_secretsmanager-1.35.0-py3-none-any.whl.metadata (13 kB)
Collecting prefect>=3.0.0rc1 (from prefect-aws)
  Downloading prefect-3.0.0-py3-none-any.whl.metadata (11 kB)
Collecting pyparsing>=3.1.1 (from prefect-aws)
  Downloading pyparsing-3.1.4-py3-none-any.whl.metadata (5.1 kB)
Collecting tenacity>=8.0.0 (from prefect-aws)
  Downloading tenacity-9.0.0-py3-none-any.whl.metadata (1.2 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3>=1.24.53->prefect-aws)
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3>=1.24.53->prefect-aws)
  Downloading s3transfer-0.10.2-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.10/site-packages (from botocore>=1.27.53->prefect-aws) (2.9.0.post0)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/local/lib/python3.10/site-packages (from botocore>=1.27.53->prefect-aws) (2.2.2)
Requirement already satisfied: typing-extensions>=4.1.0 in /usr/local/lib/python3.10/site-packages (from mypy-boto3-s3>=1.24.94->prefect-aws) (4.12.2)
Requirement already satisfied: aiosqlite<1.0.0,>=0.17.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.20.0)
Requirement already satisfied: alembic<2.0.0,>=1.7.5 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.13.2)
Requirement already satisfied: apprise<2.0.0,>=1.1.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.8.1)
Requirement already satisfied: asyncpg<1.0.0,>=0.23 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.29.0)
Requirement already satisfied: click<8.2,>=8.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (8.1.7)
Requirement already satisfied: cryptography>=36.0.1 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (43.0.0)
Requirement already satisfied: dateparser<2.0.0,>=1.1.1 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.2.0)
Requirement already satisfied: docker<8.0,>=4.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (7.1.0)
Requirement already satisfied: graphviz>=0.20.1 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.20.3)
Requirement already satisfied: jinja2<4.0.0,>=3.0.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (3.1.4)
Requirement already satisfied: jinja2-humanize-extension>=0.4.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.4.0)
Requirement already satisfied: humanize<5.0.0,>=4.9.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (4.10.0)
Requirement already satisfied: pytz<2025,>=2021.1 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (2024.1)
Requirement already satisfied: readchar<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (4.2.0)
Requirement already satisfied: sqlalchemy<3.0.0,>=2.0 in /usr/local/lib/python3.10/site-packages (from sqlalchemy[asyncio]<3.0.0,>=2.0->prefect>=3.0.0rc1->prefect-aws) (2.0.32)
Requirement already satisfied: typer!=0.12.2,<0.13.0,>=0.12.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.12.5)
Requirement already satisfied: anyio<5.0.0,>=4.4.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (4.4.0)
Requirement already satisfied: asgi-lifespan<3.0,>=1.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (2.1.0)
Requirement already satisfied: cachetools<6.0,>=5.3 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (5.5.0)
Requirement already satisfied: cloudpickle<4.0,>=2.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (3.0.0)
Requirement already satisfied: coolname<3.0.0,>=1.0.4 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (2.2.0)
Requirement already satisfied: croniter<4.0.0,>=1.0.12 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (2.0.7)
Requirement already satisfied: exceptiongroup>=1.0.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.2.2)
Collecting fastapi<1.0.0,>=0.111.0 (from prefect>=3.0.0rc1->prefect-aws)
  Downloading fastapi-0.112.2-py3-none-any.whl.metadata (27 kB)
Requirement already satisfied: fsspec>=2022.5.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (2024.6.1)
Requirement already satisfied: griffe<2.0.0,>=0.49.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.2.0)
Requirement already satisfied: httpcore<2.0.0,>=1.0.5 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.0.5)
Requirement already satisfied: httpx!=0.23.2,>=0.23 in /usr/local/lib/python3.10/site-packages (from httpx[http2]!=0.23.2,>=0.23->prefect>=3.0.0rc1->prefect-aws) (0.27.2)
Requirement already satisfied: jsonpatch<2.0,>=1.32 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.33)
Requirement already satisfied: jsonschema<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (4.23.0)
Requirement already satisfied: orjson<4.0,>=3.7 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (3.10.7)
Requirement already satisfied: packaging<24.3,>=21.3 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (24.1)
Requirement already satisfied: pathspec>=0.8.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.12.1)
Collecting pendulum<4,>=3.0.0 (from prefect>=3.0.0rc1->prefect-aws)
  Downloading pendulum-3.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.9 kB)
Collecting prometheus-client>=0.20.0 (from prefect>=3.0.0rc1->prefect-aws)
  Downloading prometheus_client-0.20.0-py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: pydantic<3.0.0,>=2.7 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (2.8.2)
Requirement already satisfied: pydantic-core<3.0.0,>=2.12.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (2.20.1)
Collecting pydantic-extra-types<3.0.0,>=2.8.2 (from prefect>=3.0.0rc1->prefect-aws)
  Downloading pydantic_extra_types-2.9.0-py3-none-any.whl.metadata (3.4 kB)
Collecting pydantic-settings (from prefect>=3.0.0rc1->prefect-aws)
  Downloading pydantic_settings-2.4.0-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: python-slugify<9.0,>=5.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (8.0.4)
Requirement already satisfied: pyyaml<7.0.0,>=5.4.1 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (6.0.2)
Requirement already satisfied: rfc3339-validator<0.2.0,>=0.1.4 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.1.4)
Requirement already satisfied: rich<14.0,>=11.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (13.8.0)
Requirement already satisfied: ruamel.yaml>=0.17.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.18.6)
Requirement already satisfied: sniffio<2.0.0,>=1.3.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (1.3.1)
Requirement already satisfied: toml>=0.10.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.10.2)
Requirement already satisfied: ujson<6.0.0,>=5.8.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (5.10.0)
Requirement already satisfied: uvicorn!=0.29.0,>=0.14.0 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (0.30.6)
Requirement already satisfied: websockets<14.0,>=10.4 in /usr/local/lib/python3.10/site-packages (from prefect>=3.0.0rc1->prefect-aws) (13.0.1)
Requirement already satisfied: Mako in /usr/local/lib/python3.10/site-packages (from alembic<2.0.0,>=1.7.5->prefect>=3.0.0rc1->prefect-aws) (1.3.5)
Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/site-packages (from anyio<5.0.0,>=4.4.0->prefect>=3.0.0rc1->prefect-aws) (3.8)
Requirement already satisfied: certifi in /usr/local/lib/python3.10/site-packages (from apprise<2.0.0,>=1.1.0->prefect>=3.0.0rc1->prefect-aws) (2024.8.30)
Requirement already satisfied: requests in /usr/local/lib/python3.10/site-packages (from apprise<2.0.0,>=1.1.0->prefect>=3.0.0rc1->prefect-aws) (2.32.3)
Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.10/site-packages (from apprise<2.0.0,>=1.1.0->prefect>=3.0.0rc1->prefect-aws) (2.0.0)
Requirement already satisfied: markdown in /usr/local/lib/python3.10/site-packages (from apprise<2.0.0,>=1.1.0->prefect>=3.0.0rc1->prefect-aws) (3.7)
Requirement already satisfied: async-timeout>=4.0.3 in /usr/local/lib/python3.10/site-packages (from asyncpg<1.0.0,>=0.23->prefect>=3.0.0rc1->prefect-aws) (4.0.3)
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/site-packages (from cryptography>=36.0.1->prefect>=3.0.0rc1->prefect-aws) (1.17.0)
Requirement already satisfied: regex!=2019.02.19,!=2021.8.27 in /usr/local/lib/python3.10/site-packages (from dateparser<2.0.0,>=1.1.1->prefect>=3.0.0rc1->prefect-aws) (2024.7.24)
Requirement already satisfied: tzlocal in /usr/local/lib/python3.10/site-packages (from dateparser<2.0.0,>=1.1.1->prefect>=3.0.0rc1->prefect-aws) (5.2)
Collecting starlette<0.39.0,>=0.37.2 (from fastapi<1.0.0,>=0.111.0->prefect>=3.0.0rc1->prefect-aws)
  Downloading starlette-0.38.4-py3-none-any.whl.metadata (6.0 kB)
Requirement already satisfied: colorama>=0.4 in /usr/local/lib/python3.10/site-packages (from griffe<2.0.0,>=0.49.0->prefect>=3.0.0rc1->prefect-aws) (0.4.6)
Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/site-packages (from httpcore<2.0.0,>=1.0.5->prefect>=3.0.0rc1->prefect-aws) (0.14.0)
Requirement already satisfied: h2<5,>=3 in /usr/local/lib/python3.10/site-packages (from httpx[http2]!=0.23.2,>=0.23->prefect>=3.0.0rc1->prefect-aws) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/site-packages (from jinja2<4.0.0,>=3.0.0->prefect>=3.0.0rc1->prefect-aws) (2.1.5)
Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/site-packages (from jsonpatch<2.0,>=1.32->prefect>=3.0.0rc1->prefect-aws) (3.0.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/site-packages (from jsonschema<5.0.0,>=4.0.0->prefect>=3.0.0rc1->prefect-aws) (24.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/site-packages (from jsonschema<5.0.0,>=4.0.0->prefect>=3.0.0rc1->prefect-aws) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/site-packages (from jsonschema<5.0.0,>=4.0.0->prefect>=3.0.0rc1->prefect-aws) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/site-packages (from jsonschema<5.0.0,>=4.0.0->prefect>=3.0.0rc1->prefect-aws) (0.20.0)
Collecting tzdata>=2020.1 (from pendulum<4,>=3.0.0->prefect>=3.0.0rc1->prefect-aws)
  Downloading tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting time-machine>=2.6.0 (from pendulum<4,>=3.0.0->prefect>=3.0.0rc1->prefect-aws)
  Downloading time_machine-2.15.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/site-packages (from pydantic<3.0.0,>=2.7->prefect>=3.0.0rc1->prefect-aws) (0.7.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore>=1.27.53->prefect-aws) (1.16.0)
Requirement already satisfied: text-unidecode>=1.3 in /usr/local/lib/python3.10/site-packages (from python-slugify<9.0,>=5.0->prefect>=3.0.0rc1->prefect-aws) (1.3)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/site-packages (from rich<14.0,>=11.0->prefect>=3.0.0rc1->prefect-aws) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/site-packages (from rich<14.0,>=11.0->prefect>=3.0.0rc1->prefect-aws) (2.18.0)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /usr/local/lib/python3.10/site-packages (from ruamel.yaml>=0.17.0->prefect>=3.0.0rc1->prefect-aws) (0.2.8)
Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/site-packages (from sqlalchemy<3.0.0,>=2.0->sqlalchemy[asyncio]<3.0.0,>=2.0->prefect>=3.0.0rc1->prefect-aws) (3.0.3)
Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/site-packages (from typer!=0.12.2,<0.13.0,>=0.12.0->prefect>=3.0.0rc1->prefect-aws) (1.5.4)
Collecting python-dotenv>=0.21.0 (from pydantic-settings->prefect>=3.0.0rc1->prefect-aws)
  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=36.0.1->prefect>=3.0.0rc1->prefect-aws) (2.22)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]!=0.23.2,>=0.23->prefect>=3.0.0rc1->prefect-aws) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]!=0.23.2,>=0.23->prefect>=3.0.0rc1->prefect-aws) (4.0.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich<14.0,>=11.0->prefect>=3.0.0rc1->prefect-aws) (0.1.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/site-packages (from requests->apprise<2.0.0,>=1.1.0->prefect>=3.0.0rc1->prefect-aws) (3.3.2)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/site-packages (from requests-oauthlib->apprise<2.0.0,>=1.1.0->prefect>=3.0.0rc1->prefect-aws) (3.2.2)
Downloading prefect_aws-0.5.0-py3-none-any.whl (49 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.8/49.8 kB 1.4 MB/s eta 0:00:00
Downloading boto3-1.35.12-py3-none-any.whl (139 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.2/139.2 kB 3.3 MB/s eta 0:00:00
Downloading botocore-1.35.12-py3-none-any.whl (12.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.5/12.5 MB 21.9 MB/s eta 0:00:00
Downloading mypy_boto3_s3-1.35.2-py3-none-any.whl (84 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.0/84.0 kB 2.4 MB/s eta 0:00:00
Downloading mypy_boto3_secretsmanager-1.35.0-py3-none-any.whl (26 kB)
Downloading prefect-3.0.0-py3-none-any.whl (6.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 21.7 MB/s eta 0:00:00
Downloading pyparsing-3.1.4-py3-none-any.whl (104 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.1/104.1 kB 3.4 MB/s eta 0:00:00
Downloading tenacity-9.0.0-py3-none-any.whl (28 kB)
Downloading fastapi-0.112.2-py3-none-any.whl (93 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.5/93.5 kB 2.9 MB/s eta 0:00:00
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading pendulum-3.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (384 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 384.9/384.9 kB 6.8 MB/s eta 0:00:00
Downloading prometheus_client-0.20.0-py3-none-any.whl (54 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 kB 1.3 MB/s eta 0:00:00
Downloading pydantic_extra_types-2.9.0-py3-none-any.whl (30 kB)
Downloading s3transfer-0.10.2-py3-none-any.whl (82 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.7/82.7 kB 2.2 MB/s eta 0:00:00
Downloading pydantic_settings-2.4.0-py3-none-any.whl (23 kB)
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Downloading starlette-0.38.4-py3-none-any.whl (71 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.4/71.4 kB 2.3 MB/s eta 0:00:00
Downloading time_machine-2.15.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34 kB)
Downloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 345.4/345.4 kB 9.7 MB/s eta 0:00:00
Installing collected packages: tzdata, tenacity, python-dotenv, pyparsing, prometheus-client, mypy-boto3-secretsmanager, mypy-boto3-s3, jmespath, time-machine, starlette, botocore, s3transfer, pydantic-settings, pydantic-extra-types, pendulum, fastapi, prefect, boto3, prefect-aws
  Attempting uninstall: pendulum
    Found existing installation: pendulum 2.1.2
    Uninstalling pendulum-2.1.2:
      Successfully uninstalled pendulum-2.1.2
  Attempting uninstall: prefect
    Found existing installation: prefect 2.20.4
    Uninstalling prefect-2.20.4:
      Successfully uninstalled prefect-2.20.4
Successfully installed boto3-1.35.12 botocore-1.35.12 fastapi-0.112.2 jmespath-1.0.1 mypy-boto3-s3-1.35.2 mypy-boto3-secretsmanager-1.35.0 pendulum-3.0.0 prefect-3.0.0 prefect-aws-0.5.0 prometheus-client-0.20.0 pydantic-extra-types-2.9.0 pydantic-settings-2.4.0 pyparsing-3.1.4 python-dotenv-1.0.1 s3transfer-0.10.2 starlette-0.38.4 tenacity-9.0.0 time-machine-2.15.0 tzdata-2024.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
Warning!  Failed to load collection 'prefect_aws': RuntimeError: no validator found for <class 'pydantic.types.SecretStr'>, see `arbitrary_types_allowed` in Config
Unable to start worker. Please ensure you have the necessary dependencies installed to run your desired worker type.

Version info (prefect version output)

Version:             2.20.4
API version:         0.8.4
Python version:      3.10.14
Git commit:          3b951c35
Built:               Wed, Aug 28, 2024 9:09 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.40.1

Additional context

A workaround that seems to work is to use EXTRA_PIP_PACKAGES instead of --install-policy if-not-present

docker run --rm -it -e EXTRA_PIP_PACKAGES="prefect-aws==0.4.19" prefecthq/prefect:2-latest prefect worker start --type ecs --pool batch
desertaxle commented 3 weeks ago

Thanks for the bug report @altso!

We updated adhoc installs like this to use pip extras in https://github.com/PrefectHQ/prefect/pull/14242, but it looks like that never made it to the 2.x branch. I'll port this change over now so that this can be fixed in the next release!

altso commented 3 weeks ago

Thanks. Looking forward to the next release.

desertaxle commented 3 weeks ago

Fix for this has been released in 2.20.5!