pull_step might install incompatible dependencies

First check

[X] I added a descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the Prefect documentation for this issue.
[X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

A pull_step might install its own requirements, which might contain conflicting versions of already installed dependencies (e.g. prefect itself). Version constraints of a pull_step requirement might change even for minor versions (e.g. https://github.com/PrefectHQ/prefect-aws/compare/v0.4.1...v0.4.2, prefect-aws 0.4.1 requires prefect>=2.10.11, prefect-aws 0.4.2 requires prefect>=2.13.5).

This can lead to the situation where:

Prefect version A is installed on machine
pull_step requirements installs Prefect version B while Prefect is running
added functionality from version B is not available in already running version A, leading to import errors or possibly unexpected behavior.

While pinning the requirements versions of a pull step to a specific version solves the issue, I would expect that Prefect checks if the installation of pull_step requirements would overwrite already installed code and refuse to install, instead of running unexpected errors later on.

Reproduction

- install prefect==2.13.4
- create deployment with a pull_step that requires prefect-aws>=0.4.0
- start an agent and run deployment

This shows that in the process of installing prefect-aws>=0.4.0, prefect-aws==0.4.2 is chosen,  installs prefect-2.14.3 and uninstalls prefect-2.13.4.

This causes an error related to some newly introduced functionality that is not yet available in prefect-2.13.4

Error

prefect agent start -q local
Starting v2.13.4 agent connected to https://api.prefect.cloud/api/accounts/...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|

Agent started! Looking for work from queue(s): local...
15:58:37.503 | INFO    | prefect.agent - Submitting flow run 'f498c976-c1cf-41fe-a9fc-1fcd4302fcce'
15:58:38.043 | INFO    | prefect.infrastructure.process - Opening process 'weightless-reindeer'...
15:58:38.279 | INFO    | prefect.agent - Completed submission of flow run 'f498c976-c1cf-41fe-a9fc-1fcd4302fcce'
Collecting prefect-aws>=0.4.0
  Downloading prefect_aws-0.4.2-py3-none-any.whl.metadata (3.7 kB)
  ...
Installing collected packages: ujson, dnspython, email-validator, prefect, prefect-aws
  Attempting uninstall: prefect
    Found existing installation: prefect 2.13.4
    Uninstalling prefect-2.13.4:
      Successfully uninstalled prefect-2.13.4
Successfully installed dnspython-2.4.2 email-validator-2.1.0.post1 prefect-2.14.3 prefect-aws-0.4.2 ujson-5.8.0

15:58:49.127 | ERROR   | Flow run 'weightless-reindeer' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "prefect/deployments/steps/core.py", line 124, in run_steps
  File "prefect/deployments/steps/core.py", line 94, in run_step
    Steps are assumed to be in the format `{"importable.func.name": {"kwarg1": "value1", ...}}`.
  File "prefect/deployments/steps/core.py", line 61, in _get_function_for_step
  File "prefect/utilities/importtools.py", line 212, in import_object
    module = load_module(module_name)
  File "prefect/utilities/importtools.py", line 183, in load_module
    return importlib.import_module(module_name)
  File "python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "prefect_aws/__init__.py", line 5, in <module>
    from .ecs import ECSTask
  File "prefect_aws/ecs.py", line 135, in <module>
    from prefect_aws.workers.ecs_worker import _TAG_REGEX
  File "prefect_aws/workers/__init__.py", line 1, in <module>
    from .ecs_worker import ECSWorker
  File "prefect_aws/workers/ecs_worker.py", line 65, in <module>
    from prefect.workers.base import (
  File "prefect/workers/__init__.py", line 1, in <module>
    from .process import ProcessWorker
  File "prefect/workers/process.py", line 46, in <module>
    from prefect.workers.base import (
  File "prefect/workers/base.py", line 51, in <module>
    from prefect.settings import (
ImportError: cannot import name 'PREFECT_EXPERIMENTAL_WARN_ENHANCED_CANCELLATION' from 'prefect.settings' (prefect/settings.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "prefect/engine.py", line 394, in retrieve_flow_then_begin_flow_run
    client: PrefectClient,
  File "prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "prefect/deployments/deployments.py", line 217, in load_flow_from_flow_run
    storage_document = await client.read_block_document(
  File "prefect/deployments/steps/core.py", line 152, in run_steps
    for warning in w:
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_aws.deployments.steps.pull_from_s3
 > Running pull_from_s3 step...
Unable to load step function: prefect_aws.deployments.steps.pull_from_s3. Attempting install of prefect-aws>=0.4.0.
15:58:50.039 | INFO    | prefect.infrastructure.process - Process 'weightless-reindeer' exited cleanly.


### Versions

```Text
Version:             2.13.4
API version:         0.8.4
Python version:      3.8.15
Git commit:          48d9d9a2
Built:               Fri, Sep 29, 2023 11:26 AM
OS/Arch:             linux/x86_64
Server type:         cloud

Additional context

While the example uses an older Prefect version, I think this is still relevant as it shows the possibility of an unintended upgrade.

facing the same issue on 2.14.9 . added the env variable using prefect config command but still failed

Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps step_func = _get_function_for_step(fqn, requires=keywords.get("requires")) File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 94, in run_step File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 61, in _get_function_for_step File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 212, in import_object module = load_module(module_name) File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 183, in load_module return importlib.import_module(module_name) File "/usr/local/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 992, in _find_and_load_unlocked File "", line 241, in _call_with_frames_removed File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 992, in _find_and_load_unlocked File "", line 241, in _call_with_frames_removed File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/site-packages/prefect_aws/init.py", line 5, in from .ecs import ECSTask File "/usr/local/lib/python3.10/site-packages/prefect_aws/ecs.py", line 135, in from prefect_aws.workers.ecs_worker import _TAG_REGEX File "/usr/local/lib/python3.10/site-packages/prefect_aws/workers/init.py", line 1, in from .ecs_worker import ECSWorker File "/usr/local/lib/python3.10/site-packages/prefect_aws/workers/ecs_worker.py", line 65, in from prefect.workers.base import ( File "/usr/local/lib/python3.10/site-packages/prefect/workers/init.py", line 1, in from .process import ProcessWorker File "/usr/local/lib/python3.10/site-packages/prefect/workers/process.py", line 46, in from prefect.workers.base import ( File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 51, in from prefect.settings import ( ImportError: cannot import name 'PREFECT_EXPERIMENTAL_WARN_ENHANCED_CANCELLATION' from 'prefect.settings' (/usr/local/lib/python3.10/site-packages/prefect/settings.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 394, in retrieve_flow_then_begin_flow_run client: PrefectClient, File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 217, in load_flow_from_flow_run storage_document = await client.read_block_document( File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps category=DeprecationWarning, prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_aws.deployments.steps.pull_from_s3

PrefectHQ / prefect