`3.0.0rc14` or `3.0.0rc15` starts prefect server on every app start

[Chris-Perkins]

Bug summary

Sometime after 3.0.0rc13, a change was introduced that causes prefect to start a server on every app where it's installed as a package.

I see this in my logs now, which was not present in 3.0.0rc13: 17:08:38.351 | INFO | prefect - Starting server on http://127.0.0.1:8096

Because a prefect server is started

Version info (prefect version output)

# Note: the below may not match what was installed on web server

$ prefect version
Version:             3.0.0rc14
API version:         0.8.4
Python version:      3.11.4
Git commit:          ad86991a
Built:               Thu, Aug 1, 2024 12:38 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         cloud
Pydantic version:    2.8.2

Additional context

This causes an issue as I utilize a decentralized container service which mirrors the packages of our main service, where prefect flows are hosted.

These decentralized containers have the same packages of the main hosting service; they also have prefect installed on them.

However, because a local prefect server is starting, calls from these containers to our main service intercepts calls to our main prefect host, and reaches an error:

Timed out while attempting to connect to ephemeral Prefect API server.

[desertaxle]

Thanks for the issue @Chris-Perkins! Can you share the settings you use on your decentralized container service? Setting PREFECT_API_URL via your profile or an environment variable should prevent the local temporary server from starting, so your settings will help me determine where to look to resolve this issue.

[Chris-Perkins]

Hey @desertaxle! Thanks for helping me out. PREFECT_API_URL is not set on these containers at the moment as we're mid-migration and don't need prefect's task management on them yet.

• Downgrading to 3.0.0rc13 did prevent the local server from starting.

[desertaxle]

Thanks for the info! You can explicitly choose not to start the ad hoc local server by setting PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=False (it's set to True in the default ephemeral profile.) However, that will cause any API calls the client makes to fail since it won't have a server to communicate with.

If I'm understanding correctly, these containers are running functions decorated with @flow, but you don't need to orchestrate or observe those functions yet, so you don't have a API URL configured. Do I have that right?

[Chris-Perkins]

Noted about PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=False! Thanks for the tip. To confirm, was the change to initiate the ephemeral server between rc13 and rc15 intended in this case?

Yes, the containers are calling functions decorated with @flow, but we don't need them to be orchestrated nor is an API URL configured on these containers.

[desertaxle]

Yes, the change to how the ephemeral server runs was intentional. You can look at https://github.com/PrefectHQ/prefect/issues/14716 for more information about the motivation behind that change.

It looks like your flows were running against the previous in-memory API server, but your containers are having trouble starting the subprocess version of the API server. I think there are three options here:

1. You could remove @flow from your functions until you can configure an API URL.
2. You could call .fn() on your @flow decorated functions to call the decorated function directly until you can configure an API URL.
3. We can develop a mock client that allows flows to run in a "test mode" and does not require server connectivity.

The third option interests me the most and could also benefit other users. Let me know what you think will work best for your setup!

[desertaxle]

Hey @Chris-Perkins! Just checking in to see if you're still experiencing this issue.

[pgierz]

Hi @desertaxle,

I have a similar problem. In my case, I also would like to have a flow, but I don't need a server to deal with orchestration (yet). Here is a simplified version of what I am trying:

class Pipeline:
    def __init__(self, *args, name=None, prefectize=True):
        self._steps = args
        self.name = name or randomname.get_name()
        self._db = PipelineDB(self)
        self._workflow_backend = "native"
        if prefectize:
            self._workflow_backend = "prefect"
            self._prefectize_steps()

    def _prefectize_steps(self):
        # Turn all steps into Prefect tasks:
        prefect_tasks = []
        for i, step in enumerate(self._steps):
            logger.debug(
                f"[{i}/{len(self._steps)}] Converting step {step.__name__} to Prefect task."
            )
            prefect_tasks.append(Task(step))
        self._steps = prefect_tasks

    def run(self, data, rule_spec):
        if self._workflow_backend == "native":
            return self._run_native(data, rule_spec)
        elif self._workflow_backend == "prefect":
            return self._run_prefect(data, rule_spec)
        else:
            raise ValueError("Invalid workflow backend!")

    def _run_native(self, data, rule_spec):
        for step in self.steps:
            with self._db as db:
                if db.read(step).get("status") == "done":
                    continue
                else:
                    self._start_step(step)
                    data = step(data, rule_spec)
                    self._end_step(step)
        return data

    @flow(task_runner=DaskTaskRunner)
    def _run_prefect(self, data, rule_spec):
        return self._run_native(data, rule_spec)

I also get notes in my logs about starting a temporary server and not being able to communicate. Any ideas of how I can proceed?

[Chris-Perkins]

Hey! Thanks for pinging. Setting PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=False ended up resolving my issue, and so I don't have an opinion on any of the three options provided.

[Chris-Perkins]

@pgierz you'll likely benefit from setting the same environment variable.

Thank you for the help here @desertaxle