Open AMontgomerie opened 1 week ago
hi @AMontgomerie - hmm interesting. I have reproduced but I'm not sure what's causing this to hang
I will say that you should be able to unblock yourself by deferring your imports, something like
# hello.py
from prefect import flow
from prefect.logging import get_run_logger
@flow
def hello():
from azure.ai.ml import MLClient # noqa
from mlflow.tracking import MlflowClient # noqa
get_run_logger().info("hello")
if __name__ == "__main__":
hello()
will try to understand more about what's going on
Thanks for the workaround! It was a little awkward to implement because these imports are inside a utility package that is imported throughout a lot of different flows, but I am able to deploy the flows now at least.
Bug summary
After upgrading from prefect 2 to 3, running
prefect --no-prompt deploy --prefect-file prefect.yaml --all
gets stuck indefinitely. With log level set to debug, the only console output is:After lots of trial and error, I managed to track down the lines of code that are causing it to get stuck. If my flow imports both
azure.ai.ml.MLClient
andmlflow.tracking.MlflowClient
then this issue occurs. If either class is not imported then deployment goes ahead normally.For example:
Version info (
prefect version
output)Additional context
Other dependencies:
I have a local self-hosted Prefect server running inside Docker Desktop Kubernetes, and also a Prefect Cloud account with a work pool that connects to an AKS cluster, and the same issue occurs when attempting to deploy to either.