Deploy flow from Jupyter Notebook

[zelterNN]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar request and didn't find it.
• [X] I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the proposed behavior

In Prefect 1.0, I can use the Jupyter Notebook to draft the code and test each portion of it, then directly deploy the code in python using flow.register()

Would like to see a solution similar to that which allows me to construct flow and deloy all in one place.

Describe the current behavior

In Prefect 2.0, the Deployment.build_from_flow() in Jupyter Notebook will enter into the async context, which causes issue in the further apply(). I have no choice but to run the script in .py or to use CLI for deployment.

Example Use

No response

Additional context

No response

[zanieb]

Hi! Can you please include details of the issue you've encountered?

[zelterNN]

Morning Michael,

It's related to a thread I posted in Prefect Slack yesterday: https://prefect-community.slack.com/archives/CL09KU1K7/p1661379434885579

So I test a very simply flow that just return a fix value, and when I try to deploy using build_from_flow() method, it seems the object is entered in async context, which prevent me further deploy it in apply(). [image: image.png] One of your team members suggests me to add await before build_from_flow(), but that doesn't work.

Hope this gives enough context ;)

Best, Jason

Michael Adkins @.***> 于2022年8月25日周四 09:53写道：

Hi! Can you please include details of the issue you've encountered?

— Reply to this email directly, view it on GitHub https://github.com/PrefectHQ/prefect/issues/6568#issuecomment-1227525342, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2MR7BL7UBX7VCJWN2M3YRDV26QJJANCNFSM57TZE4MQ . You are receiving this because you authored the thread.Message ID: @.***>

[zanieb]

Thanks!

A few internal notes:

• Jupyter notebook support seems important, @billpalombi can you ensure it's on the roadmap?
• We need MREs of the bugs encountered when using a Jupyter Notebook in two cases
  1. Run a flow and report its status to an external API
  2. Create a deployment from a flow written in the notebook @anna-geller / @desertaxle perhaps you can lead these?
• We then need a design for creating a deployment from notebooks, this likely will require pickle. @abrookins want to make sure this is on your radar.

[anna-geller]

thanks Michael, I'll discuss with Bill how we can best support notebooks

[rjhawar]

@anna-geller Might I add something here there is a python library that already enables this kind of functionality: https://papermill.readthedocs.io/en/latest/

This is similar to the route taken by dagster. Not sure if that is helpful.

[anna-geller]

thanks so much @rjhawar - we'll definitely consider Papermill collection to run Jupyter notebooks - this here was slightly different question but your point is still valid and helpful

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

[knl]

I've also encountered this issue. Here is a demonstration of the problem: https://github.com/knl/prefect_notebook_runner. It matches exactly the problem described by @zelterNN (and its the last point in @madkinsz summary) -- I do not get the deployment in prefect if I call Deployment.build_from_flow(..., apply=True) from a jupyterhub notebook. Even with all possible awaits, I do get Deployment UUID, but no deployment nor flow in the UI. The same code works from the command line.