Netflix / metaflow

Open Source Platform for developing, scaling and deploying serious ML, AI, and data science systems
https://metaflow.org
Apache License 2.0
8.09k stars 762 forks source link

synchronous run and resume functionality + nbrun for runner API #1845

Closed madhur-ob closed 4 months ago

madhur-ob commented 4 months ago

Should only be merged after https://github.com/Netflix/metaflow/pull/1732

This allows us to do stuff like:

    runner = Runner('../try.py', metadata="local")
    result = runner.run(alpha=5, tags=["abc", "def"], max_workers=5)

    while not result.run.finished:
        print('trying again...')
        result = runner.resume()

Also allows us to use run inside environments which already have a running event loop such as Jupyter notebooks.