robinhood / faust

Python Stream Processing
Other
6.7k stars 538 forks source link

Faust shouldn't force an origin and/or re-attempt autodiscovery #737

Open kurtrwall opened 2 years ago

kurtrwall commented 2 years ago

Checklist

Steps to reproduce

I have figured out that the autodiscover feature option of passing a list of modules to find has a bug in the _finalize_concrete_app method. When that method is invoked, the app has already loaded the autodiscovery modules given. More to the bug, that method uses the sys.argv[0] argument as the origin of the App. When the App is loaded again, it has the origin set which triggers the behavior of autodiscovery at the origin. In the following case, that means I cannot narrow down what subpackages I want to load from where the App lives.

  1. Create a package with a module that has a __main__.py that passes a list of modules for the Faust app to autodiscover.

    # directory structure
    # pkg
    # | __init__.py
    # | __main__.py
    # | subpkg1
    # | | __init__.py
    # | | agent.py
    # | | models.py
    # | subpkg2
    # | | __init__.py
    # | | agent.py
    # | | models.py
    
    # __main__.py
    from faust import App
    subpkgs = get_subpkgs()  # returns subpackages to load
    app = App(..., autodiscover=subpkgs)
    app.main()
  2. Invoke the pkg for a specific subpackage to load
    $ python -m pkg --subpkg subpkg1 agents
  3. You should see all the agents from every subpackage print out, indicating it's loading everything.

Expected behavior

I expect that only the subpackages I give to the autodiscover parameter of App will be loaded.

Actual behavior

Everything under pkg is loaded.

Versions

bobh66 commented 2 years ago

This project appears to have been abandoned.

You might want to check out the fork of this project - https://github.com/faust-streaming/faust