freddyzinsou commented 1 year ago

Checklist

[x] I have included information about relevant versions
[x] I have verified that the issue persists when using the master branch of Faust.

Steps to reproduce

I have installed the latest faust-streaming package and conf faust with autodiscovery set to True

import faust
os.environ.setdefault('FAUST_LOOP', 'eventlet')

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = faust.App('identity-worker', autodiscover=True, origin='worker', key_serializer='json',
                value_serializer='json', topic_disable_leader=True, store="rocksdb://", broker="kafka:9092")

And run my worker with python -m worker.app worker -l info. And it get this and I seems that faust try to load some python package files and my migrations in addition to file which should be loaded

Expected behavior

It should load only the faust tagged file, ie files containing agent decorator and others, not my migrations or others packages files

Actual behavior

I'm trying to migrate from faust which is not maintained no more to faust-streaming. I did the configuration properly and when i run my faust worker it seems that the venusian package detect either my django migration files and try to load it. With this my application return some warning followed by non-blocking error for each migration file i have in my project

Full traceback

[2023-05-05 15:58:44,838] [397891] [WARNING] Autodiscovery importing module 'rest_framework.authtoken.admin' raised error: ImproperlyConfigured('The model TokenProxy is abstract, so it cannot be registered with admin.') 
Traceback (most recent call last):
  File "/home/user/service/venv/lib/python3.8/site-packages/venusian/__init__.py", line 220, in scan
    __import__(modname)
  File "/home/user/service/venv/lib/python3.8/site-packages/rest_framework/authtoken/admin.py", line 51, in <module>
    admin.site.register(TokenProxy, TokenAdmin)

[2023-05-05 15:58:44,848] [397891] [WARNING] Autodiscovery importing module 'identity.migrations.0001_initial' raised error: ModuleNotFoundError("No module named 'phone_field'") 
Traceback (most recent call last):
  File "/home/user/service/venv/lib/python3.8/site-packages/venusian/__init__.py", line 220, in scan
    __import__(modname)
  File "/home/user/service/identity/migrations/0001_initial.py", line 6, in <module>
    import phone_field.models

Versions

Python version 3.8
Faust version 0.10.12
Operating system Ubuntu 20.04
Kafka version 7.4.0
RocksDB version (if applicable)

SiDChik commented 1 year ago

if you really need autodiscover you can pass a list of packages like: autodiscover=['package1', 'package2']

this will ignore your django migrations

freddyzinsou commented 1 year ago

I forgot to mention this, but have already try it, and it got the same issue

munkeycigar commented 1 year ago

any solutions to this problem? why does is it still scanning everything in INSTALLED_APPS, even if you specify a list of packages to scan?

munkeycigar commented 1 year ago

it looks like an issue in the discover() method here.

    for fixup in self.fixups:
        modules |= set(fixup.autodiscover_modules())

the call to django fixup's autodiscover_modules() returns everything in INSTALLED_APPS and the logic in discover() needs to consider that if self.conf.autodiscover is a List, then don't union the modules returned from the fixup?

faust-streaming / faust