search

faust-streaming / faust

Python Stream Processing. A Faust fork
https://faust-streaming.github.io/faust/
Other
1.65k stars 183 forks source link

Issues With Starting Worker - Basic Quickstart App #243

Closed Thelin90 closed 2 years ago

Thelin90 commented 2 years ago

Checklist

Kafka

Kafka is running within a strimzi operator in a local k8s cluster, and it works 100% fine, no issues at all to consume either console or from other applications.

Steps to reproduce

poetry run faust -W workers/first/ -A main_app worker -f info

Or

poetry run faust -A main_app worker -f info

Structure

.
├── Makefile
├── README.md
├── workers
│   └── first
│       └── <files be dumped here for first worker>
├── main.py
├── poetry.lock
├── pyproject.toml

Code

import faust

app = faust.App(
    "main_app",
    broker="kafka://localhost:31659",
    value_serializer='raw',
)

yahoo_topic = app.topic("yahoo-finance-quote-stream-api")

@app.agent(yahoo_topic)
async def process(yahoo):
    async for value in yahoo:
        print(value)

if __name__ == '__main__':
    app.main()

Expected behavior

Worker starting.

Actual behavior

For some reason, I keep getting:

ModuleNotFoundError: No module named 'main_app'

Full traceback

Traceback (most recent call last):
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/bin/faust", line 8, in <module>
    sys.exit(cli())
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/lib/python3.9/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/lib/python3.9/site-packages/faust/cli/base.py", line 423, in make_context
    self._maybe_import_app()
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/lib/python3.9/site-packages/faust/cli/base.py", line 388, in _maybe_import_app
    find_app(appstr)
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/lib/python3.9/site-packages/faust/cli/base.py", line 312, in find_app
    val = symbol_by_name(app, imp=imp)
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/lib/python3.9/site-packages/mode/utils/imports.py", line 262, in symbol_by_name
    module = imp(  # type: ignore
  File "/Users/simon/Library/Caches/pypoetry/virtualenvs/seuteullim-m0GyzYC9-py3.9/lib/python3.9/site-packages/mode/utils/imports.py", line 376, in import_from_cwd
    return imp(module, package=package)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'main_app'

Similar issues

https://github.com/robinhood/faust/issues/152

But this is too old to be correlated to my issue.

Versions

Thelin90 commented 2 years ago

I solved the problem by doing:

├── worker
│   ├── __init__.py
│   └── app.py
poetry run faust --datadir=worker-1/ -A main worker -l info

__init__.py

from worker import app

__all__ = ["app"]

app.py

import faust

app = faust.App(
    "main",
    broker="kafka://localhost:31659",
    value_serializer="raw",
)

yahoo_topic = app.topic("yahoo-finance-quote-stream-api")

@app.agent(yahoo_topic)
async def process(yahoo):
    async for value in yahoo:
        print(value)

if __name__ == "__main__":
    app.main()

It feels however strange to me I had to modify the __init__.py.

Ah well I can use it this way...