`ModuleNotFoundError` on `saq` start

[lxmnk]

Hello! Thank you for nice framework!

I'm trying to use it, but getting an exception on attempt to start worker:

$ saq app.worker.settings
Traceback (most recent call last):
  File "********/saq-test/.venv/bin/saq", line 8, in <module>
    sys.exit(main())
  File "********/saq-test/.venv/lib/python3.10/site-packages/saq/__main__.py", line
 73, in main
    start(
  File "********/saq-test/.venv/lib/python3.10/site-packages/saq/worker.py", line 2
83, in start
    settings = import_settings(settings)
  File "********/saq-test/.venv/lib/python3.10/site-packages/saq/worker.py", line 2
78, in import_settings
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'app'

Project structure:

 ├── app
 │  ├── __init__.py
 │  └── worker.py
 ├── poetry.lock
 └── pyproject.toml

worker.py contents:

from typing import Any, Dict

from saq import Queue

async def startup(_: Dict[str, Any]) -> None:
    print("Hello from startup job!")

settings = {
    "queue": Queue.from_url("redis://localhost"),
    "functions": [],
    "concurrency": 8,
    "startup": startup,
}

If I try to use importlib from interpreter, everything is OK:

>>> import importlib
>>> importlib.import_module("app.worker")
<module 'app.worker' from '********/saq-test/app/worker.py'>
>>> _.settings
{'queue': <saq.queue.Queue object at 0x7f81fcd52740>, 'functions': [], 'concurrency': 8, 'startup': <function startup at 0x7f81fcd681f0>}
>>>

I've found, that current directory not in sys.path and used PYTHONPATH to this fix the problem:

$ PYTHONPATH="$PYTHONPATH:$PWD" saq app.worker.settings
Hello from startup job!

Could you help me to figure out, is this a bug or my fault?

[aprilahijriyan]

I experienced the same thing. When I run the "sample" project it has no issues. But, when I try to implement it into my project it has a problem. So, to run the worker I need to make my own script, it can be seen here https://github.com/getpay-id/getpay-api/blob/main/runworker.py.

@tobymao I've made a PR for this https://github.com/tobymao/saq/pull/27

[barakalon]

Your app module needs to be discoverable by the import system.

Two common ways to do this are:

1. Add the project directory to the PYTHONPATH, like you've done. (I believe the python interpreter might do this by default? Which would explain why that worked)
2. Install your package itself. What does your pyproject.toml file look like? Did you run poetry install?

[tobymao]

yea, this is just how python works. you either need to install your app or add it in python path, it's a bit annoying, but adding some special handling in saq for this sounds a bit dangerous

[lxmnk]

Thank you all for help!

I've figured out that my package is not valid. I forgot about it, because I don't use this package as a library. So I've added packages to pypoject.toml and the problem was gone.

[tool.poetry]
name = "test-bot"
version = "0.1.0"
description = "TODO"
packages = [{ include = "app" }]
authors = []

[phihung]

Maybe the Readme example should be changed to python -m saq examples.simple.settings ? Running as python module (-m) automatically adds the current directory to sys.path.

[tobymao]

sure, wanna make a pr?