search

openwrt / asu

An image on demand server for OpenWrt based distributions
https://sysupgrade.openwrt.org
GNU General Public License v2.0
315 stars 80 forks source link

asu: can't run local server #975

Open efahl opened 4 days ago

efahl commented 4 days ago

Running asu main at 92ee38499d24c08869739a2b2a26ae86c8c56d1c (kept up-to-date with weekly pulls).

Should I need to modify ports or container ACLs or ???

$ git pull
Already up to date.

$ poetry install
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: asu (0.0.0)

$ podman-compose up -d
87888240a54b00b9c2ff2459638f45ca893f44064541ce2937ad77120f53baf8
e54487bd6e5464c6ff9206c7186f74e5c3548e637aba5dc0d5e903c2f0bb3c23
03ec5bc44c3710e2acb9912e86e0235bf90eb8241cf850695d6e7e7401bebeae
1897a16806792df012ed38f7e79543af1a768fd5dd08e6d8335516b092f38e84

$ podman logs asu_redis_1
3:C 16 Sep 2024 20:42:11.045 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3:C 16 Sep 2024 20:42:11.045 * Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=3, just started
3:C 16 Sep 2024 20:42:11.045 * Configuration loaded
3:M 16 Sep 2024 20:42:11.045 * monotonic clock: POSIX clock_gettime
3:M 16 Sep 2024 20:42:11.046 * Running mode=standalone, port=6379.
3:M 16 Sep 2024 20:42:11.046 * Module 'RedisCompat' loaded from /opt/redis-stack/lib/rediscompat.so
3:M 16 Sep 2024 20:42:11.047 * <search> Redis version found by RedisSearch : 7.4.0 - oss
3:M 16 Sep 2024 20:42:11.047 * <search> RediSearch version 2.10.5 (Git=2.10-e2f28a9)
3:M 16 Sep 2024 20:42:11.047 * <search> Low level api version 1 initialized successfully
3:M 16 Sep 2024 20:42:11.047 * <search> gc: ON, prefix min length: 2, min word length to stem: 4, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  10000,
3:M 16 Sep 2024 20:42:11.047 * <search> Initialized thread pools!
3:M 16 Sep 2024 20:42:11.047 * <search> Enabled role change notification
3:M 16 Sep 2024 20:42:11.047 * Module 'search' loaded from /opt/redis-stack/lib/redisearch.so
3:M 16 Sep 2024 20:42:11.048 * <timeseries> RedisTimeSeries version 11202, git_sha=5643fd4d6fcb1e9cf084fb2deb9285b08f4a6672
3:M 16 Sep 2024 20:42:11.048 * <timeseries> Redis version found by RedisTimeSeries : 7.4.0 - oss
3:M 16 Sep 2024 20:42:11.048 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
3:M 16 Sep 2024 20:42:11.048 * <timeseries> loaded server DUPLICATE_POLICY: block
3:M 16 Sep 2024 20:42:11.049 * <timeseries> loaded default IGNORE_MAX_TIME_DIFF: 0
3:M 16 Sep 2024 20:42:11.049 * <timeseries> loaded default IGNORE_MAX_VAL_DIFF: 0.000000
3:M 16 Sep 2024 20:42:11.049 * <timeseries> Setting default series ENCODING to: compressed
3:M 16 Sep 2024 20:42:11.049 * <timeseries> Detected redis oss
3:M 16 Sep 2024 20:42:11.049 * Module 'timeseries' loaded from /opt/redis-stack/lib/redistimeseries.so
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> Created new data type 'ReJSON-RL'
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> version: 20803 git sha: unknown branch: unknown
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> Exported RedisJSON_V1 API
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> Exported RedisJSON_V2 API
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> Exported RedisJSON_V3 API
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> Exported RedisJSON_V4 API
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> Exported RedisJSON_V5 API
3:M 16 Sep 2024 20:42:11.049 * <ReJSON> Enabled diskless replication
3:M 16 Sep 2024 20:42:11.050 * Module 'ReJSON' loaded from /opt/redis-stack/lib/rejson.so
3:M 16 Sep 2024 20:42:11.050 * <search> Acquired RedisJSON_V5 API
3:M 16 Sep 2024 20:42:11.050 * <bf> RedisBloom version 2.8.2 (Git=unknown)
3:M 16 Sep 2024 20:42:11.050 * Module 'bf' loaded from /opt/redis-stack/lib/redisbloom.so
3:M 16 Sep 2024 20:42:11.050 * <redisgears_2> Created new data type 'GearsType'
3:M 16 Sep 2024 20:42:11.050 * <redisgears_2> Detected redis oss
3:M 16 Sep 2024 20:42:11.051 # <redisgears_2> could not initialize RedisAI_InitError

3:M 16 Sep 2024 20:42:11.051 * <redisgears_2> Failed loading RedisAI API.
3:M 16 Sep 2024 20:42:11.051 * <redisgears_2> RedisGears v2.0.20, sha='9b737886bf825fe29ddc2f8da81f73cbe0b4e858', build_type='release', built_for='Linux-ubuntu22.04.x86_64', redis_version:'7.4.0', enterprise:'false'.
3:M 16 Sep 2024 20:42:11.051 * <redisgears_2> Registered backend: js.
3:M 16 Sep 2024 20:42:11.052 * Module 'redisgears_2' loaded from /opt/redis-stack/lib/redisgears.so
3:M 16 Sep 2024 20:42:11.052 * Server initialized
3:M 16 Sep 2024 20:42:11.052 * <search> Loading event starts
3:M 16 Sep 2024 20:42:11.052 * <redisgears_2> Got a loading start event, clear the entire functions data.
3:M 16 Sep 2024 20:42:11.052 * Loading RDB produced by version 7.4.0
3:M 16 Sep 2024 20:42:11.052 * RDB age 317 seconds
3:M 16 Sep 2024 20:42:11.052 * RDB memory usage when created 1.66 Mb
3:M 16 Sep 2024 20:42:11.052 * Done loading RDB, keys loaded: 2, keys expired: 2.
3:M 16 Sep 2024 20:42:11.052 * <search> Loading event ends
3:M 16 Sep 2024 20:42:11.052 * <redisgears_2> Loading finished, re-enable key space notificaitons.
3:M 16 Sep 2024 20:42:11.052 * DB loaded from disk: 0.000 seconds
3:M 16 Sep 2024 20:42:11.052 * Ready to accept connections tcp

$ podman logs asu_worker_1
20:42:11 Worker rq:worker:1a91ab60a51d4501a1863208a360769e started with PID 1, version 1.16.2
20:42:11 Subscribing to channel rq:pubsub:1a91ab60a51d4501a1863208a360769e
20:42:11 *** Listening on default...
20:42:11 Cleaning registries for queue: default
20:42:11 Scheduler for default started with PID 3

$ podman logs asu_server_1
INFO:     Started server process [2]
INFO:     Waiting for application startup.
INFO:root:ASU server starting up
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

$ poetry run python3 misc/update_all_targets.py
Reloading 23.05.4
Reloading 23.05.4/kirkwood/generic
Traceback (most recent call last):
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connection.py", line 196, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connection.py", line 236, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x769a1961ef50>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/v1/update/23.05.4/kirkwood/generic (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x769a1961ef50>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/efahlgren/asu/misc/update_all_targets.py", line 39, in <module>
    reload_all()
  File "/home/efahlgren/asu/misc/update_all_targets.py", line 23, in reload_all
    session.get(
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/v1/update/23.05.4/kirkwood/generic (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x769a1961ef50>: Failed to establish a new connection: [Errno 111] Connection refused'))
efahl commented 4 days ago

After modifying update_all_targets.py to use port 8000:

$ poetry run python3 misc/update_all_targets.py
Reloading 23.05.4
Reloading 23.05.4/kirkwood/generic
Traceback (most recent call last):
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/urllib3/connection.py", line 464, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer

Same for curl:

$ curl -v http://localhost:8000/
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.81.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
efahl commented 3 days ago

I'm now trying to make sure the underlying pieces work, so testing redis access gives this:

$ poetry run python -c 'from asu.util import get_redis_client; c = get_redis_client(); print(c.dbsize())'
Traceback (most recent call last):
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/connection.py", line 277, in connect
    sock = self.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/retry.py", line 62, in call_with_retry
    return do()
           ^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/connection.py", line 278, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
            ^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/connection.py", line 639, in _connect
    raise err
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/connection.py", line 627, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/commands/core.py", line 884, in dbsize
    return self.execute_command("DBSIZE", **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/client.py", line 545, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/connection.py", line 1074, in get_connection
    connection.connect()
  File "/home/efahlgren/asu-venv/lib/python3.11/site-packages/redis/connection.py", line 283, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

Checking on the port gives ambiguous results:

$ podman  logs asu_redis_1
...
3:M 17 Sep 2024 13:41:34.238 * Running mode=standalone, port=6379.

$ nmap -p 6379 localhost
...
PORT     STATE  SERVICE
6379/tcp closed redis

I added an explicit port section to podman-compose.yml, and retry:

$ tail podman-compose.yml
  redis:
    image: "docker.io/redis/redis-stack-server"
    restart: always
    volumes:
      - redis:/data/:rw
    ports:                     <<<--- Added this section
      - "127.0.0.1:6379:6379"  <<<---

# restart pods
$ nmap -p 6379 localhost
PORT     STATE SERVICE
6379/tcp open  redis

$ python -c 'from asu.util import get_redis_client; c = get_redis_client(); print(c.dbsize())'
4
$ python -c 'from asu.util import get_redis_client; c = get_redis_client(); print(c.keys())'
[b'rq:workers', b'rq:workers:default', b'rq:scheduler-lock:default', b'rq:worker:da577feb552443bc8e352d01f3c945ec']
(asu-venv)

So redis appears to be working, but the server is still refusing connections.

$ podman logs asu_server_1
INFO:     Started server process [2]
INFO:     Waiting for application startup.
INFO:root:ASU server starting up
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

$ nmap -p 8000 localhost
PORT     STATE SERVICE
8000/tcp open  http-alt

$ curl -v http://localhost:8000/
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.81.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
efahl commented 2 days ago

@aparcar I've got everything working with a handful of changes, mostly in container config relating to container_sock vs container_host.

One last issue: where are the docker images?

podman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (ghcr.io/openwrt/imagebuilder:x86-64-master: image not known)

I've tried ghcr.io, docker.io and quay.io, all of which give the above error...