Migration to new server

[aparcar]

Hi all, the service moved to a new server, which is faster and hopefully more stable - looking at you non responding Podman.

Please enjoy and report issues here. I'm sure there are still issues with SNAPSHOT being delayed, this is something I'll tackle in the near future or maybe @efahl much sooner?

Sunshine, Paul

[efahl]

Yup, I've got a local ASU server running pretty solid, and am running my owut tests against it as we speak, but still can't get the build request to work (can't find docker imagebuilder images https://github.com/openwrt/asu/issues/975#issuecomment-2357390013).

[efahl]

I'm getting HTTP/1.1 502 Bad Gateway from any attempt to access the new server. Looks like the IP has been update to the right one, I think?

$ nslookup sysupgrade.openwrt.org
Server:         127.0.0.1
Address:        127.0.0.1:53

Non-authoritative answer:
sysupgrade.openwrt.org  canonical name = asu-02.infra.openwrt.org
Name:   asu-02.infra.openwrt.org
Address: 45.140.183.87

Non-authoritative answer:
sysupgrade.openwrt.org  canonical name = asu-02.infra.openwrt.org
Name:   asu-02.infra.openwrt.org
Address: 2001:678:6e1:1001:be24:11ff:fe23:4c6d

[efahl]

Here's some more output, in case it helps debug the issue.

$ wget -vSO - https://sysupgrade.openwrt.org/json/v1/overview.json
--2024-09-18 14:15:22--  https://sysupgrade.openwrt.org/json/v1/overview.json
Resolving sysupgrade.openwrt.org... 2001:678:6e1:1001:be24:11ff:fe23:4c6d, 45.140.183.87
Connecting to sysupgrade.openwrt.org|2001:678:6e1:1001:be24:11ff:fe23:4c6d|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 502 Bad Gateway
  Access-Control-Allow-Credentials: true
  Access-Control-Allow-Headers: *
  Access-Control-Allow-Methods: POST, GET, OPTIONS
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Authorization
  Alt-Svc: h3=":443"; ma=2592000
  Server: Caddy
  Vary: Origin
  Date: Wed, 18 Sep 2024 21:15:23 GMT
  Content-Length: 0
2024-09-18 14:15:23 ERROR 502: Bad Gateway.

[lorand-horvath]

@aparcar Thanks for the heads up, hopefully you'll get the new server to work. https://sysupgrade.openwrt.org/api/v1/build still returns 502, ASU and firmware-selector are both dead.

[aparcar]

wget -vSO - https://sysupgrade.openwrt.org/json/v1/overview.json

I can't reproduce this, I'm seeing a fine JSON file

[aparcar]

@aparcar Thanks for the heads up, hopefully you'll get the new server to work. https://sysupgrade.openwrt.org/api/v1/build still returns 502, ASU and firmware-selector are both dead.

Same here, I tested with multiple browsers and things work fine.

[alucryd]

Confirmed working now, it wasn't a couple hours ago.

[aparcar]

Yea but it's my bad after all, I did not enable "linger" in loginctl and therefore the service would go down shortly after logging out of the user account. Should work now even if I'm not logged into the server, what a stupid thing...

[alucryd]

Still having an issue, but I'm not sure it's related. I'd like to switch my newly-acquired GL-MT6000 to a snapshot build using auc -b snapshot, but I'm getting the following:

root@gl-mt6000:~# auc -b snapshot
auc/0.3.2-1
Server:    https://sysupgrade.openwrt.org
Running:   23.05.4 r24012-d8dd03c46f on mediatek/filogic (glinet,gl-mt6000)
Invalid argument (22)

No error when targeting the 23.05.4 branch.

[efahl]

root@gl-mt6000:~# auc -b snapshot

Try again, but use auc -b snapshot -B snapshot and see if the problem persists. I seem to recall you need to specify both branch and rev sometimes with auc.

[alucryd]

auc -b snapshot -B snapshot

Awesome, that did the trick, thank you!

[divinehawk]

Not faster. My last upgrade a few months ago there were only a handful of requests in the queue, now I see reports of over 500/600 and now 700 in the queue.

[aparcar]

Not faster. My last upgrade a few months ago there were only a handful of requests in the queue, now I see reports of over 500/600 and now 700 in the queue.

Storage runs full and build requests sum up

[efahl]

Storage runs full

Since redis offers no means for detecting expired jobs with a nice callback, I did this and added it to a cron on my home test server. It simply grabs all the active job hashes and deletes the storage when the job no longer exists... Is there something like this running on the production server?

from asu.util import get_redis_client
from pathlib import Path
from asu.config import settings
from shutil import rmtree

redis_client = get_redis_client()
active_hashes = {key.split(":")[-1] for key in redis_client.keys("rq:results:*")}
print(f"{active_hashes = }")

store = settings.public_path / "store"
for dir in store.glob("*"):
    # We check length as an added precaution against doing something stupid.
    # Currently all build hashes are 32 long, see util.py for details.
    if len(dir.name) == 32 and dir.name not in active_hashes:
        print(f"Delete it {dir = !s}")
        rmtree(dir)
    else:
        print(f"Keep it {dir = !s}")