A new kernel breaks image builder images

[fantom-x]

I understand that 22.03 (SNAPHOT) images are built once week, but they periodically break for up to seven days when a kernel is refreshed: below is an attempt to generate a 22.03-SNAPSHOT image right now. The issue will resolve itself in five days or so when a new image builder image is generated.

Is it possible to refresh these images each time a new kernel is built?

Configuring arp-scan.
Configuring collectd-mod-sqm.
Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-crypto-md5
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for iputils-ping:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package iputils-ping.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-i2c-smbus
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-i2c-i801
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-i2c-i801:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-i2c-i801.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-input-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nls-base
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-usb-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-usb-hid
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-usb-hid:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-usb-hid.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-libphy
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-usb-net
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-usb-net-asix-ax88179
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-usb-net-asix-ax88179:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-usb-net-asix-ax88179.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-usb-net-cdc-ether
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-usb-net-rtl8152
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-usb-net-rtl8152:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-usb-net-rtl8152.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-ikconfig
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-veth
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-sched-cake
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-ifb
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nf-reject
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nf-ipt
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nf-log
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-ipt-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nf-reject6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nf-log6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-lib-crc32c
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nft-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-ipt-ipopt
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for luci-app-sqm:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package luci-app-sqm.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nft-offload
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for luci-ssl:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package luci-ssl.
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for luci-app-nlbwmon:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package luci-app-nlbwmon.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-fs-exfat
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-lib-crc16
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-fs-ext4
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-fs-ext4:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-fs-ext4.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nls-cp437
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nls-iso8859-1
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-nls-utf8
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-fs-vfat
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-fs-vfat:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-fs-vfat.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-usb-storage
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-usb-storage:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-usb-storage.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-amazon-ena
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-ptp
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-mdio-devres
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-amd-xgbe
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-amd-xgbe:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-amd-xgbe.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-bnx2
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-button-hotplug
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-e1000
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-e1000e
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-forcedeth
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-hwmon-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-igb
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-igb:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-igb.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-igc
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-mdio
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-ixgbe
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-ixgbe:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-ixgbe.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-r8169
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-r8169:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package kmod-r8169.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-tg3
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74) for kmod-lib-crc-ccitt
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for ppp:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package ppp.
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for ppp-mod-pppoe:
 *  kernel (= 5.10.194-1-7865481e154fa056bd3b6ca1ae9b5f74)
 * opkg_install_cmd: Cannot install package ppp-mod-pppoe.
make[2]: *** [Makefile:172: package_install] Error 255
make[1]: *** [Makefile:127: _call_image] Error 2
make: *** [Makefile:245: image] Error 2

[aparcar]

I'm guessing that's very hard to implement. A PR on GitHub would have to inform the Buildbots to then notify GitHub actions to build something.

[fantom-x]

Yeah, I realize that. My workaround is to just delete all files and folders under /builder/ , download the latest IB file, untar it, then run make. Works very well for me. Could the official image also download the latest file?

[aparcar]

Yea I thought about that, too. At least for snapshot a script inside the image builder could just download the latest image builder an unpack it. Mind creating a PR?

[fantom-x]

I could do a PR, but I am not sure where to get all the info to generate a URL to an appropriate IB file. Any advice?

BTW, why only snapshots? Even stable releases could download the IB tar file: it does not change.

[fantom-x]

Would not it be simpler to generate images every second day like in master?

[aparcar]

High resource usage...

[fantom-x]

Then there could be just one Docker image for all releases (or one per a release) and it downloads the latest IB tar. This will save even more resources: this image will not have to be rebuilt over and over...

[aparcar]

for all snapshots, yea. I think that could be a viable solution for snapshots as well as nightly builds of releases.

[aparcar]

import requests
from time import sleep
from os import getenv

dispatch_url = "https://api.github.com/repos/openwrt/docker/actions/workflows/containers.yml/dispatches"
last_completed_at = 1696021746

def webhook(ref, target):
    req = requests.post(
        dispatch_url,
        headers={
            "Accept": "application/vnd.github.v3+json",
            "X-GitHub-Api-Version": "2022-11-28",
            "Authorization": f"Bearer {getenv('GITHUB_TOKEN')}",
        },
        json={
            "ref": "main",
            "inputs": {
                "ref": ref,
                "target": target,
            },
        },
    )
    req.raise_for_status()
    sleep(2)

while True:
    builds = requests.get(
        "https://buildbot.staging.openwrt.org/images/api/v2/builds?complete=true&limit=20&order=-buildid"
    ).json()["builds"]

    for build in reversed(builds):
        properties = requests.get(
            f"https://buildbot.staging.openwrt.org/images/api/v2/builds/{build['buildid']}/properties"
        ).json()["properties"][0]

        if "tag" in properties:
            ref = properties["tag"][0]
        else:
            ref = properties["branch"][0]

        target = properties["buildername"][0].split("_", maxsplit=1)[1]

        if build.get("state_string") == "build successful" and target != "force_build":
            completed_at = build["complete_at"]

            if completed_at > last_completed_at:
                print(
                    f"New successful build found for {target} on {ref} at {completed_at}"
                )
                webhook(ref, target)
                last_completed_at = completed_at

    sleep(60)

This script checks every 60 seconds the last builds and triggers docker rebuilds in case something finished. I'll deploy this script until Buildbot web hooks are available.

[fantom-x]

If the first build in builds completes at 1400, then you start a docker build and save 1400 to last_completed_at. Then you go to the second build and it happens to complete at 1355: this will be skipped because it is less than last_completed_at. I think you need a dictionary for last_completed_at...

[aparcar]

Yes I've figured this issue and switch to completed_at only. See the PR. Thanks for reviewing :)!

[fantom-x]

Just looked at #123 and it still the code like above: did you not push your latest changes?