Debian Bookworm Kernel with Docker/Podman Support

luckylinux commented 8 months ago

Hi,

As already discussed quite a long way in the other Issue https://github.com/inindev/rock-5b/pull/3

I would like you to share the Debian Bookworm Kernel with Docker/Podman Support.

I tried to build the kernel on my side but it does NOT boot (correction: it DOES boot but you require SERIAL access, since you drop to an emergency shell - PCIe/Ethernet/NVMe does NOT work) https://github.com/luckylinux/rock-5b

And previously when I tried to build the complete Debian Bookworm image, it also does NOT boot (correction: it PROBABLY DOES boot but you require SERIAL access, since you drop to an emergency shell - PCIe/Ethernet/NVMe does NOT work) https://github.com/inindev/rock-5b/issues/2

So there is something weird going on. Possibly related to the build toolchain (I am using GCC 12, you are using GCC 13), but apart from that I cannot really say what's wrong.

I am booting from SD card, so that should theoretically minimize some issues (e.g. broken NVMe) - I have a DATA NVMe installed, but the Root partition (/) is on SD Card.

Attached some logs rock5_6.6.0_no_ethernet_no_nvme.log rock5_6.6rc4_ethernet_nvme.log

inindev commented 8 months ago

it PROBABLY DOES boot but you require SERIAL access, since you drop to an emergency shell

The serial console is not required and networking will come up on its own. HDMI video is not supported on the rk3588 or that I have been able to get working. You will need to ssh into the device.

inindev commented 8 months ago

It looks like there are a number of kernel settings that are going to be required:

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_MANGLE: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT3_FS: enabled
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: missing
    enable these ext4 configs if you are using ext4 as backing filesystem
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: missing
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: missing
      - CONFIG_XFRM_USER: missing
      - CONFIG_XFRM_ALGO: missing
      - CONFIG_INET_ESP: missing
      - CONFIG_NETFILTER_XT_MATCH_BPF: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: missing
    - CONFIG_NF_CONNTRACK_FTP: missing
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

luckylinux commented 8 months ago

it PROBABLY DOES boot but you require SERIAL access, since you drop to an emergency shell

The serial console is not required and networking will come up on its own. HDMI video is not supported on the rk3588 or that I have been able to get working. You will need to ssh into the device.

No. Maybe with the kernel YOU built it works.

With the Kernel I built (even scratch - not even with Podman/Docker support) I get dropped to an emergency shell ("Please enter root password or Ctrl +D"). Networking does NOT come up. NVMe does not get detected. lspci returns nothing (PCIe is not detected). That's why I said SERIAL access IS required.

IF Ethernet comes up then of course you can use SSH. But if it doesn't (and it does NOT with the kernel I built, neither with your older Debian Bookworm image) then you are screwed essentially.

Could this be due to a hardware difference ? I got the "early bird" Rock 5B (the one where you paid a 5$ to get a 50$ [?] coupon). Microcode / firmware update needed maybe ?

luckylinux commented 8 months ago

It looks like there are a number of kernel settings that are going to be required:

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_MANGLE: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT3_FS: enabled
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: missing
    enable these ext4 configs if you are using ext4 as backing filesystem
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: missing
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: missing
      - CONFIG_XFRM_USER: missing
      - CONFIG_XFRM_ALGO: missing
      - CONFIG_INET_ESP: missing
      - CONFIG_NETFILTER_XT_MATCH_BPF: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: missing
    - CONFIG_NF_CONNTRACK_FTP: missing
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

Yeah I mentioned those in the related thread. And implemented those in my repository to test.

But unless the kernel boots and Ethernet & NVMe work, it's still useless.

inindev commented 8 months ago

it PROBABLY DOES boot but you require SERIAL access, since you drop to an emergency shell

The serial console is not required and networking will come up on its own. HDMI video is not supported on the rk3588 or that I have been able to get working. You will need to ssh into the device.

No. Maybe with the kernel YOU built it works.

With the Kernel I built (even scratch - not even with Podman/Docker support) I get dropped to an emergency shell ("Please enter root password or Ctrl +D"). Networking does NOT come up. NVMe does not get detected. lspci returns nothing (PCIe is not detected). That's why I said SERIAL access IS required.

IF Ethernet comes up then of course you can use SSH. But if it doesn't (and it does NOT with the kernel I built, neither with your older Debian Bookworm image) then you are screwed essentially.

Could this be due to a hardware difference ? I got the "early bird" Rock 5B (the one where you paid a 5$ to get a 50$ [?] coupon). Microcode / firmware update needed maybe ?

I see, it can't mount the rootfs for some reason. This can be challenging to work out. I would recommend to get the build fully working on removable mmc media before trying to get nvme working.

$ cat /boot/extlinux/extlinux.conf
#
# this is an automatically generated file
# edit options at the top of the /boot/mk_extlinux file
# then run /boot/mk_extlinux to rebuild
#

menu title u-boot menu
prompt 1
default l0
timeout 30

label l0
    menu label Debian GNU/Linux trixie/sid 6.6.0-rc4-1-arm64
    linux /boot/vmlinuz-6.6.0-rc4-1-arm64
    initrd /boot/initrd.img-6.6.0-rc4-1-arm64
    fdt /boot/rk3588-rock-5b.dtb-6.6.0-rc4-1-arm64
    append root=UUID=b8715dd3-c1a0-401d-bea7-2207083d7bf2 ro rootwait

Check the kernel command line for correctness. Rather than mount by uuid, try the device name:

append root=/dev/mmcblk1p1

inindev commented 8 months ago

It can also fail to mount the rootfs due to a bad device tree: /boot/rk3588-rock-5b.dtb-6.6.0-rc4-1-arm64 Or not being able to find it. This is also set in /boot/extlinux/extlinux.conf here:

fdt /boot/rk3588-rock-5b.dtb-6.6.0-rc4-1-arm64

When the kernel builds, it puts it here: /usr/lib/linux-image-6.6.0-rc4-1-arm64/rockchip/rk3588-rock-5b.dtb

luckylinux commented 8 months ago

Yeah, I don't know what's going on to be honest.

It's not your fault.

But it also doesn't feel "scientific" if I cannot replicate the result, you know what I mean ?

The Device tree is present in both places

ls -l /usr/lib/linux-image-6.6.0-1-arm64/rockchip/rk3588-rock-5b.dtb 
-rwxr-xr-x 1 root root 66397 Oct 31 16:49 /usr/lib/linux-image-6.6.0-1-arm64/rockchip/rk3588-rock-5b.dtb

Tried with root=/dev/mmcblk1p1 but no change

ls -l /boot/rk3588-rock-5b.dtb-6.6.0-1-arm64
-rw-r--r-- 1 root root 66397 Oct 31 17:57 /boot/rk3588-rock-5b.dtb-6.6.0-1-arm64

luckylinux commented 8 months ago

For this specific issue, the EXT4 options required by Docker/Podman (e.g. CONFIG_EXT4_FS_SECURITY) might have caused this issue.

But as I said, previously even without those options, I couldn't access via Ethernet (thus I assume it's a common issue).

luckylinux commented 8 months ago

/boot/extlinux/extlinux.conf right now

#
# this is an automatically generated file
# edit options at the top of the /boot/mk_extlinux file
# then run /boot/mk_extlinux to rebuild
#

menu title u-boot menu
prompt 1
default l1
timeout 50

label l0
        menu label Debian GNU/Linux 12 (bookworm) 6.6.0-1-arm64
        linux /boot/vmlinuz-6.6.0-1-arm64
        initrd /boot/initrd.img-6.6.0-1-arm64
        fdt /boot/rk3588-rock-5b.dtb-6.6.0-1-arm64
#       append root=UUID=361f0aec-7ef8-4936-849e-805e18d80916 ro rootwait
        append root=/dev/mmcblk1p1 ro rootwait

label l1
        menu label Debian GNU/Linux 12 (bookworm) 6.6.0-rc4-1-arm64
        linux /boot/vmlinuz-6.6.0-rc4-1-arm64
        initrd /boot/initrd.img-6.6.0-rc4-1-arm64
        fdt /boot/rk3588-rock-5b.dtb-6.6.0-rc4-1-arm64
        append root=UUID=361f0aec-7ef8-4936-849e-805e18d80916 ro rootwait

label l2
        menu label Debian GNU/Linux 12 (bookworm) 6.5.9-1-arm64
        linux /boot/vmlinuz-6.5.9-1-arm64
        initrd /boot/initrd.img-6.5.9-1-arm64
        fdt /boot/rk3588-rock-5b.dtb-6.5.9-1-arm64
        append root=UUID=361f0aec-7ef8-4936-849e-805e18d80916 ro rootwait

inindev commented 8 months ago

I assume you do have a serial console and are selecting boot option l0? I see the default is set to l1.

The uuid of the boot partition is found this way:

sudo blkid -o value -s UUID /dev/nvme0n1p1

or

sudo blkid -o value -s UUID /dev/mmcblk1p1

inindev commented 8 months ago

Also, when you regenerate the boot menu by running /boot/mk_extlinux it will reset the guid from what it finds in /etc/fstab

luckylinux commented 8 months ago

I assume you do have a serial console and are selecting boot option l0? I see the default is set to l1.

The uuid of the boot partition is found this way:
sudo blkid -o value -s UUID /dev/nvme0n1p1

or

sudo blkid -o value -s UUID /dev/mmcblk1p1

Yes. I set the default to something (the only kernel) where Ethernet works. I manually select the first item from the serial console.

root@Rock5B-01:~# sudo blkid -o value -s UUID /dev/mmcblk1p1
361f0aec-7ef8-4936-849e-805e18d80916
root@Rock5B-01:~# sudo blkid -o value -s UUID /dev/nvme0n1p1

(NVMe is not detected since PCIe [and Ethernet] don't work ....)

root@Rock5B-01:~# cat /etc/fstab 
# if editing the device name for the root entry, it is necessary
# to regenerate the extlinux.conf file by running /boot/mk_extlinux

# <device>                                      <mount> <type>  <options>               <dump> <pass>
UUID=361f0aec-7ef8-4936-849e-805e18d80916       /       ext4    errors=remount-ro       0       1

UUID=395f5435-5b82-42e1-b3c7-7b6cc68743a7       /data   ext4    errors=remount-ro       0       2

# Tools over NFS
nas.home:/export/tools          /tools_nfs           nfs             rw,user=tools,auto,nfsvers=3          0       0

# Podman BIND Mounts
/data/podman/CONFIG /home/podman/.config/containers none defaults,rbind 0 0
#/data/podman/STORAGE /home/podman/.local/share/containers/storage none defaults,rbind 0 0
/data/podman/STORAGE /home/podman/storage none defaults,rbind 0 0
/data/podman/DATA /home/podman/data none defaults,rbind 0 0
/data/podman/COMPOSE /home/podman/compose none defaults,rbind 0 0
/data/podman/IMAGES /home/podman/images none defaults,rbind 0 0
/data/podman/VOLUMES /home/podman/volumes none defaults,rbind 0 0
/data/podman/BUILD /home/podman/build none defaults,rbind 0 0

luckylinux commented 8 months ago

Also, when you regenerate the boot menu by running /boot/mk_extlinux it will reset the guid from what it finds in /etc/fstab

I manually changed /boot/extlinux/extlinux.conf. Not much to do around that right now to be honest ...

inindev commented 8 months ago

I see, so you are booting one kernel successfully with basically the same settings as the one you compiled. Can you send a listing of the files in /boot? Do all of the files there look like their sizes are rational too?

luckylinux commented 8 months ago

I see, so you are booting one kernel successfully with basically the same settings as the one you compiled. Can you send a listing of the files in /boot? Do all of the files there look like their sizes are rational too?

root@Rock5B-01:~# ls -la /boot/
total 175452
drwxr-xr-x  3 root root     4096 Nov  1 08:12 .
drwxr-xr-x 20 root root     4096 Oct 29 12:45 ..
-rw-r--r--  1 root root  6783367 Oct 25 10:16 System.map-6.5.9-1-arm64
-rw-r--r--  1 root root  6870666 Oct 31 16:49 System.map-6.6.0-1-arm64
-rw-r--r--  1 root root  6745813 Oct  1 21:15 System.map-6.6.0-rc4-1-arm64
-rw-r--r--  1 root root   285715 Oct 25 10:16 config-6.5.9-1-arm64
-rw-r--r--  1 root root   287856 Oct 31 16:49 config-6.6.0-1-arm64
-rw-r--r--  1 root root   286938 Oct  1 21:15 config-6.6.0-rc4-1-arm64
drwxr-xr-x  2 root root     4096 Nov  5 06:44 extlinux
-rw-r--r--  1 root root 10284664 Nov  1 08:08 initrd.img-6.5.9-1-arm64
-rw-r--r--  1 root root 10355487 Oct 31 17:57 initrd.img-6.6.0-1-arm64
-rw-r--r--  1 root root 10485327 Oct 29 13:19 initrd.img-6.6.0-rc4-1-arm64
-rwxr-xr--  1 root root     2382 Nov  1 08:12 mk_extlinux
-rw-r--r--  1 root root    81347 Nov  1 08:08 rk3588-rock-5b.dtb-6.5.9-1-arm64
-rw-r--r--  1 root root    66397 Oct 31 17:57 rk3588-rock-5b.dtb-6.6.0-1-arm64
-rw-r--r--  1 root root    81413 Oct  2 11:06 rk3588-rock-5b.dtb-6.6.0-rc4-1-arm64
-rw-r--r--  1 root root 42418688 Oct 25 10:16 vmlinuz-6.5.9-1-arm64
-rw-r--r--  1 root root 42924544 Oct 31 16:49 vmlinuz-6.6.0-1-arm64
-rw-r--r--  1 root root 41656832 Oct  1 21:15 vmlinuz-6.6.0-rc4-1-arm64

I do not see any major kernel / initramfs size issue

root@Rock5B-01:~# uname -a
Linux Rock5B-01 6.6.0-1-arm64 #1 SMP PREEMPT Debian 6.6.0-1 (2023-10-31) aarch64 GNU/Linux

Kernel config I cannot send right now (cannot scp into the Rock 5B since Ethernet not working). I posted it in the other thread though previously IIRC.

inindev commented 8 months ago

Did you install the kernel using sudo dpkg -i path_to_kernel.deb?

Check to be sure your modules exist too:

ll /usr/lib/modules
total 8.0K
drwxr-xr-x 3 root root 4.0K Oct  2 22:43 6.5.5-1-arm64
drwxr-xr-x 3 root root 4.0K Oct  2 01:24 6.6.0-rc4-1-arm64

luckylinux commented 8 months ago

EDIT: the config I posted was for 6.5.9 here https://github.com/inindev/rock-5b/files/13224828/kernel-6.5.9.config.txt

I started serial debug logging again for .config for 6.6.0-1 where I just do a zcat /proc/config.gz of it.

rock5_kernel_6.6.0-1_config.log.txt

luckylinux commented 8 months ago

Did you install the kernel using sudo dpkg -i path_to_kernel.deb?

Check to be sure your modules exist too:
ll /usr/lib/modules
total 8.0K
drwxr-xr-x 3 root root 4.0K Oct  2 22:43 6.5.5-1-arm64
drwxr-xr-x 3 root root 4.0K Oct  2 01:24 6.6.0-rc4-1-arm64

I installed all .deb files that were generated using your script using dpkg -i. linux-headers linux-image linux-image (dbg) linux-libc-dev

root@Rock5B-01:/usr/src/rock-5b/kernel/kernel-6.5.9# ll /usr/lib/modules
total 16
drwxr-xr-x 2 root root 4096 Oct 29 12:12 6.1.0-13-arm64
drwxr-xr-x 3 root root 4096 Nov  1 08:08 6.5.9-1-arm64
drwxr-xr-x 3 root root 4096 Oct 31 17:57 6.6.0-1-arm64
drwxr-xr-x 3 root root 4096 Oct 29 11:53 6.6.0-rc4-1-arm64

luckylinux commented 8 months ago

Maybe I should set these as built-in (actually whatever option contains "ROCKCHIP") ?

CONFIG_PCIE_ROCKCHIP_HOST=m
# CONFIG_PCIE_ROCKCHIP_EP is not set

inindev commented 8 months ago

This script? https://github.com/inindev/rock-5b/blob/main/debian/install_kernel.sh#L27

This installs the kernel into an image file. From a booted system you would want to install it from the .deb:

   sudo dpkg -i linux-image-6.6.0-1-arm64_6.6.0-1_arm64.deb

inindev commented 8 months ago

Maybe I should set these as built-in (actually whatever option contains "ROCKCHIP") ?
CONFIG_PCIE_ROCKCHIP_HOST=m
# CONFIG_PCIE_ROCKCHIP_EP is not set

This is from a working kernel:

cat config-6.6.0-rc4-1-arm64 | grep CONFIG_PCIE_ROCKCHIP_HOST
CONFIG_PCIE_ROCKCHIP_HOST=m

luckylinux commented 8 months ago

This script? https://github.com/inindev/rock-5b/blob/main/debian/install_kernel.sh#L27

This installs the kernel into an image file. From a booted system you would want to install it from the .deb:
   sudo dpkg -i linux-image-6.6.0-1-arm64_6.6.0-1_arm64.deb

I just used ./make_kernel.sh and then dpkg -i since this is a running system.

Seing that ./install_kernel.sh installs to a SD card / NVMe / .img file, I didn't see a reason to run that from a live system.

inindev commented 8 months ago

If you want to put your linux-image-6.6.0-1-arm64_6.6.0-1_arm64.deb kernel on google drive and share it, I can just look at it directly.

luckylinux commented 8 months ago

If you want to put your linux-image-6.6.0-1-arm64_6.6.0-1_arm64.deb kernel on google drive and share it, I can just look at it directly.

Unfortunately I don't seem to have the generated binaries anymore. I guess after a lot of frustration I tried to do a "clean start" (with 6.5.9 though), probably related to the multiple overrides of defconfig if you call ./make_kernel.sh several times.

I'll reboot into the working Ethernet kernel (6.6 rc4) and try to rebuild everything for 6.6.0-1.

inindev commented 8 months ago

Check the rock-5b/kernel/kernel-6.6 directory also.

luckylinux commented 8 months ago

Check the rock-5b/kernel/kernel-6.6 directory also.

That's what i meant by clean start. I deleted it ...

luckylinux commented 8 months ago

OK rebuilt .deb files, reinstalled them, same problem.

Where should I put them so that they can be shared ?

MartinHerren commented 7 months ago

Hej,

It looks like there are a number of kernel settings that are going to be required:

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_MANGLE: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT3_FS: enabled
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: missing
    enable these ext4 configs if you are using ext4 as backing filesystem
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: missing
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: missing
      - CONFIG_XFRM_USER: missing
      - CONFIG_XFRM_ALGO: missing
      - CONFIG_INET_ESP: missing
      - CONFIG_NETFILTER_XT_MATCH_BPF: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: missing
    - CONFIG_NF_CONNTRACK_FTP: missing
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

Initially started with the same check-config.sh script to find the missing configs. Used it successfully in the past on some riscv64 with a 5.x kernel, didn't seem to work in this case. Some options it asks are deprecated now, there is still a compatible option but even activating all of them wasn't enough.

Then tried another approach, took the config from the latest Debian 6.5.x aarch64 kernel and tried to make a new defconfig from it. Kernel didn't boot,, no message on uart after 'starting kernel' and didn't find a way to get early boot messages.

Finally ported all networking/netfilter options from Debian's config to the defconfig and finally have a working 6.6.0 kernel with docker working !

My current defconfig

Still would like to investigate what is missing in Debian's config to allow it to boot, or what is added that prevents it from booting.

inindev commented 7 months ago

Nice work! I have not used Docker in a while, and am not really setup to test it. I built a kernel based on your config above. Would you be able to test it to see if I have it right?

linux-image-6.6.1-1-arm64_6.6.1-1_arm64.deb https://drive.google.com/file/d/1w42XmLJFZ3oqwhO46-Q2BZkDPHIq6iAr/view?usp=sharing

luckylinux commented 7 months ago

@MartinHerren Not sure what you mean by "Finally ported all networking/netfilter options from Debian's config to the defconfig". Can you explain your procedure in full? Which Debian's config and which defconfig (the one used by @inindev or just a new make defconfig) ?

We seem to be 3 people getting 3 different results so far.

You @MartinHerren could NOT boot at all, I could boot but NVMe/PCIe/Ethernet didn't work (thus serial access required), @inindev could boot with NVMe/PCIe/Ethernet working.

There must be something off ... Either toolchain version (gcc 13 vs 12 etc), Hardware differences, firmware/microcode, ...

luckylinux commented 7 months ago

@inindev I could give your build a try, but could you please share ALL .deb files generated (4 files in total) ? I usually like to have also the kernel headers installed.

@MartinHerren Did you also try with Kernel 6.5.x version ? I'd also like to have ZFS working, and that is not supported on Kernel 6.6.x (there is a patch on the development GitHub repo though).

MartinHerren commented 7 months ago

linux-image-6.6.1-1-arm64_6.6.1-1_arm64.deb

Thanks, your kernel installs and boots fine from NVMe (bootloader on SPI... still the one from your 6.6.0-rc4 release, will need to update it).

serial console ok
network ok
docker ok
no issue found yet

❯ uname -a
Linux ch-hb9fxx-radxa5b-1.dynv6.net 6.6.1-1-arm64 #1 SMP PREEMPT Debian 6.6.1-1 (2023-11-08) aarch64 GNU/Linux
❯ docker ps -a
CONTAINER ID   IMAGE                                                                                   COMMAND                  CREATED        STATUS         PORTS     NAMES
d052169b6e43   registry.gitlab.com/librespacefoundation/satnogs/satnogs-client/satnogs-client:master   "satnogs-client"         12 hours ago   Up 2 minutes             1868_satnogs_client_1
2c4ea0762f91   registry.gitlab.com/librespacefoundation/docker-hamlib/hamlib:latest                    "/docker-entrypoint.…"   12 hours ago   Up 2 minutes             1868_rigctld_1

@MartinHerren Not sure what you mean by "Finally ported all networking/netfilter options from Debian's config to the defconfig". Can you explain your procedure in full? Which Debian's config and which defconfig (the one used by @inindev or just a new make defconfig) ?

The procedure:

extracted the /boot/config-6.5.0-4-arm64 file from the https://packages.debian.org/sid/linux-image-6.5.0-4-arm64 package
generated a new defconfig out from it and compared it to the one used by @inindev (which is the same as the default one plus 1 module)
looked at the difference and added everyhing that was missing and looked relevant, mostly
- CONFIGNETFILTER*
- CONFIGNFT*
- CONFIGNF*

So for now everything works fine. There are still some options in the config used by Debian that prevent it from booting at all, some others from systemd to start properly. Will still be interesting to identify which ones as the ultimate goal is to run a vanilla Debian with the official kernel. There is still some patching needing to be mainlined and then to trickle down to the Debian kernels.

Which i would know more to help in this process.

luckylinux commented 7 months ago

Well I tried @MartinHerren defconfig with Kernel 6.6.2 compile (with makedefconfig accepting all defaults changes) and it fails to recognize PCIe, Ethernet, etc. defconfig_kernel_6_6_2.txt

luckylinux commented 7 months ago

@inindev I'd be OK to install the image you built, if you can provide ALL .deb files (including linux-headers). I'd particularly like to get ZFS working. Right now they just released OpenZFS 2.2.1 which should support Linux 6.6 series.

Is there a particular procedure to update bootloader, SPI, EEPROM, firmware, etc that I might have missed ?

How did you handle that ?

inindev commented 7 months ago

Here is everything from the build I did on 11/18:

-rw-r--r-- 1 build build 8.2M Nov 18 23:44 linux-headers-6.6.1-1-arm64_6.6.1-1_arm64.deb https://drive.google.com/file/d/1I_zaqnIBj19YBKPupyeHaxoGV9mBeQTN/view?usp=sharing

-rw-r--r-- 1 build build 82M Nov 18 23:46 linux-image-6.6.1-1-arm64-dbg_6.6.1-1_arm64.deb https://drive.google.com/file/d/1GQSeZOwQtDsRWjg_POZzj1hnODzcBCLz/view?usp=sharing

-rw-r--r-- 1 build build 24M Nov 18 23:43 linux-image-6.6.1-1-arm64_6.6.1-1_arm64.deb https://drive.google.com/file/d/1w42XmLJFZ3oqwhO46-Q2BZkDPHIq6iAr/view?usp=sharing

-rw-r--r-- 1 build build 1.3M Nov 18 23:43 linux-libc-dev_6.6.1-1_arm64.deb https://drive.google.com/file/d/1cDdML_lD5L0GuNNPaeepK4w8tixVOW-c/view?usp=sharing

I got busy with the holiday, but I wanted to study the config and test to be sure all of my use cases work before I update the build on the main page. My current thoughts are to provide a bookworm build with a 6.5.x or a 6.6.x kernel and a trixie build with a 6.7-rcx build.

luckylinux commented 7 months ago

Alright, thank you.

I can confirm that it boots, Ethernet works, NVME gets also detected, and lspci output is NOT empty.

@MartinHerren: after applying your defconfig, did you still call confix_fixups (and add the ROCKCHIP SBC kernel module) ? I removed that part as I assumed that it was part of your defconfig file now ...

luckylinux commented 7 months ago

And of course I tried to recompile everything to try to replicate the success/issue.

Partly due to a WARNING during ZFS module compile (kernel provided by @inindev is built using GCC 13, I am using GCC 12 since I am on Debian Bookworm).

Tried with same Kernel 6.6.1 with your config file. Also tried to call config_fixups on top of that, since the first attempt resulted in failure.

Nothing worked when I do the compile.

Might be a GCC 12 or other related Toolchain issue ...

luckylinux commented 7 months ago

I managed to compile the ZFS kernel modules & utils. They should work correctly.

However, since I also want to encrypt the data (BEFORE creating the zpool on it), I tried to use cryptsetup / DM-Crypt / LUKS, but this fails with

device-mapper: reload ioctl on name_crypt (253:0) failed: Invalid argument

This seems to be due to some potentially missing kernel options/modules. https://github.com/t-d-k/LibreCrypt/blob/master/docs/Linux_examples__LUKS.md

Specifically:

zcat /proc/config.gz | grep -i dm_crypt
# CONFIG_DM_CRYPT is not set

zcat /proc/config.gz | grep -i AES
# CONFIG_SND_MAESTRO3 is not set
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_TI is not set
# CONFIG_CRYPTO_AES_ARM64 is not set
CONFIG_CRYPTO_AES_ARM64_CE=y
CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
CONFIG_CRYPTO_AES_ARM64_NEON_BLK=m
CONFIG_CRYPTO_AES_ARM64_BS=m
CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
# CONFIG_CRYPTO_DEV_KEEMBAY_OCS_AES_SM4 is not set
# CONFIG_CRYPTO_DEV_ZYNQMP_AES is not set
CONFIG_CRYPTO_LIB_AES=y

CONFIG_CRYPTO_AES_ARM64 is probably a good idea for AES Acceleration on ARM64 devices such as the Rock 5B.

If somebody wants other ciphers than AES, it is currently not possible, since Kernel support is not enabled

zcat /proc/config.gz | grep -i serpent
# CONFIG_CRYPTO_SERPENT is not set

zcat /proc/config.gz | grep -i twofish
# CONFIG_CRYPTO_TWOFISH is not set

zcat /proc/config.gz | grep -i lrw
# CONFIG_CRYPTO_LRW is not set

While these seems to be OK

zcat /proc/config.gz | grep -i CONFIG_BLK_DEV_DM
CONFIG_BLK_DEV_DM_BUILTIN=y
CONFIG_BLK_DEV_DM=m

zcat /proc/config.gz | grep -i CONFIG_BLK_DEV_LOOP
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8

I just tried to import all modules by "brute force", this is the result and the complaints

modprobe cryptoloop

modprobe aes
modprobe anubis
modprobe arc4
modprobe blkcipher
modprobe blowfish
modprobe cast5
modprobe cast6
modprobe cbc
modprobe crc32c
modprobe crypto_algapi
modprobe crypto_hash
modprobe cryptomgr
modprobe crypto_null
modprobe deflate
modprobe des
modprobe ecb
modprobe gf128mul
modprobe hmac
modprobe khazad
modprobe lrw
modprobe md4
modprobe md5
modprobe michael_mic
modprobe serpent
modprobe sha1
modprobe sha256
modprobe sha512
modprobe tea
modprobe tgr192
modprobe twofish_common
modprobe twofish
modprobe wp512
modprobe xcbc

# dm_mod should give you dm_snapshot, dm_zero and dm_mirror?
modprobe dm_mod
modprobe dm_crypt
modprobe: FATAL: Module cryptoloop not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module anubis not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module arc4 not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module blkcipher not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module blowfish not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module cast5 not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module cast6 not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module khazad not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module lrw not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module md4 not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module serpent not found in directory /lib/modules/6.6.1-1-arm64
modprobe: ERROR: could not insert 'sha512_ce': No such device
modprobe: FATAL: Module tea not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module tgr192 not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module twofish_common not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module twofish not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module wp512 not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module xcbc not found in directory /lib/modules/6.6.1-1-arm64
modprobe: FATAL: Module dm_crypt not found in directory /lib/modules/6.6.1-1-arm64

@inindev Would it be possible for you to rebuild a kernel with these options enabled ? Also v 6.6.2 if possible ... Then I can give it a try and provide some feedback.

I find it weird that these dm-crypt and ciphers are not enabled by default on the kernel / Debian's default ARM64 which @MartinHerren used to generate his defconfig. They are defaults on AMD64 / x86_64 platforms.

luckylinux commented 7 months ago

Just to make sure. In case you got a

patching file arch/arm64/configs/defconfig
Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n]

What did you reply ? I replied with the defaults (n & n).

inindev commented 7 months ago

When I built with the config you linked, I did the following:


cd rock-5b/kernel/kernel-6.6.1/linux-6.6.1
curl https://gitlab.com/-/snippets/3622915/raw/main/docker_defconfig > arch/arm64/configs/docker_defconfig
make docker_defconfig
. . .

luckylinux commented 7 months ago

I always used your make_kernel.sh script, of course modified to handle a custom config file.

I also tried on a second Rock 5B SBC with the Debian Trixie image you provided (had to use the 6.5 image as the 6.6 latest trixie didn't have ethernet working). Anyway... I also tried to compile there with gcc 13 and same config/set of scripts. Got same result. Pcie, ethernet, NVMe NOT working.

Maybe back to the most obvious ... Maybe I need to pull theatest changes you uploaded to your git repository (I saw some commits related to kernel 6.6).

I still have my own separate repo to try a more modular approach, as I dont / didn't want to fiddle too much with the full kernel configure file directly.

By the way, using the commands you mentioned in your last post and my config file, did you get it to boot with ethernet etc working?

luckylinux commented 7 months ago

@inindev: alright, actually it's my own fault for not synchronizing your latest commits to my forked repo.

Now I could compile the Kernel and it boots with Ethernet/PCIe/NVMe support as well as DM_CRYPT.

I tried 6.6.1 but I believe I should also be able to upgrade to later versions

LUKS+Cryptsetup+DM Crypt as well as ZFS Works https://github.com/luckylinux/rock-5b/

(ZFS Requires you to disable SIMD acceleration support for AARCH64/ARM64 though - https://github.com/openzfs/zfs/issues/15401)

luckylinux commented 7 months ago

@inindev Alright, I now also compiled Kernel 6.6.2. Seems to work fine. I also uploaded the kernel binaries here https://github.com/luckylinux/rock-5b

inindev commented 6 months ago

I have published a release with an enhanced array of kernel modules which should look a lot more like a stock debian kernel. https://github.com/inindev/rock-5b/releases/tag/v12-6.7-rc5

The config should also include a superset of the modules you found to make docker work in your use case as well as requests from others.

I would be interested in your feedback if you have time.

luckylinux commented 6 months ago

I have published a release with an enhanced array of kernel modules which should look a lot more like a stock debian kernel. https://github.com/inindev/rock-5b/releases/tag/v12-6.7-rc5

The config should also include a superset of the modules you found to make docker work in your use case as well as requests from others.

I would be interested in your feedback if you have time.

It will take a few days, I just got sick. But I think 6.7 won't work for me, I don't believe it's supported by OpenZFS 2.2.2.

If you can prepare 6.6.x I can for sure give it a try. Also please upload the linux-libc and linux-image-dbg packages though.

Goetterfunke87 commented 6 months ago

I have published a release with an enhanced array of kernel modules which should look a lot more like a stock debian kernel. https://github.com/inindev/rock-5b/releases/tag/v12-6.7-rc5

The config should also include a superset of the modules you found to make docker work in your use case as well as requests from others.

I would be interested in your feedback if you have time.

Hi There, I would like to give some feedback regarding podman: since all things I need seems to work (also I have enough backups) I created a small script to migrate all my pod/containers to your RC7.7. It all worked pretty much flawlessly. Currently USB3 is not working so my appdata is using USB2 with some slowdown but it's enough for me until I can test RC8. Thank you & best Regards

inindev commented 6 months ago

Thanks for the feedback. I am in the process of spliting the linux compile out of my images as it is really just a duplicated effort. The most current kernel is based off of the 6.7 release and should have usb3 working:

https://github.com/inindev/linux-rockchip/releases

I also published all four .debs which is something you had asked about as well.

Goetterfunke87 commented 6 months ago

Thanks for the feedback. I am in the process of spliting the linux compile out of my images as it is really just a duplicated effort. The most current kernel is based off of the 6.7 release and should have usb3 working:

https://github.com/inindev/linux-rockchip/releases

I also published all four .debs which is something you had asked about as well.

You are my hero! Thank you for the hint with the new kernel, with it it's working better but still not perfect. For example a USB3.1 hub will not work: [ 2477.161592] usb 3-1: new high-speed USB device number 7 using ehci-platform [ 2477.321772] usb 3-1: New USB device found, idVendor=2109, idProduct=2822, bcdDevice=90.14 [ 2477.322657] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 2477.323389] usb 3-1: Product: USB2.0 Hub [ 2477.323913] usb 3-1: Manufacturer: VIA Labs, Inc. [ 2477.332616] usb 3-1: SerialNumber: 000000001 [ 2477.353789] hub 3-1:1.0: USB hub found [ 2477.362835] hub 3-1:1.0: 4 ports detected [ 2479.805986] usb usb6-port1: Cannot enable. Maybe the USB cable is bad? [ 2480.773974] usb usb6-port1: Cannot enable. Maybe the USB cable is bad? [ 2481.918167] usb usb6-port1: Cannot enable. Maybe the USB cable is bad?

.. until I disconnect..:

[ 2493.534779] usb 3-1: USB disconnect, device number 7

Also one different USB3.0 hub will only be recognized as USB2. I can however now connect my Touch T7 2TB external drive and it's working in USB3 mode without errors in dmesg. It is not as fast as it should be, only getting around ~300-350MB/s with the official kernel I've got fullspeed on all devices (protocol limited) samba 3.1.1 multichannel is also a bit slower I've used to have full 5Gbps throughput with multichannel enabled (2x2,5gbe) but now only around ~3Gbps. It could be related to power management... cpu usage is also higher than before. Neverthenless, I'm very happy with this image. Thanks again !

inindev commented 5 months ago

But I think 6.7 won't work for me, I don't believe it's supported by OpenZFS 2.2.2.

fwiw: I noticed the linux 6.7 compatibility changes for openzfs have been merged and are in staging: https://github.com/openzfs/zfs/pull/15694

It looks difficult to predict their next release though.

luckylinux commented 5 months ago

But I think 6.7 won't work for me, I don't believe it's supported by OpenZFS 2.2.2.

fwiw: I noticed the linux 6.7 compatibility changes for openzfs have been merged and are in staging: openzfs/zfs#15694

It looks difficult to predict their next release though.

I also saw one issue a few weeks back.

IIRC it was a minor change that was needed.

But I'm holding on until they will have something released and tested. Too many issues already with ZFS 2.2.1 for instance ...

inindev / rock-5b

Debian Bookworm Kernel with Docker/Podman Support #4