mosquitto-ssl - Segmentation fault on subscribe - Githubissues

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md

GNU General Public License v2.0

4.03k stars 3.5k forks source link

mosquitto-ssl - Segmentation fault on subscribe #22977

Closed ASerbinski closed 11 months ago

ASerbinski commented 11 months ago

The following config works perfectly on mosquitto 1.6.15 (Rocky Linux 8) and mosquitto 2.0.14 (Fedora), but results in frequent Segmentation fault on OpenWRT 23.05.2;

max_inflight_messages 1

listener 8883
cafile /etc/mosquitto/ca.crt
certfile /etc/mosquitto/dom.crt
keyfile /etc/mosquitto/dom.key

require_certificate true

log_dest syslog

log_type error
log_type warning
log_type notice
log_type information
log_type all

allow_anonymous false
password_file /etc/mosquitto/pwfile

The segfault seems to always happen when the client sends a SUBSCRIBE;

Sat Dec 23 14:14:19 2023 daemon.debug mosquitto[12688]: 1703358859: Received SUBSCRIBE from {redacted}
Sat Dec 23 14:14:19 2023 daemon.debug mosquitto[12688]: 1703358859:     prop1/thermostat/furnace/log-ac1/$format (QoS 1)

The SUBSCRIBES are sent while the client is connecting. It does some wildcard and recursive subscriptions, and somewhere in the process it often crashes.

I'm not sure how to debug this, since segmentation fault doesn't really give much to go on. Its also not going to be an upstream bug since the same configuration works on other machines without the segmentation fault.

brada4 commented 11 months ago

backtrace or at least a kernel message of the crash?

brada4 commented 11 months ago

(and which architecture, and versions of all packages in backtrace)

ASerbinski commented 11 months ago

I'll need instructions on how to collect those. Nothing is appearing in the kernel or system logs with respect to the crash, and if I start it from terminal, it outputs simply "Segmentation fault".

brada4 commented 11 months ago

kernel log via dmesg command shows segfaults, including disk file and offset in file it happened.

ASerbinski commented 11 months ago

Nothing there. And wouldn't that normally be just for crashes within the kernel?

Here's the log anyway;

$ ssh -l root 192.168.1.1
root@192.168.1.1's password: 

BusyBox v1.36.1 (2023-11-14 13:38:11 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 23.05.2, r23630-842932a63d
 -----------------------------------------------------
root@Avenue-Main2:~# dmesg
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.15.137 (builder@buildhost) (aarch64-openwrt-linux-musl-gcc (OpenWrt GCC 12.3.0 r23630-842932a63d) 12.3.0, GNU ld (GNU Binutils) 2.40.0) #0 SMP Tue Nov 14 13:38:11 2023
[    0.000000] Machine model: Dynalink DL-WRX36
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x0000000040ffffff]
[    0.000000]   node   0: [mem 0x0000000041000000-0x000000004a3fffff]
[    0.000000]   node   0: [mem 0x000000004a400000-0x00000000510fffff]
[    0.000000]   node   0: [mem 0x0000000051100000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.0
[    0.000000] percpu: Embedded 17 pages/cpu s30808 r8192 d30632 u69632
[    0.000000] pcpu-alloc: s30808 r8192 d30632 u69632 alloc=17*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048
[    0.000000] Kernel command line: console=ttyMSM0,115200n8 ubi.mtd=rootfs rootfstype=squashfs rootwait root=/dev/ubiblock0_1
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 888452K/1048576K available (7808K kernel code, 860K rwdata, 1400K rodata, 1280K init, 289K bss, 160124K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] Root IRQ handler: 0xffffffc00837cdb0
[    0.000000] GICv2m: range[mem 0x0b00a000-0x0b00affc], SPI[448:479]
[    0.000000] arch_timer: cp15 and mmio timer(s) running at 19.20MHz (virt/virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[    0.000001] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[    0.000114] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000)
[    0.000131] pid_max: default: 32768 minimum: 301
[    0.000599] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.000616] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.002576] rcu: Hierarchical SRCU implementation.
[    0.003335] smp: Bringing up secondary CPUs ...
[    0.003958] Detected VIPT I-cache on CPU1
[    0.004021] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.004634] Detected VIPT I-cache on CPU2
[    0.004673] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[    0.005281] Detected VIPT I-cache on CPU3
[    0.005318] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[    0.005384] smp: Brought up 1 node, 4 CPUs
[    0.005411] SMP: Total of 4 processors activated.
[    0.005419] CPU features: detected: 32-bit EL0 Support
[    0.005426] CPU features: detected: CRC32 instructions
[    0.005479] CPU features: emulated: Privileged Access Never (PAN) using TTBR0_EL1 switching
[    0.005488] CPU: All CPU(s) started at EL1
[    0.005510] alternatives: patching kernel code
[    0.015341] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.015376] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.015605] pinctrl core: initialized pinctrl subsystem
[    0.017036] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.017552] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.017595] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.017632] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.017998] thermal_sys: Registered thermal governor 'step_wise'
[    0.019691] cpuidle: using governor menu
[    0.019934] ASID allocator initialised with 65536 entries
[    0.047051] qcom,cpr4-apss-regulator b018000.cpr4-ctrl: CPR valid fuse count: 4
[    0.056668] SCSI subsystem initialized
[    0.056873] usbcore: registered new interface driver usbfs
[    0.056914] usbcore: registered new interface driver hub
[    0.056950] usbcore: registered new device driver usb
[    0.057359] qcom_scm: convention: smc arm 64
[    0.058857] clocksource: Switched to clocksource arch_sys_counter
[    0.059805] NET: Registered PF_INET protocol family
[    0.059954] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.060973] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[    0.061004] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.061020] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.061106] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
[    0.061255] TCP: Hash tables configured (established 8192 bind 8192)
[    0.061350] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.061391] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.061601] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.061645] PCI: CLS 0 bytes, default 64
[    0.062874] workingset: timestamp_bits=46 max_order=18 bucket_order=0
[    0.067538] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.067560] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.068571] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[    0.069698] qcom-qmp-phy 78000.phy: supply vdda-phy not found, using dummy regulator
[    0.069869] qcom-qmp-phy 78000.phy: supply vdda-pll not found, using dummy regulator
[    0.070371] qcom-qmp-phy 78000.phy: Registered Qcom-QMP phy
[    0.071264] qcom-qusb2-phy 79000.phy: supply vdda-pll not found, using dummy regulator
[    0.071393] qcom-qusb2-phy 79000.phy: supply vdda-phy-dpdm not found, using dummy regulator
[    0.071583] qcom-qusb2-phy 79000.phy: Registered Qcom-QUSB2 phy
[    0.074385] bam-dma-engine 704000.dma-controller: num-channels unspecified in dt
[    0.074410] bam-dma-engine 704000.dma-controller: num-ees unspecified in dt
[    0.078205] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.080021] msm_serial 78b3000.serial: msm_serial: detected port #0
[    0.080057] msm_serial 78b3000.serial: uartclk = 3686400
[    0.080096] 78b3000.serial: ttyMSM0 at MMIO 0x78b3000 (irq = 22, base_baud = 230400) is a MSM
[    0.080124] msm_serial: console setup on port #0
[    0.709890] printk: console [ttyMSM0] enabled
[    0.715018] msm_serial: driver initialized
[    0.723378] loop: module loaded
[    0.723449] Loading iSCSI transport class v2.0-870.
[    0.727722] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xaa
[    0.730728] nand: Micron MT29F2G08ABBGAH4
[    0.737300] nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 128
[    0.742095] spmi spmi-0: PMIC arbiter version v2 (0x20010000)
[    0.751807] s1: supplied by regulator-dummy
[    0.754791] s2: supplied by regulator-dummy
[    0.758716] vdd_s3: supplied by regulator-dummy
[    0.763094] vdd_s4: supplied by regulator-dummy
[    0.767588] s5: supplied by regulator-dummy
[    0.771950] l1: supplied by regulator-dummy
[    0.776144] l2: supplied by regulator-dummy
[    0.780293] l3: supplied by regulator-dummy
[    0.784430] l4: supplied by regulator-dummy
[    0.788630] l5: supplied by regulator-dummy
[    0.792804] l6: supplied by regulator-dummy
[    0.796932] l7: supplied by regulator-dummy
[    0.801151] l8: supplied by regulator-dummy
[    0.805302] l9: supplied by regulator-dummy
[    0.809564] l11: supplied by regulator-dummy
[    0.813650] l12: supplied by regulator-dummy
[    0.818100] l13: supplied by regulator-dummy
[    0.837337] i2c_dev: i2c /dev entries driver
[    0.841772] sdhci: Secure Digital Host Controller Interface driver
[    0.841818] sdhci: Copyright(c) Pierre Ossman
[    0.846854] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.854702] remoteproc remoteproc0: releasing cd00000.q6v5_wcss
[    0.859743] NET: Registered PF_INET6 protocol family
[    0.863604] Segment Routing with IPv6
[    0.867893] In-situ OAM (IOAM) with IPv6
[    0.871532] NET: Registered PF_PACKET protocol family
[    0.875449] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    0.880504] 8021q: 802.1Q VLAN Support v1.8
[    0.911522] qcom,cpr4-apss-regulator b018000.cpr4-ctrl: CPR valid fuse count: 4
[    0.911835] cpr4_ipq807x_apss_read_fuse_data: apc_corner: speed bin = 0
[    0.917648] cpr4_ipq807x_apss_read_fuse_data: apc_corner: CPR fusing revision = 1
[    0.924298] cpr4_ipq807x_apss_read_fuse_data: apc_corner: CPR misc fuse value = 0
[    0.931916] cpr4_ipq807x_apss_read_fuse_data: apc_corner: Voltage boost fuse config = 0 boost = disable
[    0.939421] cpr3_mem_acc_init: apc: not using memory accelerator regulator
[    0.948550] cpr4_ipq807x_apss_calculate_open_loop_voltages: apc_corner: fused      SVS: open-loop= 704000 uV
[    0.955507] cpr4_ipq807x_apss_calculate_open_loop_voltages: apc_corner: fused      NOM: open-loop= 832000 uV
[    0.965490] cpr4_ipq807x_apss_calculate_open_loop_voltages: apc_corner: fused    TURBO: open-loop= 896000 uV
[    0.975297] cpr4_ipq807x_apss_calculate_open_loop_voltages: apc_corner: fused   STURBO: open-loop= 992000 uV
[    0.985157] cpr4_ipq807x_apss_calculate_target_quotients: apc_corner: fused      SVS: quot[ 7]= 683, quot_offset[ 7]=   0
[    0.994918] cpr4_ipq807x_apss_calculate_target_quotients: apc_corner: fused      NOM: quot[ 7]= 920, quot_offset[ 7]= 235
[    1.005768] cpr4_ipq807x_apss_calculate_target_quotients: apc_corner: fused    TURBO: quot[ 7]=1025, quot_offset[ 7]= 105
[    1.016706] cpr4_ipq807x_apss_calculate_target_quotients: apc_corner: fused   STURBO: quot[ 7]=1189, quot_offset[ 7]= 160
[    1.027772] cpr3_regulator_init_ctrl: apc: Default CPR mode = closed-loop
[    1.030164] nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xaa
[    1.045367] nand: Micron MT29F2G08ABBGAH4
[    1.051765] nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 128
[    1.055994] 25 qcomsmem partitions found on MTD device qcom_nand.0
[    1.063149] Creating 25 MTD partitions on "qcom_nand.0":
[    1.069390] 0x000000000000-0x000000100000 : "0:sbl1"
[    1.076216] 0x000000100000-0x000000200000 : "0:mibib"
[    1.081120] 0x000000200000-0x000000280000 : "0:bootconfig"
[    1.085585] 0x000000280000-0x000000300000 : "0:bootconfig1"
[    1.091038] 0x000000300000-0x000000600000 : "0:qsee"
[    1.098482] 0x000000600000-0x000000900000 : "0:qsee_1"
[    1.103731] 0x000000900000-0x000000980000 : "0:devcfg"
[    1.106602] 0x000000980000-0x000000a00000 : "0:devcfg_1"
[    1.111737] 0x000000a00000-0x000000a80000 : "0:apdp"
[    1.117177] 0x000000a80000-0x000000b00000 : "0:apdp_1"
[    1.122168] 0x000000b00000-0x000000b80000 : "0:rpm"
[    1.127102] 0x000000b80000-0x000000c00000 : "0:rpm_1"
[    1.131886] 0x000000c00000-0x000000c80000 : "0:cdt"
[    1.137085] 0x000000c80000-0x000000d00000 : "0:cdt_1"
[    1.141813] 0x000000d00000-0x000000d80000 : "0:appsblenv"
[    1.146971] 0x000000d80000-0x000000e80000 : "0:appsbl"
[    1.152764] 0x000000e80000-0x000000f80000 : "0:appsbl_1"
[    1.157796] 0x000000f80000-0x000001000000 : "0:art"
[    1.162908] 0x000001000000-0x000007100000 : "rootfs"
[    1.245840] mtd: setting mtd18 (rootfs) as root device
[    1.246133] mtdsplit: no squashfs found in "rootfs"
[    1.249924] 0x000007100000-0x000007a00000 : "0:wififw"
[    1.262443] 0x000007a00000-0x00000db00000 : "rootfs_1"
[    1.341470] 0x00000db00000-0x00000e400000 : "0:wififw_1"
[    1.349330] 0x00000e400000-0x00000fa00000 : "ubifs"
[    1.367731] 0x00000fa00000-0x00000fa80000 : "0:ethphyfw"
[    1.368610] 0x00000fa80000-0x00000fd00000 : "certificate"
[    1.384192] cpufreq: cpufreq_online: CPU0: Running at unlisted initial frequency: 19200 KHz, changing to: 1017600 KHz
[    1.385390] remoteproc remoteproc0: cd00000.q6v5_wcss is available
[    1.394416] ubi0: attaching mtd18
[    1.825756] ubi0: scanning is finished
[    1.831966] ubi0: attached mtd18 (name "rootfs", size 97 MiB)
[    1.832002] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[    1.836695] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[    1.843488] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[    1.850332] ubi0: good PEBs: 776, bad PEBs: 0, corrupted PEBs: 0
[    1.857094] ubi0: user volume: 3, internal volumes: 1, max. volumes count: 128
[    1.863353] ubi0: max/mean erase counter: 3/1, WL threshold: 4096, image sequence number: 1168335248
[    1.870392] ubi0: available PEBs: 0, total reserved PEBs: 776, PEBs reserved for bad PEB handling: 40
[    1.879681] ubi0: background thread "ubi_bgt0d" started, PID 508
[    1.880363] block ubiblock0_1: created from ubi0:1(rootfs)
[    1.898058] VFS: Mounted root (squashfs filesystem) readonly on device 254:0.
[    1.898475] Freeing unused kernel memory: 1280K
[    1.999006] Run /sbin/init as init process
[    1.999025]   with arguments:
[    1.999027]     /sbin/init
[    1.999030]   with environment:
[    1.999032]     HOME=/
[    1.999034]     TERM=linux
[    2.186392] init: Console is alive
[    2.186525] init: - watchdog -
[    2.558878] random: crng init done
[    2.593662] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    2.697977] ssdk_switch_device_num_init[1335]:INFO:ess-switch dts node number: 1
[    2.698081] ssdk_dt_get_switch_node[1124]:INFO:ess-switch DT exist!
[    2.704503] ssdk_dt_parse_access_mode[999]:INFO:switch_access_mode: local bus
[    2.710445] ssdk_dt_parse_access_mode[1012]:INFO:switchreg_base_addr: 0x3a000000
[    2.717729] ssdk_dt_parse_access_mode[1013]:INFO:switchreg_size: 0x1000000
[    2.725198] ssdk_dt_parse_mac_mode[327]:INFO:mac mode = 0xb
[    2.731880] ssdk_dt_parse_mac_mode[336]:INFO:mac mode1 = 0xff
[    2.737345] ssdk_dt_parse_mac_mode[345]:INFO:mac mode2 = 0xc
[    2.743263] ssdk_dt_parse_phy_info[716]:INFO:[PORT 6] port_mac_sel = QGMAC_PORT
[    2.749014] ssdk_dt_parse_uniphy[364]:INFO:ess-uniphy DT exist!
[    2.756104] ssdk_dt_parse_intf_mac[942]:INFO:dp1 MAC 4c:ab:f8:47:c0:c2
[    2.761945] ssdk_dt_parse_intf_mac[942]:INFO:dp2 MAC 4c:ab:f8:47:c0:c2
[    2.768537] ssdk_dt_parse_intf_mac[942]:INFO:dp3 MAC 4c:ab:f8:47:c0:c2
[    2.775054] ssdk_dt_parse_intf_mac[942]:INFO:dp4 MAC 4c:ab:f8:47:c0:c2
[    2.781640] ssdk_plat_init[1692]:INFO:ssdk_plat_init start
[    2.899851] ssdk_gcc_clock_init[1275]:INFO:SSDK gcc clock init successfully!
[    2.902757] malibu_phy_api_ops_init[2846]:INFO:qca probe malibu phy driver succeeded!
[    2.945258] hsl_phy_phydev_get[773]:ERROR:phy_addr 4 phydev is NULL
[    2.956102] qca808x_phy_api_ops_init[2566]:INFO:qca probe qca808x phy driver succeeded!
[    2.966930] regi_init[3953]:INFO:Initializing HPPE!!
[    3.198861] ssdk_ppe_reset_init[1535]:INFO:ppe reset successfully!
[    3.202355] qca_hppe_portctrl_hw_init[126]:INFO:Hawkeye PPE port initializing
[    3.214006] hsl_phy_phydev_get[773]:ERROR:phy_addr 4 phydev is NULL
[    3.217325] hsl_phy_phydev_get[773]:ERROR:phy_addr 4 phydev is NULL
[    3.914928] ssdk_switch_register[2017]:INFO:Chip version 0x1500
[    3.914959] qca_link_polling_select[1487]:INFO:link-polling-required node does not exist
[    3.919658] ssdk_switch_register[2043]:INFO:polling is selected
[    3.927982] regi_init[3957]:INFO:Initializing HPPE Done!!
[    3.933703] ssdk_genl_init[211]:INFO:ssdk genl family register success.
[    3.939191] regi_init[4020]:INFO:qca-ssdk module init succeeded!
[    3.947505] EDMA ver 1 hw init
[    3.952085] EDMA HW Reset completed succesfully
[    3.954742] Num rings - TxDesc:1 (23-23) TxCmpl:1 (7-7)
[    3.959145] RxDesc:1 (15-15) RxFill:1 (7-7)
[    3.965384] Generic PHY 90000.mdio-1:00: attached PHY driver (mii_bus:phy_addr=90000.mdio-1:00, irq=POLL)
[    3.969152] nss-dp 3a001000.dp1 lan4: Registered netdev lan4(qcom-id:1)
[    3.978952] Generic PHY 90000.mdio-1:01: attached PHY driver (mii_bus:phy_addr=90000.mdio-1:01, irq=POLL)
[    3.985235] nss-dp 3a001200.dp2 lan3: Registered netdev lan3(qcom-id:2)
[    3.995137] Generic PHY 90000.mdio-1:02: attached PHY driver (mii_bus:phy_addr=90000.mdio-1:02, irq=POLL)
[    4.001392] nss-dp 3a001400.dp3 lan2: Registered netdev lan2(qcom-id:3)
[    4.011316] Generic PHY 90000.mdio-1:03: attached PHY driver (mii_bus:phy_addr=90000.mdio-1:03, irq=POLL)
[    4.017502] nss-dp 3a001600.dp4 lan1: Registered netdev lan1(qcom-id:4)
[    4.232732] QCA808X ethernet 90000.mdio-1:1c: attached PHY driver (mii_bus:phy_addr=90000.mdio-1:1c, irq=POLL)
[    4.233297] nss-dp 3a007000.dp6-syn wan: Registered netdev wan(qcom-id:6)
[    4.241738] **********************************************************
[    4.248482] * NSS Data Plane driver
[    4.254905] **********************************************************
[    4.331890] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    4.331935] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 1
[    4.336394] xhci-hcd xhci-hcd.1.auto: hcc params 0x0220fe65 hci version 0x110 quirks 0x0000000002010010
[    4.343878] xhci-hcd xhci-hcd.1.auto: irq 89, io mem 0x08a00000
[    4.353285] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    4.359108] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 2
[    4.364664] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
[    4.372663] hub 1-0:1.0: USB hub found
[    4.378831] hub 1-0:1.0: 1 port detected
[    4.382837] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    4.386867] hub 2-0:1.0: USB hub found
[    4.394722] hub 2-0:1.0: 1 port detected
[    4.399646] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    4.408456] init: - preinit -
[    5.769972] nss-dp 3a001600.dp4 lan1: PHY Link up speed: 1000
[    5.770060] IPv6: ADDRCONF(NETDEV_CHANGE): lan1: link becomes ready
[    6.797045] UBIFS (ubi0:2): Mounting in unauthenticated mode
[    6.797155] UBIFS (ubi0:2): background thread "ubifs_bgt0_2" started, PID 733
[    6.825214] UBIFS (ubi0:2): recovery needed
[    6.898822] UBIFS (ubi0:2): recovery completed
[    6.898963] UBIFS (ubi0:2): UBIFS: mounted UBI device 0, volume 2, name "rootfs_data"
[    6.902189] UBIFS (ubi0:2): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[    6.910106] UBIFS (ubi0:2): FS size: 79613952 bytes (75 MiB, 627 LEBs), max 637 LEBs, journal size 3936256 bytes (3 MiB, 31 LEBs)
[    6.919988] UBIFS (ubi0:2): reserved for root: 3760363 bytes (3672 KiB)
[    6.931609] UBIFS (ubi0:2): media format: w5/r0 (latest is w5/r0), UUID A9647E73-C817-432F-8A57-744E258434F3, small LPT model
[    6.941804] mount_root: switching to ubifs overlay
[    6.952818] overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.
[    6.958041] urandom-seed: Seeding with /etc/urandom.seed
[    7.013372] nss-dp 3a001600.dp4 lan1: PHY Link is down
[    7.018821] procd: - early -
[    7.018966] procd: - watchdog -
[    7.572115] procd: - watchdog -
[    7.573461] procd: - ubus -
[    7.628254] procd: - init -
[    8.003403] kmodloader: loading kernel modules from /etc/modules.d/*
[    8.039778] Loading modules backported from Linux version v6.1.24-0-g0102425ac76b
[    8.039816] Backport generated by backports.git v5.15.92-1-44-gd6ea70fafd36
[    8.049421] NET: Registered PF_QIPCRTR protocol family
[    8.075665] PPP generic driver version 2.4.2
[    8.076399] NET: Registered PF_PPPOX protocol family
[    8.084502] ath11k c000000.wifi: ipq8074 hw2.0
[    8.084534] ath11k c000000.wifi: FW memory mode: 0
[    8.088410] remoteproc remoteproc0: powering up cd00000.q6v5_wcss
[    8.093165] remoteproc remoteproc0: Booting fw image IPQ8074/q6_fw.mdt, size 668
[    8.128612] urngd: v1.0.2 started.
[    9.093345] remoteproc remoteproc0: remote processor cd00000.q6v5_wcss is now up
[    9.096331] kmodloader: done loading kernel modules from /etc/modules.d/*
[    9.137292] ath11k c000000.wifi: qmi ignore invalid mem req type 3
[    9.144726] ath11k c000000.wifi: chip_id 0x0 chip_family 0x0 board_id 0xff soc_id 0xffffffff
[    9.144773] ath11k c000000.wifi: fw_version 0x290c84a5 fw_build_timestamp 2023-03-25 07:34 fw_build_id WLAN.HK.2.9.0.1-01385-QCAHKSWPL_SILICONZ-1
[    9.463086] ath11k c000000.wifi: htt event 48 not handled
[   10.967386] br-lan: port 1(lan1) entered blocking state
[   10.967431] br-lan: port 1(lan1) entered disabled state
[   10.971861] device lan1 entered promiscuous mode
[   10.988116] br-lan: port 2(lan2) entered blocking state
[   10.988164] br-lan: port 2(lan2) entered disabled state
[   10.992562] device lan2 entered promiscuous mode
[   11.008494] br-lan: port 3(lan3) entered blocking state
[   11.008540] br-lan: port 3(lan3) entered disabled state
[   11.012893] device lan3 entered promiscuous mode
[   11.025596] br-lan: port 4(lan4) entered blocking state
[   11.025643] br-lan: port 4(lan4) entered disabled state
[   11.029905] device lan4 entered promiscuous mode
[   11.677979] br-lan: port 5(phy1-ap0) entered blocking state
[   11.678023] br-lan: port 5(phy1-ap0) entered disabled state
[   11.682661] device phy1-ap0 entered promiscuous mode
[   11.688106] br-lan: port 5(phy1-ap0) entered blocking state
[   11.693191] br-lan: port 5(phy1-ap0) entered forwarding state
[   11.698870] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   11.969014] br-lan: port 5(phy1-ap0) entered disabled state
[   11.996325] IPv6: ADDRCONF(NETDEV_CHANGE): phy1-ap0: link becomes ready
[   11.996565] br-lan: port 5(phy1-ap0) entered blocking state
[   12.001803] br-lan: port 5(phy1-ap0) entered forwarding state
[   12.214882] br-lan: port 6(phy0-ap0) entered blocking state
[   12.214933] br-lan: port 6(phy0-ap0) entered disabled state
[   12.219604] device phy0-ap0 entered promiscuous mode
[   12.224910] br-lan: port 6(phy0-ap0) entered blocking state
[   12.230064] br-lan: port 6(phy0-ap0) entered forwarding state
[   12.236142] device phy0-ap0 left promiscuous mode
[   12.241437] br-lan: port 6(phy0-ap0) entered disabled state
[   12.320288] br-lan: port 6(phy0-ap0) entered blocking state
[   12.320331] br-lan: port 6(phy0-ap0) entered disabled state
[   12.324888] device phy0-ap0 entered promiscuous mode
[   12.330450] br-lan: port 6(phy0-ap0) entered blocking state
[   12.335445] br-lan: port 6(phy0-ap0) entered forwarding state
[   13.049544] br-lan: port 6(phy0-ap0) entered disabled state
[   13.156111] IPv6: ADDRCONF(NETDEV_CHANGE): phy0-ap0: link becomes ready
[   13.156290] br-lan: port 6(phy0-ap0) entered blocking state
[   13.161575] br-lan: port 6(phy0-ap0) entered forwarding state
[   13.270810] br-lan: port 7(phy0-ap1) entered blocking state
[   13.270858] br-lan: port 7(phy0-ap1) entered disabled state
[   13.275415] device phy0-ap1 entered promiscuous mode
[   13.282668] br-lan: port 7(phy0-ap1) entered blocking state
[   13.285968] br-lan: port 7(phy0-ap1) entered forwarding state
[   13.292287] device phy0-ap1 left promiscuous mode
[   13.297226] br-lan: port 7(phy0-ap1) entered disabled state
[   13.340389] br-lan: port 7(phy0-ap1) entered blocking state
[   13.340431] br-lan: port 7(phy0-ap1) entered disabled state
[   13.344980] device phy0-ap1 entered promiscuous mode
[   13.350456] br-lan: port 7(phy0-ap1) entered blocking state
[   13.355538] br-lan: port 7(phy0-ap1) entered forwarding state
[   13.409975] IPv6: ADDRCONF(NETDEV_CHANGE): phy0-ap1: link becomes ready
[   14.090939] nss-dp 3a001600.dp4 lan1: PHY Link up speed: 1000
[   14.091018] br-lan: port 1(lan1) entered blocking state
[   14.095677] br-lan: port 1(lan1) entered forwarding state
[   15.450180] nss-dp 3a007000.dp6-syn wan: PHY Link up speed: 2500
[   15.450253] IPv6: ADDRCONF(NETDEV_CHANGE): wan: link becomes ready
[   32.488924] l11: disabling
[  732.230637] kmodloader: loading kernel modules from /etc/modules.d/*
[  732.237292] GACT probability on
[  732.237909] Mirror/redirect action on
[  732.241983] u32 classifier
[  732.243055]     input device check on
[  732.245711]     Actions configured
[  732.258187] xt_time: kernel timezone is -0500
[  732.260218] kmodloader: done loading kernel modules from /etc/modules.d/*
[  732.286925] kmodloader: loading kernel modules from /etc/modules.d/*
[  732.289355] kmodloader: done loading kernel modules from /etc/modules.d/*
[  732.316349] kmodloader: loading kernel modules from /etc/modules.d/*
[  732.319023] kmodloader: done loading kernel modules from /etc/modules.d/*
[  732.380216] kmodloader: loading kernel modules from /etc/modules.d/*
[  732.383571] kmodloader: done loading kernel modules from /etc/modules.d/*
[  732.432492] kmodloader: loading kernel modules from /etc/modules.d/*
[  732.436102] kmodloader: done loading kernel modules from /etc/modules.d/*
[  732.466974] kmodloader: loading kernel modules from /etc/modules.d/*
[  732.470174] kmodloader: done loading kernel modules from /etc/modules.d/*
[15655.815909] hrtimer: interrupt took 1979 ns
[17403.329940] device phy0-ap0 left promiscuous mode
[17403.330086] br-lan: port 6(phy0-ap0) entered disabled state
[17403.421430] device phy0-ap1 left promiscuous mode
[17403.421547] br-lan: port 7(phy0-ap1) entered disabled state
[17404.857166] br-lan: port 6(phy0-ap0) entered blocking state
[17404.857204] br-lan: port 6(phy0-ap0) entered disabled state
[17404.861946] device phy0-ap0 entered promiscuous mode
[17404.867188] br-lan: port 6(phy0-ap0) entered blocking state
[17404.872337] br-lan: port 6(phy0-ap0) entered forwarding state
root@Avenue-Main2:~# uptime
 17:20:48 up  5:13,  load average: 0.03, 0.02, 0.02
root@Avenue-Main2:~#

brada4 commented 11 months ago

qualcommax/ipq807x Dynalink DL-WRX36

did you install package from default repos? Like opkg opkg list-installed will show versions.

ASerbinski commented 11 months ago

Everything is from default repos.

ath11k-firmware-ipq8074 - 2023-03-31-a039049a-1
base-files - 1550-r23630-842932a63d
busybox - 1.36.1-1
cJSON - 1.7.15-3
ca-bundle - 20230311-1
ca-certificates - 20230311-1
cgi-io - 2022-08-10-901b0f04-21
ddns-scripts - 2.8.2-42
ddns-scripts-services - 2.8.2-42
dnsmasq - 2.89-4
dropbear - 2022.82-5
e2fsprogs - 1.47.0-2
firewall4 - 2023-09-01-598d9fbb-1
fstools - 2023-02-28-bfe882d5-1
fwtool - 2019-11-12-8f7fe925-1
getrandom - 2022-08-13-4c7b720b-2
hostapd-common - 2023-09-08-e5ccbfc6-6
ipq-wifi-dynalink_dl-wrx36 - 2023-11-10-0c2e810e-1
iptables-mod-ipopt - 1.8.8-2
iptables-zz-legacy - 1.8.8-2
iw - 5.19-1
iwinfo - 2023-07-01-ca79f641-1
jansson4 - 2.14-3
jshn - 2023-05-23-75a3b870-1
jsonfilter - 2018-02-04-c7e938d6-1
kernel - 5.15.137-1-c0be4d8060b09729c42faeda72adef10
kmod-ath - 5.15.137+6.1.24-3
kmod-ath11k - 5.15.137+6.1.24-3
kmod-ath11k-ahb - 5.15.137+6.1.24-3
kmod-cfg80211 - 5.15.137+6.1.24-3
kmod-crypto-acompress - 5.15.137-1
kmod-crypto-aead - 5.15.137-1
kmod-crypto-ccm - 5.15.137-1
kmod-crypto-cmac - 5.15.137-1
kmod-crypto-crc32c - 5.15.137-1
kmod-crypto-ctr - 5.15.137-1
kmod-crypto-gcm - 5.15.137-1
kmod-crypto-gf128 - 5.15.137-1
kmod-crypto-ghash - 5.15.137-1
kmod-crypto-hash - 5.15.137-1
kmod-crypto-hmac - 5.15.137-1
kmod-crypto-manager - 5.15.137-1
kmod-crypto-michael-mic - 5.15.137-1
kmod-crypto-null - 5.15.137-1
kmod-crypto-rng - 5.15.137-1
kmod-crypto-seqiv - 5.15.137-1
kmod-crypto-sha512 - 5.15.137-1
kmod-fs-ext4 - 5.15.137-1
kmod-gpio-button-hotplug - 5.15.137-3
kmod-hwmon-core - 5.15.137-1
kmod-ifb - 5.15.137-1
kmod-ipt-core - 5.15.137-1
kmod-ipt-ipopt - 5.15.137-1
kmod-leds-gpio - 5.15.137-1
kmod-lib-crc-ccitt - 5.15.137-1
kmod-lib-crc16 - 5.15.137-1
kmod-lib-crc32c - 5.15.137-1
kmod-lib-lzo - 5.15.137-1
kmod-libphy - 5.15.137-1
kmod-mac80211 - 5.15.137+6.1.24-3
kmod-nf-conntrack - 5.15.137-1
kmod-nf-conntrack6 - 5.15.137-1
kmod-nf-flow - 5.15.137-1
kmod-nf-ipt - 5.15.137-1
kmod-nf-log - 5.15.137-1
kmod-nf-log6 - 5.15.137-1
kmod-nf-nat - 5.15.137-1
kmod-nf-reject - 5.15.137-1
kmod-nf-reject6 - 5.15.137-1
kmod-nfnetlink - 5.15.137-1
kmod-nft-core - 5.15.137-1
kmod-nft-fib - 5.15.137-1
kmod-nft-nat - 5.15.137-1
kmod-nft-offload - 5.15.137-1
kmod-nls-base - 5.15.137-1
kmod-phy-aquantia - 5.15.137-1
kmod-ppp - 5.15.137-1
kmod-pppoe - 5.15.137-1
kmod-pppox - 5.15.137-1
kmod-qca-nss-dp - 5.15.137+2022-04-30-72e9ec41-1
kmod-qca-ssdk - 5.15.137+2022-09-12-628b22bc-2
kmod-qrtr - 5.15.137-1
kmod-qrtr-smd - 5.15.137-1
kmod-sched-cake - 5.15.137-1
kmod-sched-core - 5.15.137-1
kmod-slhc - 5.15.137-1
kmod-thermal - 5.15.137-1
kmod-usb-core - 5.15.137-1
kmod-usb-dwc3 - 5.15.137-1
kmod-usb-dwc3-qcom - 5.15.137-1
kmod-usb-xhci-hcd - 5.15.137-1
kmod-usb3 - 5.15.137-1
libblkid1 - 2.39-2
libblobmsg-json20230523 - 2023-05-23-75a3b870-1
libbz2-1.0 - 1.0.8-1
libc - 1.2.4-4
libcap - 2.69-1
libcomerr0 - 1.47.0-2
libcurl4 - 8.5.0-1
libext2fs2 - 1.47.0-2
libgcc1 - 12.3.0-4
libip4tc2 - 1.8.8-2
libip6tc2 - 1.8.8-2
libiptext0 - 1.8.8-2
libiptext6-0 - 1.8.8-2
libiwinfo-data - 2023-07-01-ca79f641-1
libiwinfo20230701 - 2023-07-01-ca79f641-1
libjson-c5 - 0.16-3
libjson-script20230523 - 2023-05-23-75a3b870-1
liblua5.1.5 - 5.1.5-10
liblucihttp-lua - 2023-03-15-9b5b683f-1
liblucihttp-ucode - 2023-03-15-9b5b683f-1
liblucihttp0 - 2023-03-15-9b5b683f-1
libmbedtls12 - 2.28.5-2
libmnl0 - 1.0.5-1
libnftnl11 - 1.2.6-1
libnghttp2-14 - 1.57.0-1
libnl-tiny1 - 2023-07-27-bc92a280-1
libopenssl3 - 3.0.12-1
libpcap1 - 1.10.4-1
libpthread - 1.2.4-4
libpython3-3.11 - 3.11.6-1
librespeed-cli - 1.0.10-3
librt - 1.2.4-4
libsmartcols1 - 2.39-2
libss2 - 1.47.0-2
libstdcpp6 - 12.3.0-4
libubox20230523 - 2023-05-23-75a3b870-1
libubus-lua - 2023-06-05-f787c97b-1
libubus20230605 - 2023-06-05-f787c97b-1
libuci20130104 - 2023-08-10-5781664d-1
libuclient20201210 - 2023-04-13-007d9454-1
libucode20230711 - 2023-11-07-a6e75e02-1
libustream-mbedtls20201210 - 2023-02-25-498f6e26-1
libuuid1 - 2.39-2
libwebsockets-openssl - 4.3.2-2
libxtables12 - 1.8.8-2
logd - 2022-08-13-4c7b720b-2
losetup - 2.39-2
lua - 5.1.5-10
luci - git-23.051.66410-a505bb1
luci-app-ddns - git-23.346.52990-28c4a65
luci-app-firewall - git-23.338.81541-d56ca14
luci-app-mosquitto - git-22.134.64583-4e3d6c1
luci-app-opkg - git-23.311.75635-769b30c
luci-app-sqm - git-22.360.73151-127c900
luci-base - git-23.306.39416-c86c256
luci-compat - git-23.351.85440-817a5e6
luci-lib-base - git-22.308.54612-9118452
luci-lib-ip - git-23.311.79290-c2a887e
luci-lib-jsonc - git-23.298.74571-62eb535
luci-lib-nixio - git-23.338.82551-ea30bd9
luci-light - git-23.024.33244-34dee82
luci-lua-runtime - git-23.233.52805-dae2684
luci-mod-admin-full - git-19.253.48496-3f93650
luci-mod-network - git-23.342.43356-fc288de
luci-mod-status - git-23.340.83787-b989fcd
luci-mod-system - git-23.306.39416-7d3abf8
luci-proto-ipv6 - git-21.148.48881-79947af
luci-proto-ppp - git-21.158.38888-88b9d84
luci-ssl - git-23.035.26083-7550ad6
luci-theme-bootstrap - git-23.338.81541-94798fc
mosquitto-ssl - 2.0.17-1
mtd - 26
netifd - 2023-11-10-35facc83-1.1
netperf - 2.7.0-3
nftables-json - 1.0.8-1
odhcp6c - 2023-05-12-bcd28363-20
odhcpd-ipv6only - 2023-10-24-d8118f6e-1
openwrt-keyring - 2022-03-25-62471e69-2
opkg - 2022-02-24-d038e5b6-2
ppp - 2.4.9.git-2021-01-04-4
ppp-mod-pppoe - 2.4.9.git-2021-01-04-4
procd - 2023-06-25-2db83655-2
procd-seccomp - 2023-06-25-2db83655-2
procd-ujail - 2023-06-25-2db83655-2
px5g-mbedtls - 10
python3-base - 3.11.6-1
python3-codecs - 3.11.6-1
python3-email - 3.11.6-1
python3-light - 3.11.6-1
python3-logging - 3.11.6-1
python3-openssl - 3.11.6-1
python3-pkg-resources - 68.2.2-1
python3-speedtest-cli - 2.1.3-1
python3-urllib - 3.11.6-1
python3-xml - 3.11.6-1
rpcd - 2023-07-01-c07ab2f9-1
rpcd-mod-file - 2023-07-01-c07ab2f9-1
rpcd-mod-iwinfo - 2023-07-01-c07ab2f9-1
rpcd-mod-luci - 20230123-1
rpcd-mod-rrdns - 20170710
rpcd-mod-ucode - 2023-07-01-c07ab2f9-1
sqm-scripts - 1.6.0-1
tc-tiny - 6.3.0-1
tcpdump - 4.99.4-1
ubi-utils - 2.1.5-1
uboot-envtools - 2023.04-1
ubox - 2022-08-13-4c7b720b-2
ubus - 2023-06-05-f787c97b-1
ubusd - 2023-06-05-f787c97b-1
uci - 2023-08-10-5781664d-1
uclient-fetch - 2023-04-13-007d9454-1
ucode - 2023-11-07-a6e75e02-1
ucode-mod-fs - 2023-11-07-a6e75e02-1
ucode-mod-html - 1
ucode-mod-lua - 1
ucode-mod-math - 2023-11-07-a6e75e02-1
ucode-mod-nl80211 - 2023-11-07-a6e75e02-1
ucode-mod-rtnl - 2023-11-07-a6e75e02-1
ucode-mod-ubus - 2023-11-07-a6e75e02-1
ucode-mod-uci - 2023-11-07-a6e75e02-1
ucode-mod-uloop - 2023-11-07-a6e75e02-1
uhttpd - 2023-06-25-34a8a74d-1
uhttpd-mod-ubus - 2023-06-25-34a8a74d-1
urandom-seed - 3
urngd - 2023-11-01-44365eb1-1
usign - 2020-05-23-f1f65026-1
wireless-regdb - 2023.09.01-1
wpad-basic-mbedtls - 2023-09-08-e5ccbfc6-6
xtables-legacy - 1.8.8-2
zlib - 1.2.13-1

brada4 commented 11 months ago

Looks sane, like all currentoackages intact.

ASerbinski commented 11 months ago

Exactly. So where do I go from here? Any way you know of to coerce it into handing over a trace?

brada4 commented 11 months ago

strace, gdb, check wiki.

ASerbinski commented 11 months ago

Hmm. It may have been an issue with mosquitto 2.0.17. I installed 2.0.18 from the snapshot repository and I'm having a hard time reproducing the problem now.

brada4 commented 11 months ago

ping @neheb

neheb commented 11 months ago

You want it back ported?

ASerbinski commented 11 months ago

It would probably make a lot of sense. I've been trying for over an hour to reproduce the crash with 2.0.18 and have failed. With 2.0.17, I could make it happen within 3 tries over maybe 30 seconds.

neheb commented 11 months ago

done.

BKPepe commented 11 months ago

Btw, why are you trying to post something or everything, where there isnt anything related to mosquitto? You can see it even yourself that it was not helpful.

Backported even to OpenWrt 22.03.

Closing as I think 4 days was enough to reproduce it and test the version 2.0.18 on the stable branches.