Kernel panic when connecting to AP

[frestuc]

Hi, I am getting a kernel panic when I am trying to associate to the AP after launching fosdem.sh. It happens with both the default BOOT/devicetree and the ones compiled from scratch:

root@analog:~/openwifi# sdr0: STA 1c:36:bb:8e:9b:06 IEEE 802.11: did not acknowledge authentication response
sdr0: STA 1c:36:bb:8e:9b:06 IEEE 802.11: associated (aid 1)
sdr0: AP-STA-CONNECTED 1c:36:bb:8e:9b:06
sdr0: STA 1c:36:bb:8e:9b:06 RADIUS: starting accounting session 5EBC28FE-00000000
Unable to handle kernel NULL pointer dereference at virtual address 00000004
pgd = c0004000
[00000004] *pgd=00000000
Internal error: Oops: 817 [#1] PREEMPT SMP ARM
Modules linked in: 8021q arc4 sdr(O) xpu(O) openofdm_rx(O) openofdm_tx(O) rx_intf(O) ad9361_drv(O) tx_intf(O) xilinx_dma mac80211 cfg80211 mwipcore mwipcore_iio_streaming mwipcore_dma_streaming mathworks_ip_common
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  O    4.14.0-g4fea7c58ad92-dirty #1
Hardware name: Xilinx Zynq Platform
task: c0c0da80 task.stack: c0c00000
PC is at skb_dequeue+0x54/0x6c
LR is at skb_dequeue+0x14/0x6c
pc : [<c05e337c>]    lr : [<c05e333c>]    psr: 200b0193
sp : c0c01e60  ip : 00000000  fp : 40000006
r10: c0c02080  r9 : ee07a898  r8 : bf0cd990
r7 : bf0cd938  r6 : ee07a894  r5 : ee07a888  r4 : eeabfc00
r3 : 00000000  r2 : 00000000  r1 : 600b0113  r0 : 600b0113
Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 18c5387d  Table: 2ebe804a  DAC: 00000051
Process swapper/0 (pid: 0, stack limit = 0xc0c00210)
Stack: (0xc0c01e60 to 0xc0c02000)
1e60: eeabfc00 ee07a888 ee07a460 bf074610 ee07a874 ee07a878 00000000 c0c5b200
1e80: c0b41294 ffffe000 c0c02080 c0120fb4 00000000 00000006 ffffe000 c0c02098
1ea0: c0c02080 00000101 c0c02080 c0101548 c0c0534c 00000000 ef003600 c0b412f8
1ec0: c0c5b200 0000000a ffff9a7f c0c02d00 c0801860 00200102 ef162364 c0b48280
1ee0: 00000035 00000000 00000000 ef003600 f8f01100 00000000 00000000 c01213cc
1f00: c0b48280 c015bc08 c0c05344 c0c1a114 f8f0010c f8f00100 c0c01f40 c0101424
1f20: c0108484 600b0013 ffffffff c0c01f74 c0c54c60 c0c00000 00000000 c010c20c
1f40: 00000001 00000000 00000001 c01179e0 ffffe000 c0c050f4 c0c050a8 c0b47a48
1f60: c0c54c60 00000000 00000000 00000000 2ec85000 c0c01f90 c0108480 c0108484
1f80: 600b0013 ffffffff 00000051 00000000 ffffe000 c0150e14 000000be c0c5a8c0
1fa0: c0c05080 ffffffff c0c5a8c0 efffcd40 c0b35a30 c0151148 c0c5a90c c0b00c6c
1fc0: ffffffff ffffffff 00000000 c0b00680 00000000 c0b35a30 c0c5ab54 c0c05098
1fe0: c0b35a2c c0c0ed04 0000406a 413fc090 00000000 0000807c 00000000 00000000
[<c05e337c>] (skb_dequeue) from [<bf074610>] (ieee80211_tasklet_handler+0x30/0xd4 [mac80211])
[<bf074610>] (ieee80211_tasklet_handler [mac80211]) from [<c0120fb4>] (tasklet_action+0x7c/0x118)
[<c0120fb4>] (tasklet_action) from [<c0101548>] (__do_softirq+0xe0/0x248)
[<c0101548>] (__do_softirq) from [<c01213cc>] (irq_exit+0xd0/0xd4)
[<c01213cc>] (irq_exit) from [<c015bc08>] (__handle_domain_irq+0x60/0xb4)
[<c015bc08>] (__handle_domain_irq) from [<c0101424>] (gic_handle_irq+0x4c/0x90)
[<c0101424>] (gic_handle_irq) from [<c010c20c>] (__irq_svc+0x6c/0xa8)
Exception stack(0xc0c01f40 to 0xc0c01f88)
1f40: 00000001 00000000 00000001 c01179e0 ffffe000 c0c050f4 c0c050a8 c0b47a48
1f60: c0c54c60 00000000 00000000 00000000 2ec85000 c0c01f90 c0108480 c0108484
1f80: 600b0013 ffffffff
[<c010c20c>] (__irq_svc) from [<c0108484>] (arch_cpu_idle+0x38/0x3c)
[<c0108484>] (arch_cpu_idle) from [<c0150e14>] (do_idle+0x16c/0x1f8)
[<c0150e14>] (do_idle) from [<c0151148>] (cpu_startup_entry+0x18/0x1c)
[<c0151148>] (cpu_startup_entry) from [<c0b00c6c>] (start_kernel+0x394/0x3a0)
Code: e5853008 e1c420d0 e584c000 e584c004 (e5823004) 
---[ end trace fc63c07a424ba159 ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D W  O    4.14.0-g4fea7c58ad92-dirty #1
Hardware name: Xilinx Zynq Platform
[<c010edbc>] (unwind_backtrace) from [<c010b6f4>] (show_stack+0x10/0x14)
[<c010b6f4>] (show_stack) from [<c06edd44>] (dump_stack+0x8c/0xa0)
[<c06edd44>] (dump_stack) from [<c010d4dc>] (ipi_cpu_stop+0x64/0x74)
[<c010d4dc>] (ipi_cpu_stop) from [<c010dc1c>] (handle_IPI+0x6c/0x7c)
[<c010dc1c>] (handle_IPI) from [<c0101464>] (gic_handle_irq+0x8c/0x90)
[<c0101464>] (gic_handle_irq) from [<c010c20c>] (__irq_svc+0x6c/0xa8)
Exception stack(0xef05ff78 to 0xef05ffc0)
ff60:                                                       00000001 00000000
ff80: 00000001 c01179e0 ffffe000 c0c050f4 c0c050a8 c0b47a48 c0c54c60 00000000
ffa0: 00000000 00000000 2ec95000 ef05ffc8 c0108480 c0108484 60000013 ffffffff
[<c010c20c>] (__irq_svc) from [<c0108484>] (arch_cpu_idle+0x38/0x3c)
[<c0108484>] (arch_cpu_idle) from [<c0150e14>] (do_idle+0x16c/0x1f8)
[<c0150e14>] (do_idle) from [<c0151148>] (cpu_startup_entry+0x18/0x1c)
[<c0151148>] (cpu_startup_entry) from [<0010174c>] (0x10174c)
---[ end Kernel panic - not syncing: Fatal exception in interrupt
sdr,sdr openwifi_rx_interrupt:  84bytes ht0 54M FC0108 DI002c addr1/2/3:665544332236/1c36bb8e9b06/333300000002 SCa9d0 fcs1 sn29 i29 -56dBm

[JiaoXianjun]

Which board are you using? I think it is most probably zcu102 board, right? The issue can be reproduced steadily or randomly? Could you tell me how to reproduce it?

For zcu102 board, we do observe this crash when downlink (openwifi to other) traffic is too big. Suspect that it is related to FPGA queue and Linux queue status unsync/conflict because zcu102 FPGA and Linux run in higher clock speed. We are working on this.

[frestuc]

The board is ZC706. I just loaded the BOOT.bin and devicetree.dtb from the openwifi folder in the SD card. Same thing happens with BOOT.bin and devicetree.dtb built from scratch.

[JiaoXianjun]

which RF board? We know that many people succeed with zc706. You could check: SD card flashing method. We use Ubuntu native tool to flash. You need 32bit image instead of 64bit image. copy file from openwifi/zc706_fmcs2/* to the base directory of BOOT partition. copy openwifi/zynq-common/uImage to the base directory of BOOT partition.

[JiaoXianjun]

You can also share me the output/dmesg of fosdem.sh

[frestuc]

which RF board?

FMCOMMS2

We know that many people succeed with zc706. You could check: SD card flashing method. We use Ubuntu native tool to flash. You need 32bit image instead of 64bit image. copy file from openwifi/zc706_fmcs2/* to the base directory of BOOT partition. copy openwifi/zynq-common/uImage to the base directory of BOOT partition.

Done, still getting the same output

You can also share me the output/dmesg of fosdem.sh

This is the output I get

root@analog:~/openwifi# ./fosdem.sh 
hostapd: no process found
network-manager stop/waiting
num_arg  0
last_input xxxxxxxxx
test_mode 0
rmmod: ERROR: Module sdr is not currently loaded
rmmod: ERROR: Module xilinx_dma is not currently loaded
check xilinx_dma module is loaded or not
xilinx_dma is loaded!
rmmod: ERROR: Module tx_intf is not currently loaded
check tx_intf module is loaded or not
tx_intf is loaded!
rmmod: ERROR: Module ad9361_drv is not currently loaded
check ad9361_drv module is loaded or not
ad9361_drv is loaded!
set RF frontend
Found openwifi_ad9361_fir.ftr
+ test -f /sys/bus/iio/devices/iio:device0/in_voltage_rf_bandwidth
+ test -f /sys/bus/iio/devices/iio:device1/in_voltage_rf_bandwidth
+ test -f /sys/bus/iio/devices/iio:device2/in_voltage_rf_bandwidth
+ cd /sys/bus/iio/devices/iio:device2/
+ set +x
FIR Rx: 32,1 Tx: 32,1
1
0
rx0 agc fast_attack
slow_attack
fast_attack
rx1 agc fast_attack
slow_attack
fast_attack
40000000
17500000
40000000
37500000
rssi
115.75 dB
115.50 dB
rx0 gain to 70
54.000000 dB
sh: echo: I/O error
54.000000 dB
rx1 gain to 70
53.000000 dB
sh: echo: I/O error
53.000000 dB
tx0 gain -89dB
-10.000000 dB
-89.000000 dB
tx1 gain 0dB
-10.000000 dB
0.000000 dB
rmmod: ERROR: Module rx_intf is not currently loaded
check rx_intf module is loaded or not
rx_intf is loaded!
rmmod: ERROR: Module openofdm_tx is not currently loaded
check openofdm_tx module is loaded or not
openofdm_tx is loaded!
rmmod: ERROR: Module openofdm_rx is not currently loaded
check openofdm_rx module is loaded or not
openofdm_rx is loaded!
rmmod: ERROR: Module xpu is not currently loaded
check xpu module is loaded or not
xpu is loaded!
rmmod: ERROR: Module sdr is not currently loaded
insert sdr.ko test_mode=0
check sdr module is loaded or not
sdr is loaded!
the end
Booting Linux on physical CPU 0x0
Linux version 4.14.0-g4fea7c58ad92-dirty (jxj@jxj-xps) (gcc version 7.3.1 20180314 (Linaro GCC 7.3-2018.04-rc3)) #1 SMP PREEMPT Thu Apr 23 21:41:40 CEST 2020
CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
OF: fdt: Machine model: Xilinx Zynq ZC706
OF: fdt: earlycon: stdout-path /amba@0/uart@E0001000 not found
Memory policy: Data cache writealloc
cma: Reserved 128 MiB at 0x38000000
On node 0 totalpages: 262144
free_area_init_node: node 0, pgdat c0c54000, node_mem_map ef7f7000
  Normal zone: 1536 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 196608 pages, LIFO batch:31
  HighMem zone: 65536 pages, LIFO batch:15
random: fast init done
percpu: Embedded 16 pages/cpu @ef7c5000 s33036 r8192 d24308 u65536
pcpu-alloc: s33036 r8192 d24308 u65536 alloc=16*4096
pcpu-alloc: [0] 0 [0] 1 
Built 1 zonelists, mobility grouping on.  Total pages: 260608
Kernel command line: console=ttyPS0,115200 root=/dev/mmcblk0p2 rw earlycon rootfstype=ext4 rootwait cpuidle.off=1
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 895972K/1048576K available (7168K kernel code, 339K rwdata, 2388K rodata, 1024K init, 165K bss, 21532K reserved, 131072K cma-reserved, 131072K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
    lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
      .text : 0xc0008000 - 0xc0800000   (8160 kB)
      .init : 0xc0b00000 - 0xc0c00000   (1024 kB)
      .data : 0xc0c00000 - 0xc0c54ea0   ( 340 kB)
       .bss : 0xc0c5a894 - 0xc0c83f7c   ( 166 kB)
Preemptible hierarchical RCU implementation.
    RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
    Tasks RCU enabled.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
efuse mapped to f0802000
slcr mapped to f0804000
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 512 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001
zynq_clock_init: clkc starts at f0804100
Zynq clock init
sched_clock: 64 bits at 333MHz, resolution 3ns, wraps every 4398046511103ns
clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x4ce07af025, max_idle_ns: 440795209040 ns
Switching to timer-based delay loop, resolution 3ns
clocksource: ttc_clocksource: mask: 0xffff max_cycles: 0xffff, max_idle_ns: 537538477 ns
timer #0 at f080c000, irq=17
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 666.66 BogoMIPS (lpj=3333333)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x100000 - 0x100060
Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
smp: Brought up 1 node, 2 CPUs
SMP: Total of 2 processors activated (1333.33 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 512 (order: 3, 32768 bytes)
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
zynq-ocm f800c000.ocmc: ZYNQ OCM pool: 256 KiB @ 0xf0880000
zynq-pinctrl 700.pinctrl: zynq pinctrl initialized
e0001000.serial: ttyPS0 at MMIO 0xe0001000 (irq = 25, base_baud = 3125000) is a xuartps
console [ttyPS0] enabled
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
media: Linux media interface: v0.10
Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
FPGA manager framework
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arm_global_timer
NET: Registered protocol family 2
TCP established hash table entries: 8192 (order: 3, 32768 bytes)
TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
UDP hash table entries: 512 (order: 2, 16384 bytes)
UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
NET: Registered protocol family 1
hw perfevents: no interrupt-affinity property for /pmu@f8891000, guessing.
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
workingset: timestamp_bits=30 max_order=18 bucket_order=0
bounce: pool size: 64 pages
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
io scheduler mq-deadline registered
io scheduler kyber registered
dma-pl330 f8003000.dmac: Loaded driver for PL330 DMAC-241330
dma-pl330 f8003000.dmac:    DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
brd: module loaded
loop: module loaded
m25p80 spi2.0: found s25fl128s, expected n25q128a11
m25p80 spi2.0: s25fl128s (32768 Kbytes)
5 ofpart partitions found on MTD device spi2.0
Creating 5 MTD partitions on "spi2.0":
0x000000000000-0x000000500000 : "boot"
0x000000500000-0x000000520000 : "bootenv"
0x000000520000-0x000000540000 : "config"
0x000000540000-0x000000fc0000 : "image"
0x000000fc0000-0x000002000000 : "spare"
MACsec IEEE 802.1AE
libphy: Fixed MDIO Bus: probed
tun: Universal TUN/TAP device driver, 1.6
libphy: MACB_mii_bus: probed
macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 29 (00:0a:35:00:01:22)
Marvell 88E1116R e000b000.ethernet-ffffffff:07: attached PHY driver [Marvell 88E1116R] (mii_bus:phy_addr=e000b000.ethernet-ffffffff:07, irq=POLL)
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
usbcore: registered new interface driver usb-storage
usbcore: registered new interface driver usbserial
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial support registered for generic
usbcore: registered new interface driver ftdi_sio
usbserial: USB Serial support registered for FTDI USB Serial Device
chipidea-usb2 e0002000.usb: e0002000.usb supply vbus not found, using dummy regulator
ULPI transceiver vendor/product ID 0x0424/0x0007
Found SMSC USB3320 ULPI transceiver.
ULPI integrity check: passed.
ci_hdrc ci_hdrc.0: EHCI Host Controller
ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 1
ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
i2c /dev entries driver
si570 1-005d: registered, current frequency 156250000 Hz
i2c i2c-0: Added multiplexed i2c bus 1
adv7511 2-0039: 2-0039 supply avdd not found, using dummy regulator
adv7511 2-0039: 2-0039 supply dvdd not found, using dummy regulator
adv7511 2-0039: 2-0039 supply pvdd not found, using dummy regulator
adv7511 2-0039: 2-0039 supply bgvdd not found, using dummy regulator
adv7511 2-0039: 2-0039 supply dvdd-3v not found, using dummy regulator
i2c i2c-0: Added multiplexed i2c bus 2
at24 3-0054: 1024 byte 24c08 EEPROM, writable, 1 bytes/write
i2c i2c-0: Added multiplexed i2c bus 3
pca953x 4-0021: 4-0021 supply vcc not found, using dummy regulator
pca953x 4-0021: interrupt support not compiled in
i2c i2c-0: Added multiplexed i2c bus 4
rtc rtc0: invalid alarm value: 2020-5-13 45:8:0
rtc-pcf8563 5-0051: rtc core: registered rtc-pcf8563 as rtc0
i2c i2c-0: Added multiplexed i2c bus 5
i2c i2c-0: Added multiplexed i2c bus 6
at24 7-0050: 256 byte 24c02 EEPROM, writable, 1 bytes/write
i2c i2c-0: Added multiplexed i2c bus 7
i2c i2c-0: Added multiplexed i2c bus 8
pca954x 0-0074: registered 8 multiplexed busses for I2C switch pca9548
IR NEC protocol handler initialized
IR RC5(x/sz) protocol handler initialized
IR RC6 protocol handler initialized
IR JVC protocol handler initialized
IR Sony protocol handler initialized
IR SANYO protocol handler initialized
IR Sharp protocol handler initialized
IR MCE Keyboard/mouse protocol handler initialized
IR XMP protocol handler initialized
usbcore: registered new interface driver uvcvideo
USB Video Class driver (1.1.1)
gspca_main: v2.14.0 registered
cdns-wdt f8005000.watchdog: Xilinx Watchdog Timer at f090a000 with timeout 10s
Xilinx Zynq CpuIdle Driver started
failed to register cpuidle driver
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
mmc0: SDHCI controller on e0100000.mmc [e0100000.mmc] using ADMA
ledtrig-cpu: registered to indicate activity on CPUs
hidraw: raw HID events driver (C) Jiri Kosina
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
adf4350 spi1.0: spi1.0 supply vcc not found, using dummy regulator
adf4350 spi1.0: Probe failed (muxout)
adf4350 spi1.1: spi1.1 supply vcc not found, using dummy regulator
adf4350 spi1.1: Probe failed (muxout)
mmc0: new high speed SDHC card at address aaaa
fpga_manager fpga0: Xilinx Zynq FPGA Manager registered
Netfilter messages via NETLINK v0.30.
nfnl_acct: registering with nfnetlink.
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ctnetlink v0.93: registering with nfnetlink.
nf_tables: (c) 2007-2009 Patrick McHardy <kaber@trash.net>
nf_tables_compat: (c) 2012 Pablo Neira Ayuso <pablo@netfilter.org>
xt_time: kernel timezone is -0000
ip_tables: (C) 2000-2006 Netfilter Core Team
ipt_CLUSTERIP: ClusterIP Version 0.8 loaded successfully
mmcblk0: mmc0:aaaa SC32G 29.7 GiB 
 mmcblk0: p1 p2 p3
arp_tables: arp_tables: (C) 2002 David S. Miller
NET: Registered protocol family 17
Registering SWP/SWPB emulation handler
asoc-simple-card adv7511_hdmi_snd: spdif-hifi <-> 75c00000.axi-spdif-tx mapping ok
input: gpio_keys as /devices/soc0/gpio_keys/input/input0
rtc-pcf8563 5-0051: setting system clock to 2020-05-13 19:36:48 UTC (1589398608)
ALSA device list:
  #0: HDMI monitor
EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 179:2.
devtmpfs: mounted
Freeing unused kernel memory: 1024K
systemd-udevd[1078]: starting version 204
systemd-udevd[1123]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
systemd-udevd[1123]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
systemd-udevd[1123]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
Registered mathworks_ip class
systemd-udevd[1128]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
systemd-udevd[1128]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
systemd-udevd[1128]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
systemd-udevd[1128]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
systemd-udevd[1128]: could not open builtin file '/lib/modules/4.14.0-g4fea7c58ad92-dirty/modules.builtin.bin'
mwipcore 43c00000.mwipcore: Dev memory resource found at 43c00000 0000FFFF. 
mwipcore 43c00000.mwipcore: 'mwipcore' device not found, creating
mwipcore 43c00000.mwipcore: Char dev region registered: major num:242
mwipcore 43c00000.mwipcore: 'mwipcore' device created
init: udev-fallback-graphics main process (1545) terminated with status 1
init: samba-ad-dc main process (1767) terminated with status 1
macb e000b000.ethernet eth0: link up (1000/Full)
init: isc-dhcp-server main process (1879) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (1975) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: lightdm main process (1819) terminated with status 1
init: isc-dhcp-server main process (2018) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (2056) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: tty1 main process (2047) killed by TERM signal
init: isc-dhcp-server main process (2090) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (2120) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (2141) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (2151) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (2161) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (2171) terminated with status 1
init: isc-dhcp-server main process ended, respawning
init: isc-dhcp-server main process (2181) terminated with status 1
init: isc-dhcp-server respawning too fast, stopped
xilinx-vdma 80400000.dma: Xilinx AXI DMA Engine Driver Probed!!
xilinx-vdma 80410000.dma: Xilinx AXI DMA Engine Driver Probed!!
xilinx-vdma 40420000.dma: Xilinx AXI DMA Engine Driver Probed!!
tx_intf: loading out-of-tree module taints kernel.

sdr,tx_intf dev_probe match!
sdr,tx_intf dev_probe io start 0x83c0ffff83c00000 end 0x200ef7f4ee8 name (null) flags 0xbf11401c desc 0xef16f410
sdr,tx_intf dev_probe base_addr 0xf2810000
sdr,tx_intf dev_probe tx_intf_driver_api_inst 0xbf114284
sdr,tx_intf dev_probe             tx_intf_api 0xbf114284
sdr,tx_intf dev_probe succeed!
sdr,tx_intf hw_init mode 6
sdr,tx_intf hw_init mode TX_INTF_BW_20MHZ_AT_N_10MHZ_ANT1
sdr,tx_intf hw_init err 0
ad9361 spi0.0: ad9361_probe : enter (ad9361)
ad9361 spi0.0: ad9361_probe : AD936x Rev 2 successfully initialized
cf_axi_adc 79020000.cf-ad9361-lpc: ADI AIM (10.01.b) at 0x79020000 mapped to 0xf27a8000, probed ADC AD9361 as MASTER
cf_axi_dds 79024000.cf-ad9361-dds-core-lpc: Analog Devices CF_AXI_DDS_DDS MASTER (9.01.b) at 0x79024000 mapped to 0xf27a1000, probed DDS AD9361
random: crng init done

sdr,rx_intf dev_probe match!
sdr,rx_intf dev_probe io start 0x83c20000 end 0x83c2ffff name /fpga-axi@0/rx_intf@83c20000 flags 0x00000200 desc 0x00000000
sdr,rx_intf dev_probe base_addr 0xf2900000
sdr,rx_intf dev_probe rx_intf_driver_api_inst 0xbf144284
sdr,rx_intf dev_probe             rx_intf_api 0xbf144284
sdr,rx_intf dev_probe succeed!
sdr,rx_intf hw_init mode 2
sdr,rx_intf hw_init mode DDC_BW_20MHZ_AT_0MHZ
sdr,rx_intf hw_init err 0

sdr,openofdm_tx dev_probe match!
sdr,openofdm_tx dev_probe io start 0x83c10000 end 0x83c1ffff name /fpga-axi@0/openofdm_tx@83c10000 flags 0x00000200 desc 0x00000000
sdr,openofdm_tx dev_probe base_addr 0xf2940000
sdr,openofdm_tx dev_probe openofdm_tx_driver_api_inst 0xbf14d284
sdr,openofdm_tx dev_probe             openofdm_tx_api 0xbf14d284
sdr,openofdm_tx dev_probe succeed!
sdr,openofdm_tx hw_init mode 1
sdr,openofdm_tx hw_init mode OPENOFDM_TX_NORMAL
sdr,openofdm_tx hw_init err 0

sdr,openofdm_rx dev_probe match!
sdr,openofdm_rx dev_probe io start 0x83c30000 end 0x83c3ffff name /fpga-axi@0/openofdm_rx@83c30000 flags 0x00000200 desc 0x00000000
sdr,openofdm_rx dev_probe base_addr 0xf2980000
sdr,openofdm_rx dev_probe openofdm_rx_driver_api_inst 0xbf155284
sdr,openofdm_rx dev_probe             openofdm_rx_api 0xbf155284
sdr,openofdm_rx dev_probe succeed!
sdr,openofdm_rx hw_init mode 1
sdr,openofdm_rx hw_init mode OPENOFDM_RX_NORMAL
sdr,openofdm_rx hw_init input:
power_thres 0
min_plateau 100
sdr,openofdm_rx hw_init err 0

sdr,xpu dev_probe match!
sdr,xpu dev_probe io start 0x83c40000 end 0x83c4ffff name /fpga-axi@0/xpu@83c40000 flags 0x00000200 desc 0x00000000
sdr,xpu dev_probe base_addr 0xf29e0000
sdr,xpu dev_probe xpu_driver_api_inst 0xbf15e284
sdr,xpu dev_probe             xpu_api 0xbf15e284
sdr,xpu dev_probe reset tsf timer
sdr,xpu dev_probe XPU_REG_TSF_RUNTIME_VAL_LOW_read 1 33647 100655us
sdr,xpu dev_probe succeed!
sdr,xpu hw_init mode 1
sdr,xpu hw_init mode XPU_NORMAL
sdr,xpu hw_init err 0
sdr: no symbol version for openofdm_rx_api

sdr,sdr openwifi_dev_probe: match!
sdr,sdr custom_match_spi_dev ad9361-phy ad9361-phy 1
sdr,sdr bus_find_device ad9361-phy: ad9361. driver_data pointer ef2fd010
sdr,sdr ad9361_spi_to_phy ad9361-phy: ad9361
sdr,sdr openwifi_dev_probe: ad9361_ctrl_outs_setup en_mask 0xff index 0x16
sdr,sdr openwifi_dev_probe: ad9361_spi_read REG_CTRL_OUTPUT_POINTER 0x16
sdr,sdr openwifi_dev_probe: ad9361_spi_read REG_CTRL_OUTPUT_ENABLE 0xff
sdr,sdr custom_match_platform_dev 79024000.cf-ad9361-dds-core-lpc
sdr,sdr openwifi_dev_probe: cf-ad9361-dds-core-lpc dds_st->version 00090162 chip_info->name AD9361
sdr,sdr openwifi_dev_probe: cf_axi_dds_datasel DATA_SEL_DMA
sdr,sdr openwifi_dev_probe: priv->rf_bw == 40000000Hz. bool for 20000000 0, 40000000 1
sdr,sdr openwifi_dev_probe: test_mode 0
sdr,sdr openwifi_dev_probe: rfkill radio on
sdr,sdr openwifi_dev_probe: band_2GHz.n_channels 14 n_bitrates 12 band_5GHz.n_channels 11 n_bitrates 12
sdr,sdr openwifi_dev_probe: mac_addr 66:55:44:33:22:0f
ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
sdr,sdr openwifi_dev_probe: ieee80211_register_hw 0
ieee80211 phy0: hwaddr 66554433220f, ZYNQ + ad9361
sdr,sdr openwifi_rfkill_init: wireless switch is on
sdr,sdr openwifi_rfkill_poll: wireless radio switch turned on
sdr,sdr fpga-axi@0:sdr sdr0: renamed from wlan0
stop: Unknown instance: 
isc-dhcp-server start/running, process 2370
Configuration file: hostapd-openwifi.conf
sdr0: interface state UNINITIALIZED->COUNTRY_UPDATE
Using interface sdr0 with hwaddr 66:55:44:33:22:0f and ssid "openwifi"
sdr0: interface state COUNTRY_UPDATE->ENABLED
sdr0: AP-ENABLED 

[JiaoXianjun]

Can't see abnormal things.

After running hostapd, can you see the "openwifi" SSID from other wifi device? At this step, is there a crash?

Crash always happen when you click connect on other device to connect openwifi?

I suggest use some 5GHz channel (like our default 44 hostapd configuration file) as start. Also could check some basic functionalities:

service network-manager stop ./wgd.sh iwlist sdr0 scan (See if some SSIDs can be found)

use wpa_supplicant or "iwconfig sdr0 essid SSID_NAME" to connect other AP from openwifi.

[frestuc]

Can't see abnormal things.

After running hostapd, can you see the "openwifi" SSID from other wifi device? At this step, is there a crash?

No. The crash happens when i connect to the openwifi network.

Crash always happen when you click connect on other device to connect openwifi?

Yes

I suggest use some 5GHz channel (like our default 44 hostapd configuration file) as start. Also could check some basic functionalities:

service network-manager stop ./wgd.sh iwlist sdr0 scan (See if some SSIDs can be found)

iwlist works fine, unfortunately we get that error when connecting to openwifi as AP

[JiaoXianjun]

Could you tell me your zc706 revision?

https://www.xilinx.com/support/answers/59748.html

[frestuc]

Could you tell me your zc706 revision?

https://www.xilinx.com/support/answers/59748.html

REV 2.0

[JiaoXianjun]

Currently I can not figure out the reason. From the crash message, something abnormal happens in mac80211, not in sdr driver. Looks like the Linux ( file system ) is abnormal. So, Change the SD card, or change the board to take a look. We will also test again at our side with zc706.

[frestuc]

Currently I can not figure out the reason. From the crash message, something abnormal happens in mac80211, not in sdr driver. Looks like the Linux ( file system ) is abnormal. So, Change the SD card, or change the board to take a look. We will also test again at our side with zc706.

It looks indeed like a Linux problem. I want to say that with the earlier versions of openwifi (when you had only the zc706 code, back in late November 2019) this wasn't happening.

[JiaoXianjun]

Can you follow this section (https://github.com/open-sdr/openwifi#build-openwifi-linux-img-from-scratch) to make a new SD card?

[frestuc]

Can you follow this section (https://github.com/open-sdr/openwifi#build-openwifi-linux-img-from-scratch) to make a new SD card?

I did it, still getting the kernel panic

[JiaoXianjun]

Even after you fix your kernel issue in the closed driver compiling issue?

[frestuc]

Even after you fix your kernel issue in the closed driver compiling issue?

Yes. Is there any way we can debug this problem?

[JiaoXianjun]

Share us the SDR card image you built by script. We can take a look and try our side.

[JiaoXianjun]

We tested again. It works perfectly at our side on zc706 fmcomms2. Maybe you should change your hardware to nail it down.

[frestuc]

Sorry for the delay, please find the images here:

https://www.dropbox.com/s/jbn9k4eumubie9f/sdcard_BOOT.img?dl=0 https://www.dropbox.com/s/7xwt23mqjt1o30p/sdcard_rootfs.img?dl=0

[JiaoXianjun]

How to flash two img files into a single SD card? Can you make one whole image for the SD card?

[frestuc]

How to flash two img files into a single SD card?

Can you try with dd if=path/to/sdcard_BOOT.img of=/dev/mount_for_BOOT bs=1024 and similar for rootfs?

[JiaoXianjun]

To avoid extra uncertainty, we prefer to follow our procedure firstly to do the test. After you create your SD card with our script, please try to generate a single img file from SD card as following:

sudo fdisk /dev/mmcblk0 -l (mmcblk0 is the SD card device name in your computer! Find out the correct name!)

Disk /dev/sdc: 14,9 GiB, 15962472448 bytes, 31176704 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00096174

Device Boot Start End Sectors Size Id Type /dev/sdc1 8192 1056767 1048576 512M c W95 FAT32 (LBA) /dev/sdc2 1056768 XXXXXXX 13441024 6,4G 83 Linux /dev/sdc3 4096 6143 2048 1M a2 unknown

(From above fdisk info, the SD card Sector size 512 and end at XXXXXXX. Then:)

sudo dd bs=512 count=XXXXXXX if=/dev/mmcblk0 of=openwifi-xxxx.img

[JiaoXianjun]

Could you tell me how did you create SD card after download our pre-built image?

[frestuc]

Could you tell me how did you create SD card after download our pre-built image?

with Ubuntu's Startup Disk Creator

[frestuc]

To avoid extra uncertainty, we prefer to follow our procedure firstly to do the test. After you create your SD card with our script, please try to generate a single img file from SD card as following:

sudo fdisk /dev/mmcblk0 -l (mmcblk0 is the SD card device name in your computer! Find out the correct name!)

Disk /dev/sdc: 14,9 GiB, 15962472448 bytes, 31176704 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00096174

Device Boot Start End Sectors Size Id Type /dev/sdc1 8192 1056767 1048576 512M c W95 FAT32 (LBA) /dev/sdc2 1056768 XXXXXXX 13441024 6,4G 83 Linux /dev/sdc3 4096 6143 2048 1M a2 unknown

(From above fdisk info, the SD card Sector size 512 and end at XXXXXXX. Then:)

sudo dd bs=512 count=XXXXXXX if=/dev/mmcblk0 of=openwifi-xxxx.img

There you go https://www.dropbox.com/s/9aeog7pfag8fzz0/openwifi-frank.img?dl=0

[weiliu1011]

Hi Frank, I just tested your image on my hardware zc706 rev 2.0, fmcoms4, it works perfectly, I can connect two phones to it and stream video several times to both phones without any sign of crash. So this means a high probability of hardware issue on your side. You said that with earlier version of openwifi you did not experience of any issue, I would say test that earlier version, see if you experience issue, if same thing occurs then we are certain it is hardware related. Br, Wei

[frestuc]

I would say test that earlier version, see if you experience issue, if same thing occurs then we are certain it is hardware related.

Ok, do you know where I can get the old rootfs?

[JiaoXianjun]

You can go back to the commit you think is OK, and tell me the image file name? The very beginning one is here: https://drive.google.com/file/d/1tEL9RIES8PwsvBoqOjGoaFJ9_jflsNsI/view?usp=sharing

[frestuc]

The openwifi-hw commit is ec02b58990ff1fd89afe78853182ca3955866f28, the openwifi commit is 20101e8202c7626fd0220b8b5022cf7b2632ba91. Can you confirm the one you linked is the right one?

[frestuc]

I do not experience the same issue when using the image you posted. So I think it's not a hw issue...

[JiaoXianjun]

Thing is getting interesting. We are working on more safe queue management method which I believe will reduce a lot of chance getting crash in Linux. It will be available in next few weeks.

Could you test with more types of client (iPhone, iPad, laptop, Andriod, dongle, etc)? Because different Wi-Fi chips have different characters (Timing, packet interval, etc.). The crash only happens for a single client? Or all kinds of client can trigger this crash?

Meanwhile, if you want, you can test more images: https://drive.google.com/drive/folders/1KadYvc-dRD6iBdpHE3XfPMI0mJxKJ9Np?usp=sharing This will help us understand the issue.

Thanks!

[frestuc]

Thing is getting interesting. We are working on more safe queue management method which I believe will reduce a lot of chance getting crash in Linux. It will be available in next few weeks.

Could you test with more types of client (iPhone, iPad, laptop, Andriod, dongle, etc)? Because different Wi-Fi chips have different characters (Timing, packet interval, etc.). The crash only happens for a single client? Or all kinds of client can trigger this crash?

I have tried different clients, same thing. I think it's definitely a kernel/software issue.

Meanwhile, if you want, you can test more images: https://drive.google.com/drive/folders/1KadYvc-dRD6iBdpHE3XfPMI0mJxKJ9Np?usp=sharing This will help us understand the issue.

Will do, let me get back to you.

Thanks!

[jiskra]

我用zc706+fmcoms3会遇到一样的问题，尝试了多日依然没有解决。

[JiaoXianjun]

Hello,

Would you please try this new image (openwifi-1.1.0-taiyuan-3-32bit.img.xz) to see whether the crash issue is gone? https://drive.google.com/file/d/1G8yURR0pxFlTuZkfDQpFSIi7i0l4syOA/view?usp=sharing

I have fixed the potential crash issue by improving FPGA and driver involving more safe and exception tolerant design.

If your test give me positive result, I will update the source code to this git repository soon.

Thanks!

[frestuc]

Hi, I am getting the following error:

root@analog:~/openwifi# hostapd hostapd-openwifi.conf 
Configuration file: hostapd-openwifi.conf
nl80211: Could not configure driver mode
nl80211 driver initialization failed.
hostapd_free_hapd_data: Interface sdr0 wasn't started

However, sdr0 is up and works fine (was able to scan with iwlist).

[JiaoXianjun]

Can you just flash the SD card with the image, and run fosdem.sh to start everything?

[frestuc]

It works, I am able to associate without crash, however the link is very unstable and weak, I get disassociated after a short while. I think you fixed the bug though.

[JiaoXianjun]

Good to know! In our local test with iPhone and laptop no big performance difference. Maybe you could adjust distance/location. For iphone you can use WiFiPerf Endpoint, and HE.NET Network Tools.

[frestuc]

What is the commit where you fixed the bug? Is it 4ecf49bb3edf31ca09d11d9d5e397c3dbc353c0a?

[JiaoXianjun]

I think so. Also openwifi-hw is updated as well. It is an essential update of the FPGA, driver and Linux flow control mechanism on different queues.