skbuff_head_cache allocating all system memory and invoking OOM

[gravikumar92]

Hi All,

Im working on 80211s mesh using ralink usb dongle + openwrt + compat-wireless-2016-01-10.

i have created mesh access points ( bridging AP + MP interfaces).

with 5 to 6 boards(MAP) im facing system reboot invoked by OOM.

I debugged a little bit, found that skbuff_head_cache allocating all system memory and causing this problem. (Any idea?) I have checked mesh_stats in debugfs while skbuff_head_cache allocating memory. observation is: dropped_frames_ttl, fwded_mcast and fwded_unicast frames are increasing continuously.

Please help me.

[chunyeow]

Which Ralink chipset that you used? With or without security?

Kernel debug log?

On Tue, Feb 21, 2017 at 4:34 PM, gravikumar92 notifications@github.com wrote:

Hi All,

Im working on 80211s mesh using ralink usb dongle + openwrt + compat-wireless-2016-01-10.

i have created mesh access points ( bridging AP + MP interfaces).

with 5 to 6 boards(MAP) im facing system reboot invoked by OOM.

I debugged a little bit, found that skbuff_head_cache allocating all system memory and causing this problem. (Any idea?) I have checked mesh_stats in debugfs while skbuff_head_cache allocating memory. observation is: dropped_frames_ttl, fwded_mcast and fwded_unicast frames are increasing continuously.

Please help me.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/o11s/open80211s/issues/55, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBewulySGUJJOh3ySXnmEXUSJv_z42Vks5reqGqgaJpZM4MHA_l .

[gravikumar92]

WiFi dongle Log: [ 6.556806] usb 2-1: reset high-speed USB device number 2 using ehci-platform [ 6.717537] ieee80211 phy0: rt2x00_set_rt: Info - RT chipset 5390, rev 0502 detected [ 6.753534] ieee80211 phy0: rt2x00_set_rf: Info - RF chipset 5370 detected [ 6.761157] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht' [ 6.763387] usbcore: registered new interface driver rt2800usb [ 9.153624] ieee80211 phy0: rt2x00lib_request_firmware: Info - Loading firmware file 'rt2870.bin' [ 9.167651] ieee80211 phy0: rt2x00lib_request_firmware: Info - Firmware detected - version: 0.29

Log: root@OpenWrt:/# cat proc/meminfo | grep S SwapCached: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Shmem: 64 kB Slab: 94680 kB SReclaimable: 3672 kB SUnreclaim: 91008 kB KernelStack: 608 kB NFS_Unstable: 0 kB Committed_AS: 6256 kB

root@OpenWrt:/# cat proc/slabinfo ... dmaengine-unmap-2 1 42 96 42 1 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 98 108 448 18 2 : tunables 0 0 0 : slabdata 6 6 0 skbuff_fclone_cache 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0 skbuff_head_cache 326740 326768 256 16 1 : tunables 0 0 0 : slabdata 20423 20423 0 configfs_dir_cache 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 file_lock_cache 0 23 176 23 1 : tunables 0 0 0 : slabdata 1 1 0 file_lock_ctx 8 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 shmem_inode_cache 1279 1292 416 19 2 : tunables 0 0 0 : slabdata 68 68 0

[gravikumar92]

with security (authsae)

[chunyeow]

If you turn off security, any problems?

On Feb 21, 2017 9:35 PM, "gravikumar92" notifications@github.com wrote:

with security (authsae)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/o11s/open80211s/issues/55#issuecomment-281345734, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBewtg3P4PVe84seGejvII-D3QWGTKxks5reugqgaJpZM4MHA_l .

[gravikumar92]

with security and without security same problem. /sys/kernel/debug/ieee80211/phy0/netdev:wlan0/mesh_stats/fwded_mcast are increasing.

I am unable to reproduce this with udp iperf tests between 2 nodes.

I think the kernel mesh(80211s) code is allocating more skbs.

Can you point any variable in the 80211s code to identify the problem?

[gravikumar92]

this problem is random with 5 to 6 MAPs.

[gravikumar92]

Hi,

i have captured packets using wireshark.

there are so many broad cast packets from all nodes (HWMP path selection packets) these packets are allocation all system memory.

Please check log. give me some suggestion

log.pcapng.zip MAC : 2a:07:41:b3:1e:22 : out of memory.

root@OpenWrt:/sys/kernel/debug/ieee80211/phy0/netdev:wlan0/mesh_stats# dropped_frames_congestion = 1 dropped_frames_no_route = 26 dropped_frames_ttl = 83931 fwded_frames = 415313 fwded_mcast = 409756 fwded_unicast = 5557

[chunyeow]

Just a quick check, did you load the rt2800usb driver with "nohwcrypt=1" for secured mesh?

On Wed, Feb 22, 2017 at 4:27 PM, gravikumar92 notifications@github.com wrote:

Hi,

i have captured packets using wireshark.

there are so many broad cast packets from all nodes (HWMP path selection packets) these packets are allocation all system memory.

Please check log. give me some suggestion

log.pcapng.zip https://github.com/o11s/open80211s/files/792568/log.pcapng.zip MAC : 2a:07:41:b3:1e:22 : out of memory.

root@OpenWrt:/sys/kernel/debug/ieee80211/phy0/netdev:wlan0/mesh_stats# dropped_frames_congestion = 1 dropped_frames_no_route = 26 dropped_frames_ttl = 83931 fwded_frames = 415313 fwded_mcast = 409756 fwded_unicast = 5557

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/o11s/open80211s/issues/55#issuecomment-281601713, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBewsDF5Bsxx4prqBQKpsAKyBBaPIXAks5re_FngaJpZM4MHA_l .

[gravikumar92]

I have done that.

[gravikumar92]

is there any relation with #define TOFFSET_MAXIMUM_ADJUSTMENT 30000 / 30 ms / macro. changed it to 800. Have you seen Wireshark log?

[chunyeow]

Can you confirm whether this issue is obvious when the number of nodes perhaps let say 3 or 4?

Also all the nodes in your network have successfully peered (established) with each other?

I am wondering whether this is problem that the firmware of rt2800usb has limited number of stations.

On Thu, Feb 23, 2017 at 12:21 PM, gravikumar92 notifications@github.com wrote:

is there any relation with #define TOFFSET_MAXIMUM_ADJUSTMENT 30000 / 30 ms / macro. changed it to 800. Have you seen Wireshark log?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/o11s/open80211s/issues/55#issuecomment-281890568, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBewo_qlBjaNYStu1y0mJaBqfDgOCAMks5rfQksgaJpZM4MHA_l .

[gravikumar92]

with 3 to 4 nodes there is no problem. Yes, all nodes in my network successfully established the connections.( i have to recheck) Could you explain me, how HWMP path request triggers?(what rate). I can see all packets are HWMP path requests and broadcast.

There are so many unknown variables in kernel and authsae configuration.

I tried static mpaths using iw. i have seen same issue again.

[gravikumar92]

How to use fixed mpaths?

[chunyeow]

HWMP path request is based on broadcast PREQ propagated by the initial node.

On Thu, Feb 23, 2017 at 12:32 PM, gravikumar92 notifications@github.com wrote:

with 3 to 4 nodes there is no problem. Yes, all nodes in my network successfully established the connections.( i have to recheck) Could you explain me, how HWMP path request triggers?(what rate). I can see all packets are HWMP path requests and broadcast.

There are so many unknown valibles in kernel and authsae configuration.

I tried static mpaths using iw. i have seen same issue again.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/o11s/open80211s/issues/55#issuecomment-281892025, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBewnUrehi1grSVwlvCxET3I1owt2xdks5rfQvegaJpZM4MHA_l .

[chunyeow]

Use iw utility

On Fri, Feb 24, 2017 at 12:49 PM, gravikumar92 notifications@github.com wrote:

How to use fixed mpaths?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/o11s/open80211s/issues/55#issuecomment-282203543, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBewmRIua2lwRk48R3GQFBVxJxfYmgPks5rfmFUgaJpZM4MHA_l .