on high load chilli stop give ip

[dobrozhanskyi]

Сhiilli running on OpenWRt 15.05.

When I start the torrent nobody else can get ip. PPS ~ 5k. CPU 100%. I launch on the same device dnsmasq and on same load receive ip from dnsmasq( but get in the debug "Received packet with spoofed source").

With a high number of pps is normal that stops working dhcp?

config chilli 'test' option tundev 'tun0' option kname 'chilli' option uamlisten '101.101.101.1' option dhcplisten '101.102.103.1' option dynip '101.102.103.0/24' option dhcpstart '2' option dns1 '8.8.8.8' option dns2 '8.8.4.4' option radiussecret '**_' option network 'lan' option uamsecret 'uamsecret' option ipup '/etc/chilli/up.sh' option ipdown '/etc/chilli/down.sh' option dnslog '/var/log/111' option macauth '1' option macpasswd '' option radiusserver1 '**' option radiusserver2 '_' option uamserver 'https://*****/' option uamdomain '.facebook.com,.fbcdn.net,.akamaihd.net'

[sevan]

What happens if you 1) run both chilli & your torrent client using nice eg nice chilli 2) run you torrent client with a lower priority (using a higher number value) eg nice 20 rtorrent

Still the same behaviour?

[dobrozhanskyi]

Torrent run on client device (not on router). I run chilli with nice -20 and problem still exist.

[sevan]

Ah, sorry, I assumed you're doing both on the same box. What you're describing is that when chilli is maxing out your CPU while forwarding, it's unable to serve other requests. Next thing to look at is if it's purely a resource issue (if CPU resources were available, would things work ok) or that there are limitations within the code. Either try running chilli on a faster system or using something like cpulimit https://github.com/opsengine/cpulimit

[shkrid]

@sevan I have the same problem. But if I got the IP address before logged in - redirect stops working at 100% CPU load(100Mbps of traffic passing through Coova-chilli on Carambola2 board) Is it possible to set in a priority issue dhcp address leases, redirection and authorization? Maybe I missed some compile options that would allow Chilli to work in non-blocking mode? Since the problem is not in the Chilli under heavy load, but in getting the IP, redirection and authorization.

For example, if 5 users connect to access point, logged in, and started download 20Mbps each, then 6-th user can neither connect to AP nor authorize in Chilli.

Thanks... Hope for your help

[sevan]

is the issue there if you run 1.3.0?

[gareth41]

I run the old 1.3.0 from 2012 and have never had any of these problems its rock solid, I run a commercial hotspot operation at a lot of accommodation places and this is what we use. I tried several versions/commits from github back in 2014/15 and only ended up with issues so abandoned it, we’re sticking with the old 1.3.0 for now.

From: Sevan Janiyan [mailto:notifications@github.com] Sent: Monday, 4 January 2016 3:42 p.m. To: coova/coova-chilli Subject: Re: [coova-chilli] on high load chilli stop give ip (#185)

is the issue there if you run 1.3.0 http://coova.github.io/coova-chilli/coova-chilli-1.3.0.tar.gz ?

— Reply to this email directly or view it on GitHub https://github.com/coova/coova-chilli/issues/185#issuecomment-168568315 . https://github.com/notifications/beacon/AL0MnZb4O8Ug68xU9X0C7411a43VAHwCks5pWdN0gaJpZM4G7pBq.gif

[shkrid]

@gareth41 Do you use kmod(xt_coova) in your HotSpot? Can you show your configuration? When I disable the kmod(xt_coova) - I have everything working. Chilli issues the IP addresses, even when 100% CPU utilization, but throughput below 30Mbps on the Carambola2 board #61 That's why I think that the problem is not in the CPU load.

top without kmod(xt_coova) during load(I can get ip):

PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
Mem: 31680K used, 29516K free, 736K shrd, 3252K buff, 11604K cached
CPU:  18% usr  38% sys   0% nic   0% idle   0% io   0% irq  43% sirq
Load average: 0.69 0.74 0.54 2/34 3890
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 3826     1 root     R     1524   2%  98% chilli
........

top with kmod(xt_coova) during load:

Mem: 31364K used, 29832K free, 740K shrd, 3252K buff, 11608K cached
CPU:   7% usr  11% sys   0% nic   0% idle   0% io   0% irq  81% sirq
Load average: 0.89 0.34 0.34 2/35 4015
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 3943     1 root     R     1528   2%  90% chilli
.....

I can't get ip already. May be the problem related to kmod(xt_coova) or to sirq?

But I need the maximum throughput so I'm forced to use kmod (xt_coova)

@sevan I tried 1.3.0, which in the source OpenWrt

PKG_VERSION:=1.3.0+20141128
PKG_SOURCE_URL:=git://github.com/coova/coova-chilli
PKG_SOURCE_VERSION:=b93de20a288c01c2ba28e96e31ad6da01627f45f

Also one of the latest versions of:

PKG_VERSION:=git_a4a67f19
PKG_SOURCE_URL:=git://github.com/coova/coova-chilli
PKG_SOURCE_VERSION:=a4a67f193e6a8087050a8148e26c5e03f5dd12b8

Same result. :-(

Anything advise? Thank you.

P.S. With regard to redirection and authorization. I'm probably exaggerating the problem as the redirection after a long delay but it is certainly going on and it is related with the link loading probably. But the main problem of the issue of the IP addresses are relevant.

[sevan]

@Shkrid as you asked, with what flags/features did you build coova?

[shkrid]

@sevan ./configure --target=mips-openwrt-linux --host=mips-openwrt-linux --build=x86_64-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-chilliproxy --disable-chilliredir --disable-dnslog --disable-miniportal --disable-useragent --disable-largelimits --disable-uamdomainfile --without-matrixssl --without-cyassl --without-openssl --with-nfcoova

[sevan]

This is the board you're using? Without chilli involved, how much traffic can you pass between two systems through this device?

[shkrid]

This is the board you're using?

Yes.

Without chilli involved, how much traffic can you pass between two systems through this device?

It has 2x10/100Base-T(X) ports [eth1 - wan, eth0 - lan ] Without chilli - wan=>lan - 100Mbps Chilli without xt_coova - wan=>lan - ~30Mbps Chilli with xt_coova - wan=>lan - 100Mbps

[sevan]

Impressed, do you have systemtap or perf on that system?

[dobrozhanskyi]

After I made ​​a dhcp on dnsmasq I see another problem - at a load stops working redirect to the captive portal (unauthorized users go to the requested resource).

In debug I see many records as (00-1F-F3-BF-56-56 - my mac): Tue Jan 5 13:48:03 2016 local6.debug coova-chilli[5594]: dhcp.c: 3830: Not for our MAC or broadcast: 00-1F-F3-BF-56-56 Tue Jan 5 13:48:03 2016 local6.debug coova-chilli[5594]: dhcp.c: 3830: Not for our MAC or broadcast: 00-1F-F3-BF-56-56 Tue Jan 5 13:48:03 2016 local6.debug coova-chilli[5594]: dhcp.c: 3830: Not for our MAC or broadcast: 00-1F-F3-BF-56-56

Also, I can not understand why the authorized users when enabled xt_coova load process chilli to 100% .

[shkrid]

Impressed, do you have systemtap or perf on that system?

@sevan I compiled image with perf support. How to use it? With wich parameters to start? What i should to do?

[sevan]

Ok. Coova needs to be built with debug symbols (-G) either set your CFLAGS variable with -G during the configure stage or add AM_CFLAGS += -G to configure.ac & re-bootstrap.

then try perf record -g chilli You'll get a file called perf.data which has the profiling information. You can then analyze the information using perf report --stdio Capture the data & share it so we can see what's going on :)

[shkrid]

@sevan I use such commands: perf record -p <pidof chilli> sleep 60 perf.data: perf.data.zip

Screenshots:

root@OpenWrt:/# perf report
# Overhead|Command|Shared Object         |Symbol                                 
9.38%|chilli |[ip_tables]           |[k] ipt_do_table                       
4.19%|chilli |[kernel.kallsyms]     |[k] ag71xx_poll                        
2.68%|chilli |[kernel.kallsyms]     |[k] dev_hard_start_xmit                
2.67%|chilli |[kernel.kallsyms]     |[k] __netif_receive_skb_core           
2.63%|chilli |[kernel.kallsyms]     |[k] nf_iterate                         
2.57%|chilli |[kernel.kallsyms]     |[k] ag71xx_hard_start_xmit             
2.48%|chilli |[kernel.kallsyms]     |[k] ip_forward                         
2.42%|chilli |[kernel.kallsyms]     |[k] __dev_queue_xmit                   
2.41%|chilli |[nf_conntrack]        |[k] nf_conntrack_in                    
2.27%|chilli |[kernel.kallsyms]     |[k] r4k_dma_cache_inv                  
2.25%|chilli |[kernel.kallsyms]     |[k] ip_rcv                             
2.14%|chilli |[kernel.kallsyms]     |[k] eth_type_trans                     
1.87%|chilli |[nf_conntrack]        |[k] __nf_conntrack_find_get            
1.79%|chilli |[kernel.kallsyms]     |[k] __build_skb                        
1.76%|chilli |[kernel.kallsyms]     |[k] ip_finish_output                   
1.74%|chilli |[kernel.kallsyms]     |[k] __local_bh_enable_ip               
1.63%|chilli |[kernel.kallsyms]     |[k] ag71xx_tx_packets                  
1.61%|chilli |[kernel.kallsyms]     |[k] ip_output                          
1.57%|chilli |[kernel.kallsyms]     |[k] __slab_alloc.isra.74.constprop.78  
1.37%|chilli |[nf_conntrack]        |[k] tcp_error                          
1.34%|chilli |[kernel.kallsyms]     |[k] __slab_free.isra.75                
1.19%|chilli |[kernel.kallsyms]     |[k] __copy_user_common                 
1.14%|chilli |[kernel.kallsyms]     |[k] r4k_dma_cache_wback_inv            
1.13%|chilli |[kernel.kallsyms]     |[k] br_handle_frame                    
1.09%|chilli |[kernel.kallsyms]     |[k] __bzero                            
1.07%|chilli |[kernel.kallsyms]     |[k] __do_softirq                       
1.05%|chilli |[xt_tcpudp]           |[k] tcp_mt                             
1.00%|chilli |[kernel.kallsyms]     |[k] memcmp                             
0.98%|chilli |[nf_conntrack_ipv4]   |[k] ipv4_pkt_to_tuple                  
0.95%|chilli |[nf_conntrack]        |[k] tcp_packet                         
0.92%|chilli |[kernel.kallsyms]     |[k] ag71xx_fill_rx_buf                 
0.91%|chilli |[kernel.kallsyms]     |[k] __kmalloc                          
0.90%|chilli |[kernel.kallsyms]     |[k] packet_rcv                         
0.87%|chilli |[kernel.kallsyms]     |[k] kmem_cache_free                    
0.82%|chilli |[kernel.kallsyms]     |[k] nf_hook_slow                       
0.82%|chilli |[nf_conntrack]        |[k] tcp_pkt_to_tuple                   
0.75%|chilli |[xt_coova]            |[k] coova_table_lookup                 
0.73%|chilli |[kernel.kallsyms]     |[k] br_dev_xmit                        
0.72%|chilli |[kernel.kallsyms]     |[k] skb_put
[....]

chilli config:

root@OpenWrt:/# cat /etc/chilli.conf 
ipup=/etc/chilli/up.sh
ipdown=/etc/chilli/down.sh
kname   chilli
dhcplisten 101.102.103.1
uamlisten   101.101.101.1
dhcpif      br-lan
net 101.102.103.0/24
dns1 "8.8.8.8"
localusers /etc/chilli/localusers
radiusserver1   "127.0.0.1"
radiussecret    "xxxxxxxx"
uamserver   "http://xxxxx.net/portal/"
uamsecret "xxxxxxx"

[shkrid]

Ok. Coova needs to be built with debug symbols (-G) either set your CFLAGS variable with -G during the configure stage or add AM_CFLAGS += -G to configure.ac & re-bootstrap.

then try perf record -g chilli You'll get a file called perf.data which has the profiling information. You can then analyze the information using perf report --stdio Capture the data & share it so we can see what's going on :)

Missed your above message. Should i am recompile coova anyway?

[sevan]

perf report --stdio > something.txt & post the results up.

If you run the report on another system without the same shared libraries/modules, you get

Failed to open /lib/modules/3.18.23/nf_nat_ipv4.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_conntrack.ko, continuing without symbols Failed to open /lib/modules/3.18.23/xt_coova.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ip_tables.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_conntrack_rtcache.ko, continuing without symbols Failed to open /lib/modules/3.18.23/xt_tcpudp.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_conntrack_ipv4.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_nat.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_defrag_ipv4.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_raw.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_mangle.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_filter.ko, continuing without symbols Failed to open /usr/lib/libchilli.so.0.0.0, continuing without symbols Failed to open /lib/librt-2.19.so, continuing without symbols Failed to open /lib/modules/3.18.23/mac80211.ko, continuing without symbols Failed to open /lib/modules/3.18.23/nf_reject_ipv6.ko, continuing without symbols Failed to open /lib/modules/3.18.23/iptable_nat.ko, continuing without symbols Failed to open /lib/libc-2.19.so, continuing without symbols Failed to open /lib/modules/3.18.23/ath9k.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ath9k_hw.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ath9k_common.ko, continuing without symbols Failed to open /lib/modules/3.18.23/tun.ko, continuing without symbols Failed to open /lib/modules/3.18.23/ip6_tables.ko, continuing without symbols Failed to open /lib/modules/3.18.23/cfg80211.ko, continuing without symbols Failed to open /lib/libpthread-2.19.so, continuing without symbols Failed to open /lib/modules/3.18.23/ath.ko, continuing without symbols

[sevan]

Also, just caught this, it looks like you ran out of memory? (bottom left hand side)

[shkrid]

perf report --stdio > something.txt & post the results up.

perf.txt

Also, just caught this, it looks like you ran out of memory? (bottom left hand side)

Maybe, but what it mean? Also I see 19000K free

[sevan]

that shows after the error had happened, right? what was the memory counter leading up to that?

[shkrid]

Let's go such way. :-) I made a screen capture of the process. screen_capture_perf.zip

[shkrid]

@sevan Any idea?