Closed litinoveweedle closed 7 years ago
litinoveweedle
commented
7 years ago One additional observation, it seems that this issue is somehow bounded to CPU performance. It occurs with higher probability on slower CPU (almost certainly), while on more powerful CPU it occurs only sporadically.
litinoveweedle
commented
7 years ago Hello, is there any updates on bug? Can we help somehow to troubleshoot this one? It is really serious problem for us, so we are ready to pay bounty for solving it! BR L.
k0ste
commented
7 years ago Can we help somehow to troubleshoot this one?
How community help if you don't post your: environment, hardware, settings, kernel patches and config? This is non standard problem, because for now only you have this one.
You try 4.9 kernel?
micron10
commented
7 years ago in 4.9 have bug when use imq + hsfq + ppp machine work one day and crash.
IPacct ltd.
On 24 Feb 2017 7:33 am, "Konstantin Shalygin" notifications@github.com wrote:
Can we help somehow to troubleshoot this one? How community help if you don't post your: environment, hardware, settings, kernel patches and config? This is non standard problem, because for now only you have this one.
You try 4.9 kernel?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/imq/linuximq/issues/55#issuecomment-282208466, or mute the thread https://github.com/notifications/unsubscribe-auth/AMVh9HgEyIchJosDv5yan_JuCnVl-AaPks5rfmu4gaJpZM4LSzII .
k0ste
commented
7 years ago What is "hsfq"?
micron10
commented
7 years ago Sorry hfsc and after switch to htb problem is going but for little more ma hine crash after 4-5 day. but i not get log from crash
IPacct ltd.
On 24 Feb 2017 7:53 am, "Konstantin Shalygin" notifications@github.com wrote:
What is "hsfq"?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/imq/linuximq/issues/55#issuecomment-282210772, or mute the thread https://github.com/notifications/unsubscribe-auth/AMVh9JBPuhu2KRXRGQX07SQe3SWBnd3Aks5rfnBNgaJpZM4LSzII .
litinoveweedle
commented
7 years ago k0ste, thank for reply. I would say, that hardware and kernel version are obvious from my original report and trace:
Hardware: [ 232.398469] Hardware name: PC Engines APU/APU, BIOS 4.0 09/08/2014 https://pcengines.ch/apu.htm
Environment: What would you like to know exactly? IMQ is used as described to simulate halfduplex link (IMQ is hooked on PREROUTING and POSTROUTING of same interface) on embeded APU NIC, tc root disc using htb, wth ~ 4000classes, classified by u32 hash tables based on src/dst IPs.
Kernels: vanilla 4.4.26, 4.4.32, 4.4.38 + IMQ patch, ESFQ patch, Intel NIC drivers (e1000 project). We also tried all SUSE distro kernel patches for 4.4.38 - with no difference.
Kernel Config for 4.4.26 attached.
k0ste
commented
7 years ago Intel NIC drivers (e1000 project)
On apu.htm apu1 & apu1d4 (very uncommon hardware) devices have Realtek in specs...
ESFQ patch
Why?
litinoveweedle
commented
7 years ago As the our distribution is used on variety of hardware. From high end Intel servers used as core routers to APU platform used as CE routers. Problem occurs (as in the original bug report) only on low performance CPU (as APU, but not limited to it)
litinoveweedle
commented
7 years ago Regarding ESFQ scheduler, it has huge improvement for sfq - you can selectively hash flows on SRC or DST IP and therefore be more fair to you customers.
EDIT: Just realized, that tc flow could be used for standard SFQ to change flow tuples: http://man7.org/linux/man-pages/man8/tc-flow.8.html
But anyway we tested IMQ without ESFQ (using only SFQ for HTB classes), with same CPU soft lockup results.
k0ste
commented
7 years ago EDIT: Just realized, that tc flow could be used for standard SFQ to change flow tuples: http://man7.org/linux/man-pages/man8/tc-flow.8.html
Yes, you should not use esfq because is unmaintained for ~ 10 years and any filters you can get from flows and use with sfq, drr, fq_codel, etc (see man tc-sfq,tc-drr, etc).
But anyway we tested IMQ without ESFQ (using only SFQ for HTB classes), with same CPU soft lockup results.
Good. I noticed that too - people have issues with old hardware. What advice? Try last kernels or stay on "this last version where works good", because I think some in kernel changed, because modern systems very fast and kernel should not be a limiter for performance. And may be some of this receipts you should try. Good luck.
litinoveweedle
commented
7 years ago Thanks for reply. Anyway the APU platform is not old at all. It is actually (especially APU2) recent one, and really powerful for CER usage! Regarding mentioned IMQ patches, we tested them all, no improvements.
When we use completely same htb qdisc structure, attached to physical interface egress, it works flawlessly. If the same htb qdisc is attached to IMQ in that "halfduplex mode" it fails quickly. So it doesn't seem to be really related to HW, or kernel generic functions. Interestingly it will not fail on all APU platform nodes, even when having completely same configuration. Therefore we suspect, that this bug is in IMQ and is traffic related.
My assumptions is some kind of deadlocking will happen probably on fragmented IP packet or related to SMP. IMQ is known from past by similar issues (I found someone posting about same CPU softlockup with same IMQ halfduplex setup as we use - in IMQ for 2.X kernels!)
So right now, we have have only two options.
Support IFB to get better support for NAT and get as author state: Advantage over current IMQ; cleaner in particular in SMP; with a lot less code.
Support IMQ to get refactored and get rid of these bugs without required huge changes in our network setup.
I am currently anyway evaluating completely new traffic management with IFB / HFSC / AQM (fq_codel/cake) in our network - to test is as replacement of IMQ / HTB / (E)SFQ. It is painful process to change qos/shaping in ISP size network, but maybe we just needed such impulse to take fresh approach to traffic management. :-)
Kind regards L.
k0ste
commented
7 years ago that this bug is in IMQ and is traffic related.
Traffic related bug is most hard to reproduce. Yes, maybe for your wireless clients dtaht/sch_cake is better.
So what you mean half-duplex IMQ? May be you start to show your scripts?
litinoveweedle
commented
7 years ago I am afraid, that everything relevant was already posted in the original bug report - same as for hardware. I am not using script but more dynamic tc commands generation based on provided data files. But I would say that example bellow could help you to understand, what I meant by half-duplex line shaping.
iptables -t mangle -I PREROUTING -i eth0 -j IMQ --todev 0
iptables -t mangle -I POSTROUTING -o eth0 -j IMQ --todev 0
ip link set dev imq0 up
ip link set dev imq0 qlen 100
ip link set dev imq0 mtu 1500
tc qdisc add dev imq0 root handle 1: htb default 20
tc class add dev imq0 parent 1: classid 1:1 htb rate 32mbit ceil 32mbit burst 32k
...
a lot of complicated hash filters and classes stuff here....
...
each class has esfq or fifo qdisc attached BTW: we tested succesfully ifb on same job using this hooking:
ip link set dev ifb0 up
ip link set dev ifb0 qlen 100
ip link set dev ifb0 mtu 1500
tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
tc qdisc add dev eth0 root handle 1: htb
tc filter add dev eth0 parent 1: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
tc qdisc add dev ifb0 root handle 1: htb default 20
tc class add dev ifb0 parent 1: classid 1:1 htb rate 32mbit ceil 32mbit burst 32k
...
a lot of complicated hash filters and classes stuff here....
...
each class has esfq or fifo qdisc attachedIs there any information I can share with you? Any thoughts what could cause CPU soft lock?
k0ste
commented
7 years ago ip link set dev imq0 qlen 100 ip link set dev imq0 mtu 1500
I'm not sure. I don't see this ever, this is drop of performance.
iptables -t mangle -I POSTROUTING -o eth0 -j IMQ --todev 0
You really need this? You can attach qdisc on eth0 directly.
litinoveweedle
commented
7 years ago Root htb qdisc queue is as far as I understand used only for classification, and sfq or codel have much more smaller queues in leaf qdisc anyway. 100packets is more than enough on half duplex 32mbit link. Also we have no jumbo frames, max MTU is 1500b, so I do not see any performance impact here - maybe you can explain.
Regarding the hooks, in some of the our cases we are selecting only some DST/SRC IP to go to imq and it is much more simpler to do it in both directions by iptables, than use TC filter for egress and iptables for ingress. But it could be only performance hit, not cause of CPU lockup.
When running our routers in normal traffic, we are not able to saturate the CPU to more than about 5% per core. CPU soft lockup happen disregarding to load, also in low peak hours. The problem is definitely somewhere else.
With all regards CPU softlock is programmatic error, not problem of usage or load. And as recent patches suggested above by you shows, there are/were such issues in imq code.
P.s.: We did also tried numqeues and precise stat imq module options, without any improvement to stability.
k0ste
commented
7 years ago The best way for ISP - vlan per client. Shape on vlan to clients, instead matching IP addresses if you can do it on L2.
If you don't want change anything may you looking to your hardware? For last 3 years I use HP/DELL servers with BCM5720 adapters. And don't use Intel with e1000e - this is peace of shit. I not have routers lower than 4.8 for now, this is example - it's not locks:
[k0ste@R1 ~]$ uptime
14:41:19 up 83 days, 22:22, 5 users, load average: 0.18, 0.18, 0.34
[k0ste@R1 ~]$ uname -a
Linux R1 4.8.13-1-ARCH #1 SMP PREEMPT Sat Dec 10 14:56:05 +07 2016 x86_64 GNU/Linux
[k0ste@R1 ~]$ sudo iptables -nvL -t mangle | grep IMQ
363M 224G IMQ all -- vlan666 * 0.0.0.0/0 0.0.0.0/0 IMQ: todev 0
8438M 3392G IMQ all -- ppp0 * 0.0.0.0/0 0.0.0.0/0 IMQ: todev 1
206M 170G IMQ all -- Tunnel1 * 0.0.0.0/0 0.0.0.0/0 IMQ: todev 2This router works with ipt_netflow and ipt_ndpi and HTB/SFQ attached of course.
Your BIOS options for hardware all in Performance mode? aspm/vtx/vtd/c-states is disabled? CPU governor in performance mode? tx ring for ethernet interfaces adjusted? lro/tso/gro/txvlan/rxvlan/tx/rx/sg is disabled? Interrupt moderation, Flow Indirection and Recieve-Side Scaling is configured fine?
I have report - runned 4.9 - 2 days, over 3Tb routed - works fine.
litinoveweedle
commented
7 years ago Don't get me wrong, but I did not reported poor IMQ performance. I did reported locking issue with module. Maybe in SMP, maybe in IP fragment handling... Although I appreciate your help with performance suggestions for our network, these are not related to the bug.
Actually what you said is: Look, I have no problem and if you do, it is because of your setup. :-) Well our HW doesn't cause the CPU to lock for 30sec and we did prove that, when used the same HTB structure just with IFB instead of IMQ - without any problem in our production.
I am not really sure, if by continuing this discussion we will be able to solve this bug.
BTW:
k0ste
commented
7 years ago I am not really sure, if by continuing this discussion we will be able to solve this bug.
Me too.
as our network is complicated structure, we simply can't use central shaping.
As a final council can recommend ipt_ratelimit, I use it for small and need_deploy_shaper_fast routers.
iptables -I FORWARD -m ratelimit --ratelimit-set 32Mbit --ratelimit-mode dst -j DROP
iptables -I FORWARD -m ratelimit --ratelimit-set 32Mbit --ratelimit-mode src -j DROP
echo -e "+198.19.15.14,192.168.16.193 33554432" > /proc/net/ipt_ratelimit/32Mbit #your clients hereGood luck.
litinoveweedle
commented
7 years ago Thank you for you help, but this was not StackOverflow question how-to shape our network. This was originally bug report for IMQ. Honestly we expected a bit different approach - for example that you can add some tracing into code and we will test it in production to isolate problem etc... We never expected that you will tell us to get rid of IMQ as an answer.
We simply wanted to stay with IMQ, because any system change in the size of our network is rather complex to deploy and IMQ was doing the job in the past without glitch. But if we need to change shaping globally anyway, then we decided to give a try to IFB.
So if you see no point for fixing the bug for other users, please be free to close this issue as unresolved.
k0ste
commented
7 years ago I recommend to you - try to use 4.8+ - you ignore this. I tried to pull you out of your class configuration, network design - you post "this is obviously, not performance issue". I almost out of skin, but this is useless. Further you - itself.
Hello,
we use IMQ for long time since kernel 2.6.X for shaping half duplex radio links between our linux routers. We do that using iptables mangle table and hooking PREROUTING input and POSTROUTING output via same physical ethernet device into IMQ device and than attaching complex HTB qdisc into it. It works OK on 2.6 and 3.4 kernels, unfortunately we getting crash on every single 4.4 version we tried. (4.4.26, 4.4.32, 4.4.38) Under real traffic such crash is almost immediate (please see traces for 4.4.32 and 4.4.38)
Was there any change for IMQ which prevents such usage, or is it some IMQ/kernel bug? Thank you for any help.