Kernel panic in Ubuntu 14.04

[hotice]

douane-dkms causes Kernel panics in Ubuntu 14.04 64bit. This seems to occur randomly, while browsing some websites in Firefox for instance (so not on boot).

I'm using the Ubuntu Linux 3.13.0-24-generic Kernel.

[zedtux]

Short message to inform that I'm waiting my compute to crash (with Kdump enabled) ... but since 2 days it is still working ...

[zedtux]

As of today, and as far as I know, 3 people are impacted by the kernel freeze issue.

I'm fine to have it reported but it doesn't help to have multiple issues opened in the different repos (and I understand the issue to know where to open the issue, even if I have written a wiki page about to know where to open an issue).

This said, I'm going to close any other tickets opened regarding kernel freeze and redirect people to this issue.

[ho-mo]

My system crashed now agian ! what your distro ?! I am use Ubuntu 14.04 x64 execuse me I dont know , That this Problem related to douane-dkms ! This in my kenl.log Sep 21 01:36:50 double kernel: [ 1528.529311] r8169 0000:02:00.0 eth0: link down Sep 21 01:36:50 double kernel: [ 1528.529350] r8169 0000:02:00.0 eth0: link down Sep 21 01:36:50 double kernel: [ 1528.529363] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 21 01:36:52 double kernel: [ 1530.725048] r8169 0000:02:00.0 eth0: link up Sep 21 01:36:52 double kernel: [ 1530.725064] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sep 21 01:42:29 double kernel: [ 1867.586261] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.602276] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.615575] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.628695] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.876567] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.889773] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.903135] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.916129] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:29 double kernel: [ 1867.917577] douane:414:push: BLOCKED PUSH: process_path is blank. Sep 21 01:42:36 double kernel: [ 1874.247981] douane:457:push: Message ignored as Netfiler socket is busy. Sep 21 01:42:36 double kernel: [ 1874.248071] ------------[ cut here ]------------ Sep 21 01:42:36 double kernel: [ 1874.248112] kernel BUG at /build/buildd/linux-3.13.0/mm/slub.c:3365! Sep 21 01:42:36 double kernel: [ 1874.248158] invalid opcode: 0000 [#1] SMP Sep 21 01:42:36 double kernel: [ 1874.248190] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 xt_CHECKSUM iptable_mangle bridge stp llc ebtable_nat ebtables douane(OF) pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) bnep rfcomm bluetooth binfmt_misc ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_LOG xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm gpio_ich snd_page_alloc snd_seq_midi coretemp snd_seq_midi_event kvm_intel kvm snd_rawmidi serio_raw snd_seq lpc_ich snd_seq_device snd_timer snd ppdev parport_pc lp soundcore parport mac_hid btrfs xor raid6_pq libcrc32c hid_generic usbhid hid r8169 i915 mii floppy video i2c_algo_bit drm_kms_helper drm

doual core 3Ghz Intel 2 GB RAM Intel Onboard Graphic gnome-shell interface when i start douane with this command sudo service douane start and Use foxyproxy on firefox 127.0.0.1:9050 (socks Tor proxy) and i went to this sites jadi.net , https://epayment.bmi.ir/Remain system crashed

[zedtux]

execuse me I dont know , That this Problem related to douane-dkms !

No worries ;-).

I'm using also Ubuntu 14.04 64 bits, but since I have installed kdump and crash in order to collect crash information, my system haven't crashed. If you could try to follow the instruction on my blog : http://blog.zedroot.org/linux-kernel-debuging-using-kdump-and-crash/. In case you succeed, send me the crash file and I can analyse it.

[arlsr]

Doune is also unusable for me due to system freezes/crashes.

[zedtux]

Thank you @arlsr to join us. As of now, multiple people has issues, but no one of you are able to price me wugh a crash file. Without it, I can't do anything.

May I ask someone of you to follow the instructions I have already provided and send me the crash file!?

Thank you in advance.

[zedtux]

Any chance @arlsr that you provide me with a crash file as described on my blog ?

[arlsr]

A few reboots have damaged my system in the past, I'm hesitant to try again. I may in the future when I have more time though. I'm considering stripping Douane down to the fundamentals to see if I can find the cause of the crashes.

[zedtux]

My blog article is describing how to use kdump which will boot a second kernel when the main will crash so that your computer is going to move to the new running kernel and don't loose data.

Anyway what about using a Virtual Machine ?

[arlsr]

I've set up kdump. My computer doesn't reboot when it crashes though, it hangs and no dump is created. Even when triggering the test crash.

[themighty1]

Same here. I didnt bother to report though because I thought that was the case for everyone. I ran Douanne on a fresh 14.04 install in Virtualbox. It would take only a few minutes before the machine locked up.

[zedtux]

That's wired. I gave a try to my article after having published it in order to be sure all is working well. Your computer should not restart and you should have a file in /var/crash/ (something like this).

[arlsr]

There's only kexec_cmd in /var/crash.

cat /proc/cmdline shows crashkernel=384M-:128M

$ cat /sys/kernel/kexec_crash_loaded 
1

I have a large amount of memory and no swap, that might be the issue.

[zedtux]

I've switched to Debian and I'm considering trying Douane in order to finally fix this issue.

[junkvolny]

Yesterday, all I had to do for system freeze was run web browser. Today it worked somehow without freeze for quite some time, but I was able to force system hang with following:

1. open web browser (i used firefox), that has allow rule in place, web page loads
2. stop the daemon with #sudo /etc/init.d/douane stop
3. If not already system hang, repeat deamon start/stop few times, mix in few refreshes in web browser. Works like charm (irony: system freeze works).

Maybe I should try to watch log of kernel module, not just daemon with #tail -f /var/log/douane.log but not sure how and if it's enabled by default, in any case, if you can reproduce freeze with these steps, you might be better at it.

[junkvolny]

I did huge amount of searching today for possible alternatives today, and I found GPLed project http://sourceforge.net/projects/leopardflower/ that uses iptables and libnetfilter_queue, thus it does not require kernel module (!). I must admit I have not tested it much. I think all these troubles are because kernel module is so hard to debug.

Your "Douane personal firewall" and mentioned "Leopard flower per-application (personal) firewall" are the only application firewalls for current Linux/Ubuntu out there, I am afraid.

[arlsr]

leopardflower has plenty of issues of its own sadly, but none that crash the host.

[zedtux]

As I'm now using Debian (no more Ubuntu), I will now test it with Debian.

[zedtux]

Alright, I have a working Debian in VirtualBox (I have updated the dependencies wiki page for Debian) and I got the crash and the crash dump file has been generated.

I'm giving a look now.

[zedtux]

OK now I see the issue. The freeze come from an issue in the kernel module communication with the daemon.

Simple things like ping are working like a charm while a lot of activity is making the freeze happening. I will try to solve this in my free time...

[zedtux]

@junkvolny @arlsr @ho-mo I have created the branch fix-kernel-freeze where I did modification which allowed me to use Douane in a VM. After 15m the VM slow down and the douane-dialog process was using 100% of the CPU so I need to have a look to this but at least the kernel freeze is gone for now.

[junkvolny]

I tested today, on same virtual machine that was freezing before, and yes, it seems the kernel freeze is gone, well done.

I didn't test long run, for dialog-process freeze, but to be honest, dialog is least important component - it cannot crash kernel + if it crashes, it does not stop daemon from enforcing rules, hopefully.

[zedtux]

... if it crashes, it does not stop daemon from enforcing rules, hopefully.

You're right, it do not.

I have already identified in the issue Douane/douane-daemon#6 that the daemon is increasing the memory usage and after an amount of time is using all the memory so I need to fix it.

Anyway thank you @junkvolny for your confirmation. I would like to have another confirmation before to close this issue. Can one of you @arlsr, @ho-mo or @hotice confirm that the kernel freeze is gone and I can close this issue ?

[arlsr]

@zedtux I got a freeze while installing, probably because I didn't remove the old version but after that no freezes in the several hours I've tested so far. I'll give it more time and report back. Great work fixing it if it is fixed!

[zedtux]

:+1: thank you @arlsr. I would love to get some feedback from you :)

[arlsr]

No issues after 24 hours, I think it's fixed.

[arlsr]

I had a freeze while sending a lot of traffic. I've unloaded the kernel module again.

[smarek]

I finally struggled with this too, on Debian 8 x86_64 with kernel 4.4

I can reproduce the problem with 100% success rate, by running nmap -T4 -A MY_PUBLIC_IP

Sincerely, it seems my kdump config does not work currently, so I'll try to obtain proper crashdump, so far I have only photo of screen of crashed machine, with stack trace and point to netfiler_packet_hook.isra.5 within douane kernel module

[smarek]

I've also tried with patch provided in branch fix-kernel-freeze but the behavior remains the same

[zedtux]

There should be a problem of something freed to earlier (especially when pushing messages from the kernel space to the user space). But this needs to be debug with gdb and crash.

Then the next issue to solve will be the daemon memory usage. The daemon is storing too much messages coming from the kernel space and no deleting them apparently.

[zedtux]

@smarek the fix-kernel-freeze branch has been merged to master and should be deleted.

[smarek]

@zedtux thanks for info, I'm trying to get hold of crashdump, however there is no /proc/vmcore which is apparently needed for kdump-tools (try kdump-config savecore), do you maybe have any idea how to get kernel crash dump working or debug why it is not working?

[zedtux]

@smarek have you tried to follow my blog article http://blog.zedroot.org/linux-kernel-debuging-using-kdump-and-crash/ ?

[smarek]

@zedtux yes, kdump-tools configured (except for the savecore command not working), ubuntu linux-crashdump is not available, forcing crash will result in panic on tty1 not in dump in /var/crash (this folder contains only kexec_cmd file). So I cannot say that your blog helped me, but you've mentioned migrating to Debian, that's why I asked in the first place

[zedtux]

I did that blog article while I was running Ubuntu, and since I've migrated to Debian. I should take the time to redo this and do another blog article for Debian this time... So I don't know how to help you now.

[zedtux]

Regarding the linux-crashdump from Ubuntu, it's nothing else than a meta package (see http://packages.ubuntu.com/xenial/linux-crashdump) installing the packages grub-pc, kdump-tools and crash. The apport package is an Ubuntu specific package which we don't care about.

So the Debian equivalent of sudo apt-get install linux-crashdump is apt-get install grub-pc kdump-tools crash

[zedtux]

@smarek here is another blog article explaining how to install KDump on Debian : https://www.bentasker.co.uk/documentation/linux/312-installing-and-configuring-kdump-on-debian-jessie.

I just tried it but unfortunately, I can't get my system booting a kernel after the panic ... I don't know yet why.

[zedtux]

@hotice, @pavlinux @ho-mo, @arlsr, @junkvolny and @smarek I have merged @Guimli's PR this morning which fixes a lot of issues and seems to have fixed the kernel freezes! (I tested it over the week-end without freezes).

Can you please pull the master branch of this repo, rebuild the module and give me your feedbacks?

[hotice]

I'm not using Ubuntu 14.04 any more. In Ubuntu 16.04, douane-daemon and douane-dialog fail to build. Here are the builds in my test PPA: https://launchpad.net/~nilarimogard/+archive/ubuntu/test2/+packages?field.name_filter=&field.status_filter=published&field.series_filter=xenial

Looking at my earlier comments, just installing the dkms module doesn't trigger the issue, the daemon must be running...

[zedtux]

You're build issue is coming from your packing. The daemon and dialog needs to be cloned using the --recursive flag in order to download the sub modules too. See the documentation https://github.com/Douane/Douane/wiki/Compilation#the-daemon.

[hotice]

Oh, I missed that, sorry. Everything builds now. However, the douane-dialog which is supposed to let me allow access for apps to use the Internet, never shows up. I tried restarting the session, etc., nothing seems to work. Maybe something changed in Ubuntu 16.04 that prevents it from working?

[zedtux]

Nope, it's working for me with that version of Ubuntu.

Can you please have a look at the /var/log/douane.log and /var/log/kern.log files? You can also run the dialog progress in debug mode using the -D flag.

I'm there to help you if you need :)

[hotice]

It looks like the only way to get douane-dialog to show up is to run it manually, but then it segfaults after about 1 second. Even in debug mode, it doesn't display anything. I did get it once to display an output (without the debug flag):

**
Gtk:ERROR:/build/gtk+3.0-6ZPWga/gtk+3.0-3.18.9/./gtk/gtkcssinheritvalue.c:33:gtk_css_value_inherit_free: code should not be reached
**
Gtk:ERROR:/build/gtk+3.0-6ZPWga/gtk+3.0-3.18.9/./gtk/gtkcssinheritvalue.c:33:gtk_css_value_inherit_free: code should not be reached

(it might not be related to the segfault, I don't know...)

Edit: I'm not seeing anything that could point out to what's wrong in douane.log but anyway, here's the log: https://transfer.sh/k23NU/douane.log

[zedtux]

Well that's strange. As I said, I have a VM of Ubuntu 16.04.1, all up-to-date and I have the dialog showing up.

Now it is true that I'm always running the app in debug mode, launched manually, it's a long time ago I didn't tried the normal way.

The dialog process is a little buggy, but it shouldn't crash the next second after you've started it. I can see in the log you have accepted at least a rule. The worst I saw is the dialog is freezing and I had to kill it and restart it again.

Anyway I need time to improve the daemon process (consuming memory more and more overtime) and the dialog process needs to stability.

Douane is not yet ready to be used everyday, my comment to you @hotice was to show you the progresses done on the kernel module, meaning the project is not dead :)

[zedtux]

@Mato-Z can you please give more information? What is the ouput of lsb_release -a and uname -a?

[Mato-Z]

@zedtux Thanks. uname -a: Linux mato-pc2 4.15.0-kali2-amd64 #1 SMP Debian 4.15.11-1kali1 (2018-03-21) x86_64 GNU/Linux lsb_release -a:

Distributor ID: Kali
Description:    Kali GNU/Linux Rolling
Release:    kali-rolling
Codename:   kali-rolling

[zedtux]

This repo has moved to GitLab and this issue has been imported. You can access it here.