wifidog still eating lots of memory

[gareth41]

I've been running wifidog in a hotel environment and monitored it over 2 days until it crashed, i'm using the latest release 1.2.1 on the air-router with openwrt bb 14.07. At any given time there were atleast 5 active logins, and in during peak times this went upto almost 20. Wifidog ran for 2 days 15 hours before crashing. The device starts off with about 7900kb free memory and this drops quite quickly in the first few hours under heavy load to about 5500kb free, it evens out after that and gradually and slowly drops down to < 1000kb over the next couple days. The router never does actually run out of memory but wifidog seems to stop once the memory gets too low, and all the memory is freed again once wifidog stops, > 8000kb is free afterwards. I also had a bash script set up to take logs of /proc/(pid of wifidog)/status and report the VmSize and VmPeak values every 10 minutes via wget to the auth server, starts at 9369Kb for both values and progressively climbs to a VmSize of 16926kb and a VmPeak of 69624kb.

[mhaas]

That's.. not good. We do have a load testing script which I used to find some problematic code sections, using automatic tools like Valgrind. Perhaps your setup is not covered by the scenarios built into the load tester. What features of Wifidog do you use? Can you perhaps post a config file? Log files? Of course, with privacy-related details redacted.

[mhaas]

See https://github.com/wifidog/wifidog-gateway/tree/master/contrib/load-tester - I do have some remaining changes on my hard disk. I hope I will get around to committing these soon.. in the mean time: the scripts should work as-is.

edit: Perhaps you should also try latest trunk, although I am not sure if we have actually fixed any memory leaks.

[gareth41]

I think i've found out what's going on, this appears openwrt related and the leak occurs each time a new pthread is created in the main server loop. https://github.com/wifidog/wifidog-gateway/blob/master/src/gateway.c#L476

I changed the code to run single threaded and there doesn't appear to be any memory problems, just the server is slower and may occasionally hang for a few seconds under heavy load, ie lots of http requests in quick succession.

I tested the original code and hammered the server with thousands of requests whilst watching the avail memory gradually reduce! There were no clients logged in.

In single threaded mode avail memory doesn't change by very much at all

[mhaas]

Do you have a script used to hammer wifidog with requests?

[KazukiShimada]

Hi to all.

I suspect uclibc causes these crashes. I changed openwrt version in my device, whr-g301n http://wiki.openwrt.org/toh/buffalo/whr-g300nv2 from cc to latest trunk, then crashes were disappeared.

As Michael already suggested, I also recommend that you try openwrt trunk.

[acv]

We could look at setting up a thread pool and statically allocating a number of threads. That would avoid leaks from uClibc.

[acv]

I'm flagging this as an enhancement because it's to work around uClibc.

The simplest fixes I can think of is to add a threading mode configuration key that would take one of 3 values:

• Synchronous -- So single thread as @gareth41 hacked his version to do.
• Default -- One thread per request.
• Pre-spawn -- Spawn 1 or more worker thread and just have them all run the accept() loop. I'm almost sure that socket API guarantees one and only one thread will be successful on the accept() call. It does for processes.

In the pre-spawn case, there would be a further mandatory key to configured the number of threads to spawn. I figure 3-5 should provide acceptable throughput in all but the busiest access points.

[gareth41]

@mhaas, I wrote a bash script to call wget inside an infinite while loop. Have also started the process of removing libhttpd and replacing it with microhttpd, its a difficult process as i'm still learning as I go, im still fairly new to c. libhttpd is also very old from 2005 and hasn't been updated since.

[mhaas]

Hey @gareth, I just saw your emails from september.

Is it possible for you at all to try with the new OpenWrt release? Maybe there was a fix to uclibc re: threading and memory leaks.

[KazukiShimada]

@gareth41 It's interesting. Any advance?

[KazukiShimada]

@gareth41 Hi, this guy wrote he plans to support libmicrohttpd to nodogsplash which is derived from wifidog gateway. https://github.com/openwrt/packages/pull/2126#issuecomment-164567789 It might be helpful to contact him.

Thanks in advance.

[strong46066999]

pthread_create a thread need about 2M memory in my x86 ubuntu, use thread attr set thread stack smaller. pthread_create eat cpu, use thead pool well better.

[lynxis]

@gareth41 @KazukiShimada libmicrohttpd is already integrated into nodogsplash v2. take a look on the nodogsplash v2 branch. not all features are supported, but the common ones.

[gareth41]

I succeeded in integrating libmicrohttpd after some time and a lot of research, it solved the memory leak pthread issue but increased the memory footprint from about 2mb to 9mb resulting in some routers with low ram not being able to run it, I still have the code if anyone wants it. I ended up pre spawning 4 pthreads with libhttpd instead and left it at that which solved the memory leak