MHD_run - need blocking variant

[jelmd]

As far as I can see, there is no way to use libmicrohttpd in blocking mode. Using something like while (1) MHD_run(); is possible but consumes 100% CPU for no reason. So either MHD_polling_thread() or something like MHD_run() but with blocking enabled should be exposed . Otherwise one needs to duplicate all the work/code from daemon.c, which is really a big pain and kind of brain damaged.

[Karlson2k]

The simplest way to use MHD correctly is to allow MHD to run threads internally, by using MHD_USE_INTERNAL_POLLING_THREAD or MHD_USE_AUTO_INTERNAL_THREAD flags. MHD will start additional thread and will use select()/poll()/epoll() to watch sockets and process data when necessary. Other related symbols are: MHD_USE_THREAD_PER_CONNECTION and MHD_OPTION_THREAD_POOL_SIZE.

If you for whatever reasons need to run MHD in your own thread, you can start MHD without MHD_USE_INTERNAL_POLLING_THREAD, then use MHD_get_fdset(), MHD_get_timeout(), and select() to wait for events and then call MHD_run() or MHD_run_from_select() to process the data.

You can use supplied examples as m-m-m... examples. Header file contains useful doxy comments. Also you can check provided documentation.

[jelmd]

Yes, that's the problem: why to run 2 threads when one is completely sufficient? Duplicating all the work, what MHD_polling_thread() or the other mentioned functions already do isn't very smart, is it? The point is, AFAICS, everything is already there and would be a one-liner, if thoseh function[s] would be exported/would not be static.

[Karlson2k]

Separate thread is used just because modern applications are typically multi-thread applications, so MHD just offload application from managing separate thread. Moreover, single thread is not enough to completely load multicore system (read "most of modern systems"). Everything is handled by MHD as requested by application.

Anyway, if you want to run MHD in the single thread only you can easily do it. As I wrote in previous comment, you just need to use three MHD functions. Under GNU/Linux two functions is enough (if epoll FD is used). At the same time, you application can multiplex network polling with other sockets, used for other network activity.

For clarity: to use MHD in your own thread you need:

• portable way. Call in the loop MHD_get_fdset(), MHD_get_timeout(), select(), and MHD_run_from_select().
• GNU/Linux only. Get epoll FD and call the loop MHD_get_timeout(), and select()/epoll_wait(), and MHD_run().

That's it. I believe it isn't too hard to use it in this way. Everything is in the public API.

Of course we always have something in our to-do list so patches are always welcome. If you think that you can improve MHD and it will be useful for others, please send us a patch or create a PR.

You can join MHD official mailing list at https://www.gnu.org/software/libmicrohttpd/#mail to share your ideas or request any help with MHD.

[jelmd]

Hmm, your assumption seems to be wrong: If one uses microhttpd, IMHO one does not want to utilize all available cpu strands. At least in my case the opposite is needed: goal is to get a utility to serve http requests with the smallest possible footprint. And since each context switch especially on x86 system is very expensive, one should try to avoid this overhead e.g. by getting rid off unneeded threads/processes. Furthermore one should keep in mind that on modern system there is not only one process running, but several hundred. So each additional thread/process puts more pressure on the system because there are always more processes than available cpu strands. Yes, some apps may benefit from multi-threading, but definitely not all [need it] and there is no guarantee at all, that a multi-threaded app performs [much] better than a single-threaded one. The opposite might be the case and such an app may even have a negative impact on the whole system performance ...

Anyway, are there any reasons, why you do not want to expose the mentioned functions?

[Karlson2k]

Anyway, are there any reasons, why you do not want to expose the mentioned functions?

@jelmd I'm not sure that you are reading the answers, so I quote the answer:

For clarity: to use MHD in your own thread you need:

• portable way. Call in the loop MHD_get_fdset(), MHD_get_timeout(), select(), and MHD_run_from_select().
• GNU/Linux only. Get epoll FD and call the loop MHD_get_timeout(), and select()/epoll_wait(), and MHD_run().

That's it. I believe it isn't too hard to use it in this way. Everything is in the public API.

And additional quote:

If you think that you can improve MHD and it will be useful for others, please send us a patch or create a PR.

[Karlson2k]

Hmm, your assumption seems to be wrong: If one uses microhttpd, IMHO one does not want to utilize all available cpu strands.

Depends on the use-case. If one needs to efficiently serve network clients as much as possible then maximum utilization of system resources is required.

At least in my case the opposite is needed: goal is to get a utility to serve http requests with the smallest possible footprint.

That's the case where MHD fits perfectly. MHD is small and resource consumption can be limited by run-time configuration.

And since each context switch especially on x86 system is very expensive, one should try to avoid this overhead e.g. by getting rid off unneeded threads/processes.

Two points:

• When each thread is running on a separate CPU core, there is no context switches between threads.
• Each system call require context switch userspace/kernelspace and back, which is heavy as well. And while MHD was optimized to use the minimal number of syscalls, anyway more than 5-7 syscalls (the number is imprecise) are required to handle even small single HTTP request. You will hardly notice context switch between threads. You can benchmark by yourself.

Yes, some apps may benefit from multi-threading, but definitely not all [need it] and there is no guarantee at all, that a multi-threaded app performs [much] better than a single-threaded one.

That's correct. If the workload cannot be parallelized or application is badly designed, multi-core load may not bring any improvement. Luckily a typical MHD load is easily parallelized and MHD is designed well, so N core load increase performance by almost N ratio. You can test it on your own system by running supplied perf_get_concurrent.c and compare results with perf_get.c. The "external select" mode in perf_get corresponds to the single-threaded mode (with the single main tread only).

I want to repeat one more time: MHD is designed to run in the single thread mode as well as in multi-thread modes. The single-thread mode is available "out-of-the-box", i.e. in the public API.

[Karlson2k]

@jelmd New function MHD_run_wait() was introduced by https://github.com/Karlson2k/libmicrohttpd/commit/bf25ee3a3220596b83b6530b06d4106f2f0bfda1.

[Karlson2k]

@jelmd Does the new function suit your needs?

[Karlson2k]

MHD_run_wait() is available starting from GNU libmicrohttpd version 0.9.73.