Interested in creating epoll patch?

mspanc commented 8 years ago

Hello,

I am building a network app that strongly relies on GLib/GStreamer stack. However, I would like to give it a boost by switching to epoll. I've noticed abandoned epoll branch in your repo.

Would you have been interested in finalizing this project and making working, stable version of GLib with epoll? I can sponsor this somehow.

mzabaluev commented 8 years ago

@mspanc The conflicting changes do not seem to be much divergent from the last time I was doing this. Do I understand correctly that you need that branch ported to the latest stable branch of GLib? What kind of poll size is expected in your application, is it large enough for the changes to have noticeable effect?

mspanc commented 8 years ago

Hello @mzabaluev

I want to run streaming service based on gst-rtsp-server and it can easily hit 10-50k simultaneous connections in a peak so I think that's already range in which regular poll will be a bottleneck (correct me if I am wrong).

So yes, I am basically interested in having a fork of GLib that has enabled epoll.

Have you tried to merge this into main GLib code base? If yes what was the reason it is not a part of GLib? Are there any reasons why GLib should stick to poll? (i don't see any)

mzabaluev commented 8 years ago

Have you tried to merge this into main GLib code base? If yes what was the reason it is not a part of GLib?

It has been submitted as GNOME bug 699132, but nobody has seen it necessary to review the patch. I guess there is no mainstream need for it, as GLib is mostly used in application stacks.

Are there any reasons why GLib should stick to poll?

If it ain't broken, don't fix it :) poll is portable across POSIX systems, and it's been working well enough for most usages of GLib.

mzabaluev commented 8 years ago

After rebasing to recent master, a few regression tests fail. Will investigate.

mzabaluev commented 8 years ago

A regression test added since the branch was last submitted exposes an API break: if multiple polls were added on one descriptor with different priorities and different event masks, the priority in g_main_context_query decides which event mask the corresponding output GPollFD will get. The code on the pollers branch does not preserve the original poll entries' priorities.

Fixing this would require more work than I'm ready to do on my own time currently, I'm afraid. Pity that such a corner case gets in the way, because g_main_context_query in that branch is no longer used in GLib's implementation of the poll loop, and is only needed for integration with foreign event loop implementations (which the branch aims to solve in a different way).

mspanc commented 8 years ago

Well I don't mind if this fork won't be 100% compatible.

arteymix commented 8 years ago

I work on Valum, a web micro-framework based on libsoup-2.4 and gio-2.0 and I would really like to see epoll eventually powering GLib.

@mzabaluev What kind of issues have you encountered when you tried to rewrite GMainContext upon epoll? is maintaining priorities per file descriptor such a pain?

In the meantime, it is still possible to plug a polling function into GMainContext, so I though I could wrap epoll with a file descriptor cache. How well do you think it would perform?

mzabaluev commented 8 years ago

I have pushed branches:

pollers, rebased against recent master;
pollers-2-46, rebased against glib-2-46.

@arteymix

is maintaining priorities per file descriptor such a pain?

It should be doable: the members of pollfd lists in GPollBin could include the original priorities, to be considered when composing the output array of g_main_context_query.

In the meantime, it is still possible to plug a polling function into GMainContext, so I though I could wrap epoll with a file descriptor cache. How well do you think it would perform?

There's expectation that a GPollFunc callback is able to poll an arbitrary set of file descriptors passed into the function. Any implementation would have to scan the entire input array, which would negate part of the overhead reduction provided by epoll_wait.