search

Yelp / python-gearman

Gearman API - Client, worker, and admin client interfaces
http://github.com/Yelp/python-gearman/
Other
242 stars 124 forks source link

update connection multiplexing code to use epoll when available, fall back to select when not available #30

Closed kylemcc closed 11 years ago

kylemcc commented 11 years ago

The current implementation is limited by the max number of FDs supported by select (really the max fileno). On most systems this is 1024. The updated implementation in the PR will use epoll where supported, giving better performance when multiplexing lots of connections, and supporting more connections. While I'm generally not multiplexing 1000 connections to gearman, I have noticed on production systems where connections are being opened and closed frequently, it's easy to have fileno's > 1024, making this library unusable in a lot of cases.

The polling code falls back to select when epoll is not available (non linux). A kqueue implementation may be beneficial for those using BSD, but I'll leave this to someone else for now.

klange commented 11 years ago

Thank you for the patch. We'd like to have some additional documentation for the gearman.io module before we merge your pull request; docstrings for each method would be greatly appreciated.

kylemcc commented 11 years ago

Of course, I'll add some docs and update this. Thanks!

kylemcc commented 11 years ago

I updated per your suggestions - docstrings in the geaman.io module, and I mocked the _register_connections_with_poller. Let me know if there's anything else you'd like included here.