search

jjlee / mechanize

Stateful programmatic web browsing in Python, after Andy Lester's Perl module WWW::Mechanize .
http://wwwsearch.sourceforge.net/mechanize/
618 stars 123 forks source link

Threadsafety improvements using thread local storage #64

Open dhionel opened 12 years ago

dhionel commented 12 years ago

When mechanize is used in a heavily threaded context, serious contention and threadsafety concerns arise, related to the use of a global _opener object.

The proposed changes place _opener in thread.local storage, eliminating the contention. They has been tested in a scenario where each thread only uses its own mechanize objects, that's enough to support a stress test framework like multi-mechanize (http://testutils.org/multi-mechanize/) in a heavily threaded context. With those changes in the general case, if a thread uses mechanize objects created by another one it would use its local _opener object. If that is ok, there should be no issues.

Complete threadsafety is required for the most general case, and more testing. In the restricted scenario the proposed changes seem to be enough for a stable behaviour.

jamesbroadhead commented 7 years ago

Thank you for your contribution to mechanize!

Following the process in #117, future work on mechanize will be occurring here: https://github.com/python-mechanize/mechanize.

Please re-file your PR there (where it will get attention, and hopefully merged)