newgil backport

[fdb125b4-c5ef-4da3-b52b-728ae9a0ef15]

BPO	7753
Nosy	@malemburg, @nascheme, @jcea, @pitrou, @briancurtin
Files	newgil-2.7.diff: Backport of newgil to trunk gil-2.7.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core', 'performance'] title = 'newgil backport' updated_at = user = 'https://bugs.python.org/rcohen' ``` bugs.python.org fields: ```python activity = actor = 'jmehnle' assignee = 'none' closed = True closed_date = closer = 'lemburg' components = ['Interpreter Core'] creation = creator = 'rcohen' dependencies = [] files = ['15967', '15974'] hgrepos = [] issue_num = 7753 keywords = ['patch', 'needs review'] message_count = 32.0 messages = ['98122', '98134', '98139', '98140', '98143', '98150', '98156', '98158', '98164', '98165', '98166', '98167', '98168', '98191', '98192', '98388', '98393', '98411', '98412', '98417', '98418', '98423', '98429', '98432', '98433', '98468', '98480', '98481', '98482', '98540', '98542', '105992'] nosy_count = 13.0 nosy_names = ['lemburg', 'nascheme', 'jcea', 'pitrou', 'schmir', 'kevinwatters', 'Omnifarious', 'jnoller', 'twhitema', 'Ringding', 'brian.curtin', 'rcohen', 'jmehnle'] pr_nums = [] priority = 'high' resolution = 'rejected' stage = 'patch review' status = 'closed' superseder = None type = 'performance' url = 'https://bugs.python.org/issue7753' versions = ['Python 2.7'] ```

[fdb125b4-c5ef-4da3-b52b-728ae9a0ef15]

This is a backport of the newgil work to the 2.7 branch (otherwise known as trunk.) Patch is against r77678.

Created by diffing r76193 (last in the newgil branch) against r76189 and applying the result. Generally applied cleanly and it looked as though only 1 reject needed to be massaged in. Passes all tests except for test_command, but the unpatched source also fails that test on my machine.

Perf numbers for linux on Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz Using benchmark from here: http://www.mail-archive.com/python-dev@python.org/msg43407.html -j0 2.7 : 14.93s, 15.71s, 15.03s newgil: 15.32s, 15.45s, 15.78s

-j4 2.7 : 22.44s, 22.30s, 22.19s newgil: 18.75s, 18.80s, 18.79s

[nascheme]

Looks like the 2.7 patch has a spurious change to /Lib/unittest/runner.py.

[malemburg]

A few things:

• The ceval_gil.h file is missing from the patch.

• Please check whether you can move these variable declarations inside the main loop function (as statics):

+/* This single variable consolidates all requests to break out of the fast path

• in the eval loop. / +static volatile int eval_breaker = 0; +/ Request for droppping the GIL / +static volatile int gil_drop_request = 0; +/ Request for running pending calls / +static volatile int pendingcalls_to_do = 0; +/ Request for looking at the async_exc field of the current thread state */ +static volatile int pending_async_exc = 0;

  • Instead of deprecating the set/getcheckinterval APIs, convert the arguments passed to these to the switch interval. The new APIs set/getswitchinteral are then not needed.

  • Please add the fallback solutions from the time module in case gettimeofday() is not available. You cannot assume that "all modern POSIX systems" implement that API - it was introduced in POSIX 2001 and Python 2.x still supports OSes that were released prior to that year.

  • Note that gettimeofday() et al. are rather slow APIs on most systems. It would be better to directly use a simple performance timer since we don't really need any timing accuracy in managing the GIL when run with threads. Have a look at the clock_gettime() API for ways of accessing such timers. There may also be other ways of accessing such times via pthreads.

  • I haven't looked at the timing details of the implementation, but please make sure that it works even if the clock interval is a few ms (while gettimeofday() provides us resolution, the system clock may still only tick in ms intervals - another reason to use CPU performance timers instead).

Thanks.

[malemburg]

See Tools/pybench/systimes.py for more comments on clocks, ticks and update frequency.

[pitrou]

• Instead of deprecating the set/getcheckinterval APIs, convert the arguments passed to these to the switch interval. The new APIs set/getswitchinteral are then not needed.

This is certainly not a good idea. The semantics are different, which is the reason why I created a separate API in the first place instead of recycling the old one.

Actually, there should be hardly any reason to want to change the interval with the new GIL implementation. It was required with the old GIL due to the fact that counting opcodes is a very poor way of estimating wall clock durations.

Note that gettimeofday() et al. are rather slow APIs on most systems

In my measurements under Linux it seemed to be as fast as clock_gettime() actually. But I don't know about other systems. I didn't want to complicate the patch (with several different paths) more than seemed necessary.

[nascheme]

Here is an updated set of patches that includes the ceval_gil.h file as well as the documentation changes.

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

> * Instead of deprecating the set/getcheckinterval APIs, convert the > arguments passed to these to the switch interval. The new APIs > set/getswitchinteral are then not needed.

This is certainly not a good idea. The semantics are different, which is the reason why I created a separate API in the first place instead of recycling the old one.

Actually, there should be hardly any reason to want to change the interval with the new GIL implementation. It was required with the old GIL due to the fact that counting opcodes is a very poor way of estimating wall clock durations.

Applications currently use those APIs for two reasons:

• they use lots of threads and need to make sure that they all run more often than they do with the default settings e.g. to make them respond faster to external inputs: in such a case, they use a low check interval number

• they don't use any threads and thus don't need to check for other threads at all or only rarely: in such a case they use a large check interval number

While the semantics are different, the result is similar and the actual numbers used are usually determined by experiment or rough estimate - noone expects the APIs to provide any kind of exact timing and it's not needed either.

Turning the existing APIs into no-ops is certainly not a good idea, since that will change application performance for both use cases.

Esp. the second use case (single threaded process) is a rather common one in Python - using multiple processes is often the more scalable approach for Python applications, than relying on multiple threads.

> Note that gettimeofday() et al. are rather slow APIs on most systems

In my measurements under Linux it seemed to be as fast as clock_gettime() actually. But I don't know about other systems. I didn't want to complicate the patch (with several different paths) more than seemed necessary.

Right, but clock_gettime() provides a better resolution.

Both APIs are relatively slow, so if you call them often, it's better to go with a different mechanism such as one built on CPU performance counters.

[pitrou]

While the semantics are different, the result is similar and the actual numbers used are usually determined by experiment or rough estimate - noone expects the APIs to provide any kind of exact timing and it's not needed either.

There's no reasonable formula for computing an absolute switching interval from the current "check interval" number. Besides, as I said, there's no demonstrated reason to change the interval with the new GIL.

• they don't use any threads and thus don't need to check for other threads at all or only rarely: in such a case they use a large check interval number

Well, if you don't use any threads you don't have to change the setting in the first place :-). You might want to do it in order to "micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this is unnecessary with the new GIL, since the interval is ignored when there's only one thread running.

Turning the existing APIs into no-ops is certainly not a good idea, since that will change application performance for both use cases.

The new GIL will change performance far more anyway. Trying to simulate the old behaviour is doomed to failure. If we want to keep the current performance characteristics, we'd better not backport the new GIL (which I'm fine with by the way).

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

> While the semantics are different, the result is similar and > the actual numbers used are usually determined by experiment > or rough estimate - noone expects the APIs to provide any kind > of exact timing and it's not needed either.

There's no reasonable formula for computing an absolute switching interval from the current "check interval" number.

Just use some conversion formula, e.g. switching interval in micro seconds = constant * byte code check interval. We can then determine a constant to match todays CPU performance.

Given the inexact nature of the check interval that's a very reasonable formula.

Besides, as I said, there's no demonstrated reason to change the interval with the new GIL.

Didn't I just point out two use cases ?

Perhaps I'm missing some feature of the new GIL. Is there some documentation for it somewhere ?

> * they don't use any threads and thus don't need to check > for other threads at all or only rarely: in such a case they > use a large check interval number

Well, if you don't use any threads you don't have to change the setting in the first place :-). You might want to do it in order to "micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this is unnecessary with the new GIL, since the interval is ignored when there's only one thread running.

Hmm, how do you determine that only one thread is running ?

What if an extension uses threads that are not under Python control, e.g. when embedding the JVM or when using a CORBA ORB ?

> Turning the existing APIs into no-ops is certainly not a good > idea, since that will change application performance for both > use cases.

The new GIL will change performance far more anyway. Trying to simulate the old behaviour is doomed to failure. If we want to keep the current performance characteristics, we'd better not backport the new GIL (which I'm fine with by the way).

I don't follow you.

If an application currently sets the check interval to a low value, Python should switch to other threads more often, ie. the switch interval needs to be low.

If an application currently sets the check interval to a high value, Python should switch to other threads less often, ie. the switch interval needs to be higher.

Both cases can be mapped to the switch interval, so why not do it ?

[fdb125b4-c5ef-4da3-b52b-728ae9a0ef15]

Thanks Neil for fixing up the patch.

As for Marc-Andre's comments, I'm happy to backport any further changes which happen on the py3k branch. I'd like to keep this as a strict backport, only changing things as necessary to get it to work with the older codebase.

Ross

[pitrou]

Just use some conversion formula, e.g. switching interval in micro seconds = constant * byte code check interval. We can then determine a constant to match todays CPU performance.

Well, there are two problems:

• opcode execution cost varies so wildly that trying to pick up an average would be meaningless (it can vary by many order of magnitudes between e.g. a POP_TOP and a CALL_FUNCTION calling a C extension method)
• the interval doesn't have the same effect in the new GIL as it has in the old GIL; in the old GIL, it merely gives the OS a chance to switch threads if it wants to (there are many missed opportunities, by a proportion which will be system-dependent); in the new GIL, thread switching is enforced which means it is not necessary to stop so often.

So we'd have a formula like:

new GIL interval = old GIL interval * opcode duration / proportion of missed switching opportunities in the old GIL

where two of the three factors on the right side are totally unpredictable :)

Perhaps I'm missing some feature of the new GIL. Is there some documentation for it somewhere ?

Almost nothing besides what is found in ceval_gil.h and in the following thread: http://mail.python.org/pipermail/python-dev/2009-October/093321.html Dave Beazley did a high-level presentation about it: http://www.dabeaz.com/python/NewGIL.pdf

> Well, if you don't use any threads you don't have to change the setting > in the first place :-). You might want to do it in order to > "micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this > is unnecessary with the new GIL, since the interval is ignored when > there's only one thread running.

Hmm, how do you determine that only one thread is running ?

Actually, I don't determine it, it's a side effect of how the GIL is implemented. If no thread is waiting for the GIL, the main thread isn't "asked" to drop it.

What if an extension uses threads that are not under Python control, e.g. when embedding the JVM or when using a CORBA ORB ?

These don't wait for the GIL, hopefully :-)

[malemburg]

On second thought, you're right: while sys.setcheckinterval() could be made to work with the switching interval, sys.getcheckinterval() could not be made to return anything useful.

Still, please have a look at the other issues mentioned. They apply to new GIL in general, regardless of whether implemented in 3.x or 2.7.

It would also be interesting to see the affect on other threaded applications, not just the "benchmark" mentioned on the python-dev thread.

Since there aren't all that many Python 3.x applications in the wild, it's probably a good idea to check in the patch and then run a few Python 2.x apps with and without it.

How does pybench react to the patch ? It's single-threaded at the moment, so a good test of how a single-threaded application would behave with the patch.

Thanks, -- Marc-Andre Lemburg eGenix.com

---

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

[malemburg]

Antoine Pitrou wrote:

> Perhaps I'm missing some feature of the new GIL. Is there some > documentation for it somewhere ?

Almost nothing besides what is found in ceval_gil.h and in the following thread: http://mail.python.org/pipermail/python-dev/2009-October/093321.html Dave Beazley did a high-level presentation about it: http://www.dabeaz.com/python/NewGIL.pdf

Thanks. Dave's slides help a lot. The logic certainly sounds like a major improvement !

>> Well, if you don't use any threads you don't have to change the setting >> in the first place :-). You might want to do it in order to >> "micro-optimize" your app (i.e. save 1-2% on CPU-bound code), but this >> is unnecessary with the new GIL, since the interval is ignored when >> there's only one thread running. > > Hmm, how do you determine that only one thread is running ?

Actually, I don't determine it, it's a side effect of how the GIL is implemented. If no thread is waiting for the GIL, the main thread isn't "asked" to drop it.

> What if an extension uses threads that are not under Python > control, e.g. when embedding the JVM or when using a CORBA > ORB ?

These don't wait for the GIL, hopefully :-)

Well, they might call back into Python, but I guess that's not supported anyway, not even in 2.x, since such a thread wouldn't have a properly setup Python thread state.

[pitrou]

As for your other comments, Marc-Andre:

• Please check whether you can move these variable declarations inside the main loop function (as statics): [snip]

They are used in other functions than in the main loop, so it's not possible.

• Please add the fallback solutions from the time module in case gettimeofday() is not available. You cannot assume that "all modern POSIX systems" implement that API - it was introduced in POSIX 2001 and Python 2.x still supports OSes that were released prior to that year.

I didn't know about that. In the context of py3k, I'm not sure it is really annoying, but you're right that for 2.x a fallback might be needed.

By the way, the new GIL only works with POSIX and Windows NT threading APIs. Perhaps it can't be backported at all to 2.x, given that 2.x supports more threading APIs than py3k does?

I haven't looked at the timing details of the implementation, but please make sure that it works even if the clock interval is a few ms

Well, since it works under Windows (I don't have any really reliable benchmarks, since I've tested under a VM which is not an ideal condition), I assume it's robust against poor timing precisions. (under Windows, WaitForMultipleObjects() is used which according to the MSDN docs has worse than microsecond precision)

But, yes, testing would be welcome on such platforms.

[pitrou]

I meant "worse than milliseconds", sorry.

[fdb125b4-c5ef-4da3-b52b-728ae9a0ef15]

On Fri, 22 Jan 2010 09:32:36 +0000 Marc-Andre Lemburg \report@bugs.python.org\ wrote:

• Please add the fallback solutions from the time module in case gettimeofday() is not available. You cannot assume that "all modern POSIX systems" implement that API - it was introduced in POSIX 2001 and Python 2.x still supports OSes that were released prior to that year.

POSIX as a standard tends to follow, not lead. The gettimeofday() call dates back over 20 years in BSD. time.time() falls back on ftime() and then time(). ftime() was added to the POSIX spec at the same time as gettimeofday() and is now deprecated. time() probably doesn't have enough resolution.

I'd have to be pointed to a specific platform which doesn't support gettimeofday() but which is supported by python. Otherwise, I'd be coding blind.

Ross

[fdb125b4-c5ef-4da3-b52b-728ae9a0ef15]

On Sat, 23 Jan 2010 18:23:10 +0000 Antoine Pitrou \report@bugs.python.org\ wrote:

By the way, the new GIL only works with POSIX and Windows NT threading APIs. Perhaps it can't be backported at all to 2.x, given that 2.x supports more threading APIs than py3k does?

Looking at the Python/thread_*.h files, it looks like py3k still supports 9 different threading models. If that's accurate, it means py3k trunk is broken on platforms which use cthread, lwp, os2, pth, sgi, solaris and wince threading models. The 2.x series adds atheos and beos to that list.

I think the right way to fix this is to extend the thread_*.h files to have a proper abstraction for conditions which can be used by the newgil work. Then the maintainers for more obscure platforms can fix those instead of it all turning into a big mess in ceval_gil.h.

Ross

[malemburg]

Ross Cohen wrote:

Ross Cohen \rcohen@snurgle.org\ added the comment:

On Fri, 22 Jan 2010 09:32:36 +0000 Marc-Andre Lemburg \report@bugs.python.org\ wrote:

> * Please add the fallback solutions from the time module in case gettimeofday() is not available. You cannot assume that "all modern POSIX systems" implement that API - it was introduced in POSIX 2001 and Python 2.x still supports OSes that were released prior to that year.

POSIX as a standard tends to follow, not lead. The gettimeofday() call dates back over 20 years in BSD. time.time() falls back on ftime() and then time(). ftime() was added to the POSIX spec at the same time as gettimeofday() and is now deprecated. time() probably doesn't have enough resolution.

I'd have to be pointed to a specific platform which doesn't support gettimeofday() but which is supported by python. Otherwise, I'd be coding blind.

The point here is that we have traditionally been careful not to break Python for platforms that don't support a certain API, hence the different fallback solutions for getting the current time.

gettimeofday() is indeed available on most OSes, but not all. VxWorks is an example where it is not available and Python happily uses alternatives.

http://www.gsalcazar.altervista.org/tools/how-to-port-python-to-vxworks/?PHPSESSID=u5do9v0lh2gqdqilmikv7nmsk1

There are also cases where gettimeofday() is buggy (e.g. on Solaris: http://www.unix.com/sun-solaris/59187-bugs-clock.html or Cygwin: http://old.nabble.com/About-The---GetMicroSecondTime-on-Windows---Post-td13721510.html) or returns errors every now and then (I have observed that in mxDateTime occasionally - it may be related to NTP doing its works).

For those situations is necessary to be able to enable a fallback solution using other APIs.

I've also done some extra research yesterday and found that e.g. Ruby is using gettimeofday() for thread scheduling as well. They have observed issues with gettimeofday() causing problems due to wallclock time not being a monotonic (e.g. due to NTP running on the machine and the clock sometimes going backwards due to corrections or DST changes).

http://groups.google.com/group/ruby-talk-google/browse_thread/thread/f8a616113e2eea8f

OTOH, using process timers is also not regarded as being ideal, since on SMP systems, each CPU will have its own timer and they are not necessarily in sync.

Other implementations tend to use the new clock_gettime() APIs where available and then use the CLOCK_MONOTONIC timer, e.g.

http://stackoverflow.com/questions/88/is-gettimeofday-guaranteed-to-be-of-microsecond-resolution http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=272

Here's some example code which tries to cover even more platforms than Python 2.x:

http://www.koders.com/c/fid5A75B2B62D4024A2431D060ADD5D378DB7A1D2BD.aspx

Another aspect to consider is update frequency of these APIs. gettimeofday()'s resolution depends on various factors and appears to vary between 1us and 100ms (on systems running using a 100Hz clock).

http://answers.google.com/answers/threadview/id/203397.html http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=272 https://computing.llnl.gov/tutorials/performance_tools/#gettimeofday

clock_gettime() has similar resolution, but tends to be updated more often:

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1264587819627+28353475&threadId=1230881

And finally, using wallclock time for these things is expensive as I've already mentioned:

http://groups.google.com/group/comp.os.linux.development.apps/browse_frm/thread/dc29071f2417f75f/c46264dba0863463?lnk=st&rnum=1#c46264dba0863463 http://www.developerweb.net/forum/archive/index.php/t-4368.html http://h21007.www2.hp.com/portal/download/files/unprot/hpux/HowToTellTheTime.pdf (as HTML: http://209.85.129.132/search?q=cache%3AI1ihHYBMJvgJ%3Ah21007.www2.hp.com%2Fportal%2Fdownload%2Ffiles%2Funprot%2Fhpux%2FHowToTellTheTime.pdf+gettimeofday()+support+across+platforms&hl=en&gl=de)

It appears to be better to use clock_gettime(CLOCK_MONOTONIC) where available and only use gettimeofday() as fallback solution together with times(), ftime() and time().

[malemburg]

Ross Cohen wrote:

> By the way, the new GIL only works with POSIX and Windows NT threading APIs. Perhaps it can't be backported at all to 2.x, given that 2.x supports more threading APIs than py3k does?

Looking at the Python/thread_*.h files, it looks like py3k still supports 9 different threading models. If that's accurate, it means py3k trunk is broken on platforms which use cthread, lwp, os2, pth, sgi, solaris and wince threading models. The 2.x series adds atheos and beos to that list.

I think the right way to fix this is to extend the thread_*.h files to have a proper abstraction for conditions which can be used by the newgil work. Then the maintainers for more obscure platforms can fix those instead of it all turning into a big mess in ceval_gil.h.

I find it rather strange that Python 3.x now only supports NT and POSIX threads in ceval while it still supports the whole set of other thread implementations for the _thread module.

Has this been discussed on python-dev ?

Why can't the code necessary to get the new GIL working be added to Sjoerd's portable thread library ?

As it stands, I think this is a bummer for both Python 2.7 and 3.2.

[pitrou]

Le mercredi 27 janvier 2010 à 10:37 +0000, Marc-Andre Lemburg a écrit :

I find it rather strange that Python 3.x now only supports NT and POSIX threads in ceval while it still supports the whole set of other thread implementations for the _thread module.

Has this been discussed on python-dev ?

Yes, it was. See http://mail.python.org/pipermail/python-dev/2009-October/093276.html All these thread_*.h files are still there, but most of them are deprecated (see the #error's in thread.c). It's the consensus that came out on the mailing-list.

The one exception is OS/2, which is "supported" as long as Andrew McIntyre (or someone else, of course) takes care about it. It was decided that OS/2 compatibility wasn't important enough to bar adding new features or making improvements.

Why can't the code necessary to get the new GIL working be added to Sjoerd's portable thread library ?

The condition variable "emulation" used by the new GIL under Windows is not valid as a general condition variable mechanism; this is mentioned in ceval_gil.h, and a link is given to a much more complicated (but valid) condition variable emulation. Therefore, I think it could be detrimental to expose it publicly.

[pitrou]

Le mercredi 27 janvier 2010 à 10:33 +0000, Marc-Andre Lemburg a écrit :

It appears to be better to use clock_gettime(CLOCK_MONOTONIC) where available and only use gettimeofday() as fallback solution together with times(), ftime() and time().

Thanks for the research! I will take a look at this.

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

Le mercredi 27 janvier 2010 à 10:37 +0000, Marc-Andre Lemburg a écrit : > > I find it rather strange that Python 3.x now only supports > NT and POSIX threads in ceval while it still supports the > whole set of other thread implementations for the _thread > module. > > Has this been discussed on python-dev ?

Yes, it was. See http://mail.python.org/pipermail/python-dev/2009-October/093276.html All these thread_*.h files are still there, but most of them are deprecated (see the #error's in thread.c). It's the consensus that came out on the mailing-list.

The one exception is OS/2, which is "supported" as long as Andrew McIntyre (or someone else, of course) takes care about it. It was decided that OS/2 compatibility wasn't important enough to bar adding new features or making improvements.

The arguments given in that thread sound a bit strange to me: just because there were no changes to a few files, doesn't really say anything about whether they contain working code or not.

In any case, the new GIL implementation as it stands makes supporting any of those additional threading implementations a moot case.

You could just as well remove them right now: if the GIL doesn't work on OS/2, then having support for it in the _thread module isn't really worth much, is it ?

Regardless, instead of now having two places where threading is implemented, we should just have one: the portable thread library and extend this as necessary.

> Why can't the code necessary to get the new GIL working be > added to Sjoerd's portable thread library ?

The condition variable "emulation" used by the new GIL under Windows is not valid as a general condition variable mechanism; this is mentioned in ceval_gil.h, and a link is given to a much more complicated (but valid) condition variable emulation. Therefore, I think it could be detrimental to expose it publicly.

By moving all the thread support code to the thread library, you don't really expose anything (the lib is only used internally by Python), you just reorganize the code in a more natural and easier to maintain way.

If you think that the win32 emulation shouldn't be used for non-GIL purposes, then just add a comment to those APIs.

With just NT and POSIX thread support, I think backporting the new GIL implementation to 2.7 is not possible - we'd have to go through a standard PEP-11 deprecation process and there are not enough 2.x releases left for that. It could only be backported as optional feature, to be enabled by a configure option.

[pitrou]

The arguments given in that thread sound a bit strange to me: just because there were no changes to a few files, doesn't really say anything about whether they contain working code or not.

That was a heuristic. Files which do not get any maintenance for years while other similar files do are quite suspicious. Given that nobody stepped up to contradict this hypothesis of mine, I assume it was right after all ;)

More seriously, all the APIs in question (and most of their supporting systems: IRIX etc.) seem practically dead. I don't want to rehash that discussion here, but you can post on python-dev if you want.

You could just as well remove them right now: if the GIL doesn't work on OS/2, then having support for it in the _thread module isn't really worth much, is it ?

Andrew told me he believed it possible to port the new GIL to OS/2. So perhaps he'll do that before 3.2 is out.

With just NT and POSIX thread support, I think backporting the new GIL implementation to 2.7 is not possible - we'd have to go through a standard PEP-11 deprecation process and there are not enough 2.x releases left for that. It could only be backported as optional feature, to be enabled by a configure option.

Right. That's what I think too.

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

> The arguments given in that thread sound a bit strange to me: > just because there were no changes to a few files, doesn't really > say anything about whether they contain working code or not.

That was a heuristic. Files which do not get any maintenance for years while other similar files do are quite suspicious. Given that nobody stepped up to contradict this hypothesis of mine, I assume it was right after all ;)

We'll only be able to tell for sure when it's too late: at release time. We simply don't have any active developers working on more exotic platforms, but that doesn't mean that Python isn't used on those platforms.

More seriously, all the APIs in question (and most of their supporting systems: IRIX etc.) seem practically dead. I don't want to rehash that discussion here, but you can post on python-dev if you want.

No need... I'm tired of trying to get Python devs on track with respect to the PEP-11 process, deprecations, etc.

> You could just as well remove them right now: if the GIL doesn't > work on OS/2, then having support for it in the _thread module > isn't really worth much, is it ?

Andrew told me he believed it possible to port the new GIL to OS/2. So perhaps he'll do that before 3.2 is out.

> With just NT and POSIX thread support, I think backporting the > new GIL implementation to 2.7 is not possible - we'd have to go > through a standard PEP-11 deprecation process and there are not > enough 2.x releases left for that. It could only be backported > as optional feature, to be enabled by a configure option.

Right. That's what I think too.

I'll close the issue then.

[malemburg]

Closing the issue since we can't backport to Python 2.7 due to the missing thread library support.

The patch may still be useful for experiments by users, though, so thanks to Ross and Neil for creating it.

[pitrou]

It appears to be better to use clock_gettime(CLOCK_MONOTONIC) where available and only use gettimeofday() as fallback solution together with times(), ftime() and time().

Ok, I've tried and it's less good than expected. Using CLOCK_MONOTONIC absolutely kills efficiency. CLOCK_REALTIME is ok but it has no obvious benefits (microsecond resolution as given by gettimeofday() is probably sufficient).

The explanation AFAICT is that pthread_cond_timedwait() waits for absolute clock values as given by CLOCK_REALTIME. CLOCK_MONOTONIC gives other values (the man page says: "represents monotonic time since some unspecified starting point"). These values are probably "in the past" as seen from pthread_cond_timedwait(), which implies a busy loop of waiting for the GIL to be released, inside of being suspended gracefully until the timeout.

I can still produce a patch with only CLOCK_REALTIME but I'm not sure it's worth the code complication.

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

> It appears to be better to use clock_gettime(CLOCK_MONOTONIC) > where available and only use gettimeofday() as fallback solution > together with times(), ftime() and time().

Ok, I've tried and it's less good than expected. Using CLOCK_MONOTONIC absolutely kills efficiency. CLOCK_REALTIME is ok but it has no obvious benefits (microsecond resolution as given by gettimeofday() is probably sufficient).

The explanation AFAICT is that pthread_cond_timedwait() waits for absolute clock values as given by CLOCK_REALTIME. CLOCK_MONOTONIC gives other values (the man page says: "represents monotonic time since some unspecified starting point"). These values are probably "in the past" as seen from pthread_cond_timedwait(), which implies a busy loop of waiting for the GIL to be released, inside of being suspended gracefully until the timeout.

The CLOCK_MONOTONIC timer only guarantees that you get an accurate time count. It doesn't maintain any relationship to the wall clock time. For the case in question you don't need the wall clock time, though. It's more important not have the clock go backwards or be subject to jitter.

pthreads will default to use the real time clock. In order to have them use the monotonic timer, you have to setup a condition variable attribute: See the man-page for pthread_condattr_setclock().

I can still produce a patch with only CLOCK_REALTIME but I'm not sure it's worth the code complication.

Even if you don't use CLOCK_MONOTONIC you should still prefer clock_gettime() over gettimeofday() simply because it's faster. The resolution of both will likely be the same, unless the hardware provides a more accurate timer than kernel ticks.

The code won't get more complicated if you refactor the time querying logic into a separate function (which the compiler can then inline as necessary).

[fdb125b4-c5ef-4da3-b52b-728ae9a0ef15]

I am confused by this line of reasoning. Is it ok to ignore the deprecation process in py3k but not in 2.x? Is it only ok if a core developer does it?

If the point of 2.7 is to make it easier for apps and packages to be ported to py3k, then what would be the point of these platforms moving to 2.7 in the first place? It seems perfectly reasonable not to support platforms which are never going to care the release. If the platforms are broken for 2.7, you'll get that much more warning before 3.2 is released so it can be fixed.

Ross

On Wed, 27 Jan 2010 14:30:20 +0000 Marc-Andre Lemburg \report@bugs.python.org\ wrote:

Antoine Pitrou wrote: > More seriously, all the APIs in question (and most of their supporting > systems: IRIX etc.) seem practically dead. I don't want to rehash that > discussion here, but you can post on python-dev if you want.

No need... I'm tired of trying to get Python devs on track with respect to the PEP-11 process, deprecations, etc.

>> You could just as well remove them right now: if the GIL doesn't >> work on OS/2, then having support for it in the _thread module >> isn't really worth much, is it ? > > Andrew told me he believed it possible to port the new GIL to OS/2. So > perhaps he'll do that before 3.2 is out. > >> With just NT and POSIX thread support, I think backporting the >> new GIL implementation to 2.7 is not possible - we'd have to go >> through a standard PEP-11 deprecation process and there are not >> enough 2.x releases left for that. It could only be backported >> as optional feature, to be enabled by a configure option. > > Right. That's what I think too.

I'll close the issue then.

[pitrou]

pthreads will default to use the real time clock. In order to have them use the monotonic timer, you have to setup a condition variable attribute: See the man-page for pthread_condattr_setclock().

I'll look at that, but I'm not thrilled at the propect of complicating the code paths so much. There may be systems where CLOCK_MONOTONIC is unavailable, others where pthread_condattr_setclock() is unsupported, etc.

The code won't get more complicated if you refactor the time querying logic into a separate function (which the compiler can then inline as necessary).

It does get more complicated, since there are several paths (clock_gettime() and then a fallback on gettimeofday()). I'm not talking about complexity in the executable but about maintenance complexity.

[malemburg]

Ross Cohen wrote:

Ross Cohen \rcohen@snurgle.org\ added the comment:

I am confused by this line of reasoning. Is it ok to ignore the deprecation process in py3k but not in 2.x? Is it only ok if a core developer does it?

It's normally not ok to ignore the deprecation process for 3.x, since the 3.x branch is in a stable state, just like the 2.x branch (3.0 and, after some discussions, 3.1 were allowed to break things).

I don't know why this process wasn't followed for the new GIL implementation.

It basically breaks compatibility with the other thread implementations, but does so without actually removing the support for them in the thread library we're using for the _thread module.

Perhaps this was simply not known to other developers.

There's also no PEP for the new implementation, which you'd normally require for any such major change to the internals.

If the point of 2.7 is to make it easier for apps and packages to be ported to py3k, then what would be the point of these platforms moving to 2.7 in the first place? It seems perfectly reasonable not to support platforms which are never going to care the release. If the platforms are broken for 2.7, you'll get that much more warning before 3.2 is released so it can be fixed.

Breaking existing applications and ports of Python for 2.7 certainly won't make anything easier for anyone.

For 2.7 we will certainly not allow the above to happen, since that's the version that's going to be used by most people for a few years to come.

What we could do for 2.7, is add Py3k deprecation warnings to the alternative thread implementations, mentioning their removal in 3.2.

[fdb125b4-c5ef-4da3-b52b-728ae9a0ef15]

On Fri, 29 Jan 2010 21:15:14 +0000 Marc-Andre Lemburg \report@bugs.python.org\ wrote:

Breaking existing applications and ports of Python for 2.7 certainly won't make anything easier for anyone.

For 2.7 we will certainly not allow the above to happen, since that's the version that's going to be used by most people for a few years to come.

What we could do for 2.7, is add Py3k deprecation warnings to the alternative thread implementations, mentioning their removal in 3.2.

What if, as you proposed earlier, the patch were to leave the old behavior if the threading model on the given platform were not supported?

Ross

[jcea]

Sorry for the nosy. There is something going wrong with my Firefox caching.