Backport capsule object

[larryhastings]

BPO	7992
Nosy	@malemburg, @jcea, @pitrou, @vstinner, @larryhastings, @benjaminp, @briancurtin, @florentx

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/larryhastings' closed_at = created_at = labels = ['interpreter-core', 'type-feature'] title = 'Backport capsule object' updated_at = user = 'https://github.com/larryhastings' ``` bugs.python.org fields: ```python activity = actor = 'jcea' assignee = 'larry' closed = True closed_date = closer = 'larry' components = ['Interpreter Core'] creation = creator = 'larry' dependencies = [] files = [] hgrepos = [] issue_num = 7992 keywords = ['patch', 'buildbot'] message_count = 15.0 messages = ['99802', '101691', '101695', '101696', '101698', '101723', '101726', '101727', '102164', '102192', '102194', '102196', '102203', '103661', '104140'] nosy_count = 8.0 nosy_names = ['lemburg', 'jcea', 'pitrou', 'vstinner', 'larry', 'benjamin.peterson', 'brian.curtin', 'flox'] pr_nums = [] priority = 'normal' resolution = 'accepted' stage = 'needs patch' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue7992' versions = ['Python 2.7'] ```

[larryhastings]

Backporting the PyCapsule object from 3.1 to 2.7. I'll have a patch today.

[eda57068-96ad-4b33-8431-9c528f59a6a6]

All windows buildbots fail to compile:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%20trunk http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%20trunk http://www.python.org/dev/buildbot/all/builders/x86%20Windows7%20trunk

[pitrou]

As Marc-André pointed out, you shouldn't have converted old uses of PyCObject. It breaks compatibility, and I don't think there's any point since 2.7 will probably be the last in the 2.x line.

[pitrou]

(if you want to be perfectionist, you can have those modules provide both a PyCObject and a PyCapsule interface...)

[vstinner]

PyCapsule comes from issues bpo-5630 and bpo-5872.

[larryhastings]

This is also (naturally) being discussed on python-dev. I've posted a long message there about it; you can find that message here:

http://mail.python.org/pipermail/python-dev/2010-March/098904.html

I don't know whether it's best to conduct such a discussion here, in Python-Dev, or both. I assumed Python-Dev is superior because more people will see it. If it makes sense to move the discussion here we can move it.

[larryhastings]

Florent: that's because the Windows build projects don't build capsule.c. I don't have an appropriate Windows development environment; I'll try to fix them by hand tonight, but if that doesn't work we'll have to call for help from some Windows core developer.

[briancurtin]

(adding myself to the nosy list as a Windows dev)

[eda57068-96ad-4b33-8431-9c528f59a6a6]

The PendingDeprecationWarning is still spit by the bsddb module.

$ ./python -Wd -m test.regrtest test_bsddb
test_bsddb
./Lib/importlib/__init__.py:37: PendingDeprecationWarning: The CObject type is marked Pending Deprecation in Python 2.7.  Please use capsule objects instead.
  __import__(name)h
1 test OK.

[larryhastings]

Yes. I was told it was inappropriate for me to change the bsddb module, as it's independently maintained by Jesus Cea. I sent Mr. Cea a patch last night; I hope he chooses to merge it soon.

Since CObjects are marked Pending Deprecation, and bsddb is using a CObject, this is correct behavior.

Were you mentioning it simply to bring it my attention, or am I missing something important?

[larryhastings]

Oh, and, last night I checked in r79590. This is the checkin that ameliorates the backwards compatibility issues. Specifically, I changed four things:

• PyCObject_AsVoidPtr() can now open capsules. This addresses most of the remaining backwards-compatibility concerns about the conversion of Python 2.7 from CObjects to capsules.
• CObjects were marked Pending Deprecation.
• Documentation about this pending deprecation was added to cobject.h.
• The capsule source files were added to the legacy PC build processes.

(Someone had already added them to the current PC build process, a day or two after I committed the previous checkin.)

Sorry for forgetting to update the issue last night, but I was in a bit of a hurry to get this done before Benjamin cut 2.7b1.

[eda57068-96ad-4b33-8431-9c528f59a6a6]

Ok, I put Jésus in the nosy list.

It makes the test_all fail on trunk because it checks the warnings raised during import.

AFAIU, the patch is something like:

-#if (PY_VERSION_HEX \< 0x03020000) +#if (PY_VERSION_HEX \< 0x02070000)

[larryhastings]

The patch is a bit more involved than that. Capsules didn't exist in 3.0, and the bsddb module published a CObject in 3.1. So bsddb must continue to use CObject for those two releases.

Therefore the patch to the line you were addressing looks like this:

-#if (PY_VERSION_HEX \< 0x03020000) +#if (PY_VERSION_HEX \< ((PY_MAJOR_VERSION \< 3) ? 0x02070000 : 0x03020000))

However, Jesus's use of capsules has a bug. The capsule API requires that the string passed in to PyCapsule_New() must "outlive" the capsule; he's passing in a stack buffer. So my patch also fixes that.

Finally my bsddb patch adds the appropriate text to the header file where it describes how to import the _bsddb C API object (CObject vs capsule).

FWIW, I want to close this issue soon. How about I close it after 2.7b1 has been out for a week or so--assuming there aren't any new concerns regarding the capsule backport.

[larryhastings]

Marking closed, as promised; 2.7b1 has been out for nine days and there hasn't been a peep. If new problems crop up from the capsule backport, please create a new issue and assign it to me.

For posterity: 2.7b1 shipped with a bsddb that still publishes a CObject. This is the only CObject left in 2.7b1, and Python doesn't itself use that CObject for anything. Therefore there are no CObject exploits left in pure Python 2.7b1. I don't know if bsddb will switch to a capsule in a later Python 2.7 beta; that's up to Jesus and Benjamin and co. But I suspect it won't.

[jcea]

For future reference:

http://mail.python.org/pipermail/python-committers/2010-April/000875.html

The final consensus was not to integrate bsddb Capsule for Python 2.7, since CObject is not actually deprecated in python 2.7.