Check that new heap types cannot be created uninitialised: add Py_TPFLAGS_DISALLOW_INSTANTIATION type flag

[pablogsal]

BPO	43916
Nosy	@vstinner, @tiran, @serhiy-storchaka, @pablogsal, @miss-islington, @erlend-aasland, @shreyanavigyan
PRs	python/cpython#25653 python/cpython#25721 python/cpython#25722 python/cpython#25733 python/cpython#25745 python/cpython#25748 python/cpython#25749 python/cpython#25750 python/cpython#25751 python/cpython#25753 python/cpython#25768 python/cpython#25791 python/cpython#25854 python/cpython#26412 python/cpython#26888
Files	patch.diff disablednew.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['expert-C-API', 'type-bug', '3.9'] title = 'Check that new heap types cannot be created uninitialised: add Py_TPFLAGS_DISALLOW_INSTANTIATION type flag' updated_at = user = 'https://github.com/pablogsal' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = False closed_date = None closer = None components = ['C API'] creation = creator = 'pablogsal' dependencies = [] files = ['49989', '49992'] hgrepos = [] issue_num = 43916 keywords = ['patch'] message_count = 72.0 messages = ['391634', '391635', '391903', '391905', '391909', '391910', '391911', '391912', '391913', '391917', '391924', '391925', '391926', '391928', '391929', '391931', '391934', '391936', '391937', '391984', '391992', '391999', '392036', '392040', '392042', '392044', '392046', '392048', '392057', '392169', '392170', '392297', '392414', '392426', '392427', '392433', '392434', '392435', '392437', '392438', '392451', '392456', '392462', '392464', '392465', '392472', '392473', '392526', '392528', '392534', '392537', '392545', '392546', '392553', '392554', '392556', '392558', '392570', '392621', '392632', '392635', '392811', '394571', '394582', '394603', '396200', '396476', '411785', '411786', '411865', '411867', '411874'] nosy_count = 7.0 nosy_names = ['vstinner', 'christian.heimes', 'serhiy.storchaka', 'pablogsal', 'miss-islington', 'erlendaasland', 'shreyanavigyan'] pr_nums = ['25653', '25721', '25722', '25733', '25745', '25748', '25749', '25750', '25751', '25753', '25768', '25791', '25854', '26412', '26888'] priority = None resolution = None stage = 'patch review' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue43916' versions = ['Python 3.9'] ```

[erlend-aasland]

These lists are not complete, for example select.kevent was not listed whereas it has a bug.

If one of the admins could make that post into a wiki post, it would be open for editing, making it easier to fill the holes in the lists.

[pablogsal]

If one of the admins could make that post into a wiki post, it would be open for editing, making it easier to fill the holes in the lists.

Not sure what do you have in mind for the wiki, but the easiest is to open some GitHub issue somewhere or using https://discuss.python.org/

[pablogsal]

• _ast.AST: use {Py_tp_new, PyType_GenericNew} and {Py_tp_init, ast_type_init} slots. What happens if tp_new is called without calling tp_init?

I think this is fine since ast_type_init doesn't initialize any C fields and only mutates the object by using PyObject_SetAttr

[pablogsal]

Something is wrong after commit 3bb09947ec4837de75532e21dd4bd25db0a1f1b7.

When building Python I am getting:

WARNING: renaming "_curses" since importing it failed: /home/pablogsal/github/python/master/build/lib.linux-x86_64-3.10-pydebug/_curses.cpython-310d-x86_64-linux-gnu.so: undefined symbol: _PyStructSequence_InitType WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses'

[pablogsal]

The problem is that the curses module is using a function that requires to be built in the core:

#ifdef Py_BUILD_CORE
extern int _PyStructSequence_InitType(
    PyTypeObject *type,
    PyStructSequence_Desc *desc,
    unsigned long tp_flags);
#endif

[pablogsal]

The build errors were in the CI, but because Python doesn't complain if it cannot build a module, then we never catched this. I have opened https://github.com/python/cpython/pull/25768 to fix it to unblock the release, since the curses module is broken currently.

[pablogsal]

New changeset 558df9010915c8fe94f4d7f842e7c5aabbb06b14 by Pablo Galindo in branch 'master': bpo-43916: Export the _PyStructSequence_InitType to fix build errors in the curses module (GH-25768) https://github.com/python/cpython/commit/558df9010915c8fe94f4d7f842e7c5aabbb06b14

[erlend-aasland]

Not sure what do you have in mind for the wiki

See https://meta.discourse.org/t/what-is-a-wiki-post/30801

[pablogsal]

See https://meta.discourse.org/t/what-is-a-wiki-post/30801

Gotcha, will try to do that as soon as possible

[pablogsal]

I have transformed https://discuss.python.org/t/list-of-built-in-types-converted-to-heap-types/8403 into a wiki. Tell me if you need some alterations of something is missing.

[tiran]

New changeset ddbef71a2c166a5d5dd168e26493973053a953d6 by Christian Heimes in branch 'master': bpo-43916: Rewrite new hashlib tests, fix typo (GH-25791) https://github.com/python/cpython/commit/ddbef71a2c166a5d5dd168e26493973053a953d6

[pablogsal]

New changeset c2931d31f8ba7cf10044de276018c713ffc73592 by Pablo Galindo in branch 'master': bpo-43916: Move the _PyStructSequence_InitType function to the internal API (GH-25854) https://github.com/python/cpython/commit/c2931d31f8ba7cf10044de276018c713ffc73592

[vstinner]

Erlend added test.support.check_disallow_instantiation() function in bpo-43988 to write tests for immutable types.

[pablogsal]

Please, check also the discusion happening here:

https://mail.python.org/archives/list/python-committers@python.org/thread/FHFI7QKWNHAVWVFTCHJGTYD3ZFVEUXDD/

[vstinner]

New changeset e90e0422182f4ca7faefd19c629f84aebb34e2ee by Erlend Egeberg Aasland in branch 'main': bpo-43916: Use test.support.check_disallow_instantiation() in test_tcl (GH-26412) https://github.com/python/cpython/commit/e90e0422182f4ca7faefd19c629f84aebb34e2ee

[erlend-aasland]

FYI, bpo-44087 added the Py_TPFLAGS_DISALLOW_INSTANTIATION flag to sqlite3.Statement.

(Side note: I find the issue title a little bit confusing.)

[vstinner]

New changeset 733587011dbb47c2e461188f0574e1cc5288062a by Miss Islington (bot) in branch '3.10': bpo-43916: Use test.support.check_disallow_instantiation() in test_tcl (GH-26412) (GH-26888) https://github.com/python/cpython/commit/733587011dbb47c2e461188f0574e1cc5288062a

[vstinner]

In Python 3.11, 41 types are declared explicitly with the Py_TPFLAGS_DISALLOW_INSTANTIATION flag:

• _curses_panel.panel
• _dbm.dbm
• _gdbm.gdbm
• _hashlib.HASH
• _hashlib.HASHXOF
• _hashlib.HMAC
• _md5.md5
• _multibytecodec.MultibyteCodec
• _sha1.sha1
• _sha256.sha224
• _sha256.sha256
• _sha512.sha384
• _sha512.sha512
• _sre.SRE_Scanner
• _ssl.Certificate
• _thread._localdummy
• _thread.lock
• _tkinter.Tcl_Obj
• _tkinter.tkapp
• _tkinter.tktimertoken
• _winapi.Overlapped
• _xxsubinterpreters.ChannelID
• array.arrayiterator
• curses.ncurses_version
• functools.KeyWrapper
• functools._lru_list_elem
• os.DirEntry
• os.ScandirIterator
• pyexpat.xmlparser
• re.Match
• re.Pattern
• select.devpoll
• select.poll
• sqlite3.Statement
• stderrprinter
• sys.flags
• sys.getwindowsversion
• sys.version_info
• unicodedata.UCD
• zlib.Compress
• zlib.Decompress

[vstinner]

Can we close this issue? Or is there a remaining task?

[erlend-aasland]

Can we close this issue? Or is there a remaining task?

3.9 is still affected; we should fix those types first.

[erlend-aasland]

FYI: There are only two bug-fix releases left for 3.9.

[vstinner]

3.9 is still affected; we should fix those types first.

I'm against backporting the new type flag, but we can try to set explicitly tp_set to NULL *after* calling PyType_Ready().

[erlend-aasland]

Victor:

Can we close this issue? Or is there a remaining task?

3.9 is still affected; we should fix those types first.

3.9 has now entered security-only phase, so I guess that ship has sailed. Suggesting to close this.

[vstinner]

Right, I closed the issue.