libpython.so 3.9.8 drops symbol used by third party extension module(s)

[doko42]

BPO	45778
Nosy	@doko42, @vstinner, @tiran, @ambv, @serhiy-storchaka, @hroncok, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core', '3.10', '3.9', 'type-crash', 'release-blocker'] title = 'libpython.so 3.9.8 drops symbol used by third party extension module(s)' updated_at = user = 'https://github.com/doko42' ``` bugs.python.org fields: ```python activity = actor = 'lukasz.langa' assignee = 'none' closed = True closed_date = closer = 'lukasz.langa' components = ['Interpreter Core'] creation = creator = 'doko' dependencies = [] files = [] hgrepos = [] issue_num = 45778 keywords = ['3.9regression'] message_count = 8.0 messages = ['406085', '406086', '406087', '406088', '406089', '406091', '406100', '406101'] nosy_count = 7.0 nosy_names = ['doko', 'vstinner', 'christian.heimes', 'lukasz.langa', 'serhiy.storchaka', 'hroncok', 'pablogsal'] pr_nums = [] priority = 'release blocker' resolution = 'wont fix' stage = 'resolved' status = 'closed' superseder = None type = 'crash' url = 'https://bugs.python.org/issue45778' versions = ['Python 3.9', 'Python 3.10'] ```

[doko42]

[reported in Debian as https://bugs.debian.org/998854]

3.9.8, compared to 3.9.7 introduces a regression, making at least one third party extension fail at runtime. The symbol changes in 3.9.8 are:

- _PyUnicode_DecodeUnicodeEscape@Base
+ _PyUnicode_DecodeRawUnicodeEscapeStateful@Base
+ _PyUnicode_DecodeUnicodeEscapeInternal@Base
+ _PyUnicode_DecodeUnicodeEscapeStateful@Base

Affected at least is the typed-ast extension. Yes, you can rebuild the extension with 3.9.8, but then it doesn't work with earlier 3.9 versions. Just dropping the symbol on a stable branch is suboptimal.

[pablogsal]

Aren't these private symbols?

[tiran]

Yes, they are private symbols. 3rd parties should not use them.

The symbol _PyUnicode_DecodeUnicodeEscape was replace in python/cpython#73139: "bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec".

[pablogsal]

Yeah, the ABI guarantees are for the public C-API (both the restricted/limited and the general one) but not for private symbols.

[tiran]

Matthias mentions the upstream tickets

https://github.com/python/typed_ast/issues/169 https://github.com/python/typed_ast/issues/170

on the BDO. The issue affects typed-ast and indirectly older versions of black. Upstream is considering to EOL typed-ast package.

[doko42]

the two users of typed-ast (at least in Debian) are black and mypy. Apparently black stopped using typed-ast recently.

[vstinner]

I suggest to close this issue. We are free to remove private functions in minor releases (3.9.x) without any warning. Private functions are excluded from the backward compatibility warranty.

For example: "Names prefixed by an underscore, such as _Py_InternalState, are private API that can change without notice even in patch releases." https://docs.python.org/dev/c-api/stable.html#stable

In C, it's easy to call the decode() method of a string object and pass the expected encoding and error handler.

[vstinner]

• _PyUnicode_DecodeUnicodeEscape@Base

Removed by:

commit 7c722e32bf582108680f49983cf01eaed710ddb9 Author: Serhiy Storchaka \storchaka@gmail.com\ Date: Thu Oct 14 20:03:29 2021 +0300

[3.9] bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) (GH-28945)

They support now splitting escape sequences between input chunks.

Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
(cherry picked from commit c96d1546b11b4c282a7e21737cb1f5d16349656d)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>