distutils.sysconfig.parse_makefile() regression in Python 3.10

[bc75918c-a209-4fa3-b6cf-28cfb7317f76]

BPO	44351
Nosy	@encukou, @hroncok, @frenzymadness, @miss-islington
PRs	python/cpython#26637 python/cpython#26673

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['type-bug', 'library', '3.10', '3.11'] title = 'distutils.sysconfig.parse_makefile() regression in Python 3.10' updated_at = user = 'https://github.com/hroncok' ``` bugs.python.org fields: ```python activity = actor = 'petr.viktorin' assignee = 'none' closed = True closed_date = closer = 'petr.viktorin' components = ['Library (Lib)'] creation = creator = 'hroncok' dependencies = [] files = [] hgrepos = [] issue_num = 44351 keywords = ['patch'] message_count = 6.0 messages = ['395352', '395385', '395387', '395402', '395638', '395873'] nosy_count = 4.0 nosy_names = ['petr.viktorin', 'hroncok', 'frenzy', 'miss-islington'] pr_nums = ['26637', '26673'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue44351' versions = ['Python 3.10', 'Python 3.11'] ```

[bc75918c-a209-4fa3-b6cf-28cfb7317f76]

Hello. I think https://github.com/python/cpython/pull/23142 changed the behavior of distutils.sysconfig.parse_makefile().

A downstream Fedora report with an affected petsc package: https://bugzilla.redhat.com/show_bug.cgi?id=1959088

Reproducers (DeprecationWarnings removed for readability):

$ cat makefile
ALL: lib

DIRS = ssls asls # isls rscs
LOCDIR = src/tao/complementarity/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['ssls', 'asls']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['ssls', 'asls', '#', 'isls', 'rscs']

And:

$ cat makefile
-include ../../../../petscdir.mk
ALL: lib

LIBBASE  = libpetscksp
DIRS     = cr bcgs bcgsl cg cgs gmres cheby rich lsqr preonly tcqmr tfqmr \
           qcg bicg minres symmlq lcd ibcgs python gcr fcg tsirm fetidp hpddm
LOCDIR   = src/ksp/ksp/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', 'qcg', 'bicg', 'minres', 'symmlq', 'lcd', 'ibcgs', 'python', 'gcr', 'fcg', 'tsirm', 'fetidp', 'hpddm']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', '\\']

And:

$ cat makefile
-include ../../../../petscdir.mk
ALL: lib

LIBBASE  = libpetscksp
DIRS     = jacobi none sor shell bjacobi mg eisens asm ksp composite redundant spai is pbjacobi vpbjacobi ml\
           mat hypre tfs fieldsplit factor galerkin cp wb python \
           chowiluviennacl chowiluviennaclcuda rowscalingviennacl rowscalingviennaclcuda saviennacl saviennaclcuda\
           lsc redistribute gasm svd gamg parms bddc kaczmarz telescope patch lmvm hmg deflation hpddm hara
LOCDIR   = src/ksp/pc/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml', 'mat', 'hypre', 'tfs', 'fieldsplit', 'factor', 'galerkin', 'cp', 'wb', 'python', 'chowiluviennacl', 'chowiluviennaclcuda', 'rowscalingviennacl', 'rowscalingviennaclcuda', 'saviennacl', 'saviennaclcuda', 'lsc', 'redistribute', 'gasm', 'svd', 'gamg', 'parms', 'bddc', 'kaczmarz', 'telescope', 'patch', 'lmvm', 'hmg', 'deflation', 'hpddm', 'hara']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml\\']

[559e26c8-6e49-4373-b6ba-a0229f62f8ab]

Thanks for the report.

The regression is caused by the fact that the old implementation of parse_makefile in distutils.sysconfig was using feature-rich class TextFile which handles all the functionalities you reported as broken - stripping of comments, joining lines, and more.

The new implementation of parse_makefile in sysconfig just reads all the lines without any processing.

The TextFile class is deprecated together with the whole distutils. The class is not used outside of distutils modules, only in these functions:

distutils.extension:read_setup_file distutils.sdist:read_template + its tests in distutils/tests/test_text_file.py

If the functionality is something we want to preserve, we should find a new home for the class and then we can use it in the sysconfig module.

[encukou]

IMO, the functionality should only be preserved until distutils is removed. So:

• distutils.sysconfig.parse_makefile should use TextFile as before, so projects that use it aren't broken *yet*
• nothing else should call distutils.sysconfig.parse_makefile, so it doesn't block removing distutils

Would that make sense?

[bc75918c-a209-4fa3-b6cf-28cfb7317f76]

Yes, for the purposes of this bug, bringing TextFile-powered parse_makefile() back to distutils (and distutils only) is the right thing to do.

Whether or not Python needs a public standard library function to parse makefiles and whether that function in sysconfig is imperfect and needs improvements, that is an entirely different discussion.

[encukou]

New changeset fc98266ff627ba0f56f8ae241245b66bc983baa3 by Lumír 'Frenzy' Balhar in branch 'main': bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637) https://github.com/python/cpython/commit/fc98266ff627ba0f56f8ae241245b66bc983baa3

[encukou]

New changeset 2f2ea96c4429b81f491aa1cdc4219ef2fd6d37fb by Miss Islington (bot) in branch '3.10': bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637) (GH-26673) https://github.com/python/cpython/commit/2f2ea96c4429b81f491aa1cdc4219ef2fd6d37fb