October 2021 Release

[uranusjr]

Tracking issue for pip 21.3.

Anyone volunteers to be the release manager @pypa/pip-committers?

[sbidoul]

I can not be the release manager for this one: October being the month of the annual Odoo release I'll be too busy in the Odoo community. I may take January or April.

I'll be around to help, and at least handle any issues related to editables, though.

[pfmoore]

I might be able to make time for this, but only if it's a relatively routine release. I definitely won't be able to support multiple followup bugfix releases, so if we aren't OK with taking a hard line with "it'll be sorted in 22.1 and not before", then I'd rather not take this on. IMO, we've got into a habit of releasing too many bugfix releases, and it's not practical to sustain given the level of resource we have.

Are we comfortable with a release on a weekend early in Oct, shipped from whatever is on main at that point, with no commitment to bugfix releases for anything beyond complete showstoppers?

[pradyunsg]

I'm happy to pick up the baton for this time around, given that everyone else has limited availability. :)

I expect there to be a bunch of activity and fallout in this release, since we're going to ship in-tree-build-by-default, will likely try to ship editables support, and have a bunch of removals (there's a few gone_in="21.3" deprecations) as well.

[pfmoore]

OK, cool. I still think we should as a matter of principle try to cut back on the number of bugfix releases we do. Either that, or we may start to find that a 3-month release cycle isn't sustainable (we've barely finished doing 21.2 bugfix releases and we're starting on 21.3).

Maybe we need to have a separate issue to review our release policy?

[pradyunsg]

Maybe we need to have a separate issue to review our release policy?

I'm ambivalent. This works, so does a separate issue. :)

I still think we should as a matter of principle try to cut back on the number of bugfix releases we do. Either that, or we may start to find that a 3-month release cycle isn't sustainable

I tend to agree.

FWIW, I'm also fine with deferring that decision to whomever is cutting the releases. I personally prefer to keeping the issue open as short as possible, actually getting the first release on a week day in the first half of the month and so on -- all of which was tricky with the resolver rollout. And I was full-time working on pip in that period. :)

Anyway, for 21.2, we had a whole bunch of follow ups on the sysconfig <-> distutils migration and the resolver -- but the issue stayed open for quite a while after the final release "just in case", which... meh, it's fine but we can also get away with not doing that.

[uranusjr]

I think the sysconfig thing needs to stay open for a while since there are still some obscure issues popping up once in a while. Or alternatively, since we’ve already committed to the switch in 3.10 and those cases are obscure anyway, we can completely silence the warning in 21.3 and see what happens after 3.10.

[pradyunsg]

but the issue stayed open for quite a while after the final release "just in case"

I was referring to the release issue, not the sysconfig migration-related reporting location. :)

And, definitely agree that we should keep the warnings around for a while. Ambivalent on what to do for 3.10+ -- I'm inclined to keep them as-is, until we drop distutils entirely from 3.12. 🤷🏽

[pradyunsg]

Alrighty, two release blockers left:

• A decently important resolver improvement.
• Dropping a deprecated items
  • --build-dir
  • sdist reinstall behaviour

Looks like they're all tasks I ended up picking up, so if someone wants to help out with the reviews of my PRs or the resolvelib side of things, that'd be nice. 😅

[pradyunsg]

Alrighty. Blocked on one last improvement: https://github.com/pypa/pip/pull/10481.

And assuming that'll land, I'll cut a release today (if it happens soon enough) / tomorrow / early next week.

[pradyunsg]

And... all blockers are resolved.

I intend to cut the release 6:30pm UK time on Monday, Oct 11.

[pradyunsg]

21.3 is released!

• Uploaded on PyPI: https://pypi.org/project/pip/21.3/
• Updated get-pip.py: https://github.com/pypa/get-pip/pull/131
• Gonna go and make the announcements for the release now. :)

[uranusjr]

How about CPython, or should we wait a while to make sure there are no urgent bugs?

[pradyunsg]

Well, if we had urgent bugs, we would've seen an issue about it already. :P

None the less, I'll do the CPython PR after work tomorrow.

[pradyunsg]

One bugfix to go!

None the less, I'll do the CPython PR after work tomorrow.

Haven't done this, since we broke editable installs. 😅

[pradyunsg]

Ahoy folks -- consider main open for development. I'll cherry pick the merged PRs in the 21.3.1 milestone, for making the release.

[pradyunsg]

Alright. Will cut 21.3.1 today.

Thanks @sbidoul and @uranusjr for picking up the editable installs issue and driving it to completion. ^.^

[johnandersen777]

Does the editable install issue look something like this?

Edit: Looks like no issue on 21.2. The issue is present on 21.3

$ python -m pip install -e .[dev]
Obtaining file:///tmp/tmp.VRDDQOtFJe/intel-innersource-ml-core-2
  Installing build dependencies ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
  Build backend does not support editables, falling back to setup.py egg_info.
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/johnsa1/Documents/python/dffml/.venv/bin/python3.7 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/tmp.VRDDQOtFJe/intel-innersource-ml-core-2/setup.py'"'"'; __file__='"'"'/tmp/tmp.VRDDQOtFJe/intel-innersource-ml-core-2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-38uzriwi
       cwd: /tmp/tmp.VRDDQOtFJe/intel-innersource-ml-core-2/
  Complete output (31 lines):
  /home/johnsa1/Documents/python/dffml/.venv/bin/python3.7: No module named pip
  Traceback (most recent call last):
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 75, in fetch_build_egg
      subprocess.check_call(cmd)
    File "/usr/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/home/johnsa1/Documents/python/dffml/.venv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpoldftue5', '--quiet', 'setuptools_scm[toml]>=3.4.3']' returned non-zero exit status 1.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/tmp.VRDDQOtFJe/intel-innersource-ml-core-2/setup.py", line 8, in <module>
      setuptools.setup(use_scm_version=True)
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 152, in setup
      _install_setup_requires(attrs)
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 809, in fetch_build_eggs
      replace_conflicting=True,
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 768, in resolve
      replace_conflicting=replace_conflicting
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1051, in best_match
      return self.obtain(req, installer)
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1063, in obtain
      return installer(requirement)
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 877, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/tmp/pip-build-env-rplyhnto/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 77, in fetch_build_egg
      raise DistutilsError(str(e)) from e
  distutils.errors.DistutilsError: Command '['/home/johnsa1/Documents/python/dffml/.venv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpoldftue5', '--quiet', 'setuptools_scm[toml]>=3.4.3']' returned non-zero exit status 1.
  ----------------------------------------
WARNING: Discarding file:///tmp/tmp.VRDDQOtFJe/intel-innersource-ml-core-2. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[sbidoul]

@pdxjohnny yes, your message looks like #10573. Can you confirm your project has both a pyproject.toml and setup_requires in setup.py ?

[johnandersen777]

@sbidoul yes it has both

pyproject.toml

requires = ["setuptools>=44", "wheel", "setuptools_scm[toml]>=3.4.3"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]

[tool.black]
exclude = '''
(
  /(
      \.eggs         # exclude a few common directories in the
    | \.git          # root of the project
    | \.hg
    | \.mypy_cache
    | \.tox
    | \.venv
    | _build
    | buck-out
    | build
    | dist
  )
)
'''

setup.cfg

# https://setuptools.pypa.io/en/latest/userguide/declarative_config.html
[metadata]
name = intel-inner-source-ml-core
description = intel-inner-source-ml-core
long_description = intel-inner-source-ml-core
author = John Andersen
author_email = john.s.andersen@intel.com
maintainer = John Andersen
maintainer_email = john.s.andersen@intel.com
url = https://github.com/intel-sandbox/applications.services.isz.ml
license = MIT
# keywords = dffml
classifiers =
    Development Status :: 3 - Alpha
    Intended Audience :: Developers
    License :: OSI Approved :: MIT License
    Natural Language :: English
    Operating System :: OS Independent
    Programming Language :: Python :: 3 :: Only
    Programming Language :: Python :: 3.7
    Programming Language :: Python :: Implementation :: CPython
    Programming Language :: Python :: Implementation :: PyPy

[options]
zip_safe = False
include_package_data = True
packages = find:
package_dir =
    =apps
entry_points = file: entry_points.txt
setup_requires =
    setuptools_scm[toml]>=3.4.3
install_requires =
    dffml>=0.4.0
    PyGithub

[options.packages.find]
where = apps

[options.extras_require]
dev =
    coverage
    codecov
    sphinx
    twine
    setuptools_scm[toml]>=3.4.3
    black==19.10b0
    importlib_metadata>=4.8.1;python_version<"3.8"

[sbidoul]

@pdxjohnny thanks for confirming. Then it is it, and the pip main branch has a fix. You way want to do a quick test to confirm.

[johnandersen777]

@sbidoul Perfect! It works! Thank you!!!

[pradyunsg]

And... it's online.

• https://pypi.org/project/pip/21.3.1/
• https://github.com/pypa/get-pip/pull/132

I'll file a PR against CPython in a few minutes.

---

Unless there's a show stopper issue over the weekend, I expect this to be the final pip release for this year. :)

[pradyunsg]

And, that's a wrap for 21.3.

See y'all next year; unless I get an email from security@python.org. :)