Deploy multiple python3 flavors

[bnavigator]

This is intended as a tracker issue for the introduction of multiple python3 flavors and better reference in changelogs. The python-rpm-macros package with enabled flavors of python36 and python38 is in Staging:N and as expected various packages fail.

• related PRs: #64, #65, #67, #70, #71
• Trello card
• Mailinglist thread with some hints what to look out for.

[bnavigator]

Wrong %python3_site(arch|lib) https://github.com/openSUSE/python-rpm-macros/pull/64#issuecomment-715373348

• [x] python-six -- sr#843620
• [x] python-ordered-set -- sr#843677
• [x] python-libvirt-python -- sr#843690 --> sr#844156
• [x] zypp-plugin -- :ballot_box_with_check: https://github.com/openSUSE/zypp-plugin/pull/3 --> sr#844816
• [x] python-cmd2 -- sr#852788 --> sr#852846
• [x] python-astroid -- sr#853331 --> sr#853413

[bnavigator]

• [x] python-sip -- sr#843659 sr#843962 --> sr#844384

[bnavigator]

Python subpackages of other packages (see also #15, #73, #75, #76)

• [x] libxml2:python -- sr#843675 -> sr#844894
• [x] sanlock -- sr#843688 -> sr#844480
• [x] libpfm -- sr#849907
• [x] miniupnpc -- sr#849906
• [x] net-snmp -- sr#849908 --> sr#853302
• [x] qpid-proton see #75 -- sr#848218 --> sr#848247 , sr#850683 --> sr#851380, sr#863025
• [x] setools -- #73 sr#846428
• [x] protobuf -- sr#849909 :white_check_mark: (but the maintainer is not convinced yet) --> sr#850509
• [x] javapackages-tools-extras (python3-javapackages) -- sr#849910 --> sr#850199
• [x] subunit -- sr#852784 --> sr#852848
• [x] gpgme -- sr#853316 -- the makefile already looks for all available Python interpreters. We just need to BuildRequire the right packages and package the results :grinning:!
• [x] gaupol : python-aeidon -- sr#853408 --> sr#853597
• [x] xapian-bindings -- sr#856773

With #79, most of these packages need another iteration.

[bnavigator]

• [x] python-semanage -- the enhanced %python_expand macro (#64) does not like to substitute "$python" with the executable because of the ". --> #67

[bnavigator]

%python3_only and %ifpython3

• [x] python-pytest -- sr#844811 --> sr#844820
• [x] python-pip -- sr#844950 --> sr#845027
• [x] python-pytest5 -- sr#845670 --> sr#845674
• [x] python-pycairo -- sr#852937 --> sr#852941
• [x] python-coverage -- sr#852737 --> sr#852851
• [x] python-dbus-deviation -- sr#852774 --> sr#852870
• [x] python-notify2 -- sr#854949 --> sr#854954
• [x] python-docutils -- sr#855362
• [x] python-atspi -- sr#855513
• [x] python-pysmbc -- sr#863075 --> sr#863124
• [x] a bunch of packages not in Ring1 from devel:languages:python:jupyter -- see also https://github.com/openSUSE/python-rpm-macros/issues/66#issuecomment-740594511

[bnavigator]

BuildRequires: python3-foo where it is not enough to have one by the default provider (#69) but you need it for the specific flavor

• [x] python-iniparse sr#844934 --> sr#845026
• [x] python-typing_extensions -- sr#847634 :heavy_check_mark:
• [x] python-pyxattr sr#848432 --> sr848863 :heavy_check_mark: (waiting on importlib_metadata)
• [x] python-pep517 requires flit-core, but not for python2 -- sr#852717 --> sr#852771
  • [x] python-outcome -- sr#853200 --> sr#853472
  • [x] python-Sphinx2 -- sr#853186 --> sr#853195
  • [x] python-future -- sr#852977 --> sr#853083
• [x] python-dill -- sr#853442 --> sr#853471
• [x] python-virtualenv -- sr#853445 to @MeggyCal's preparation of the update to 20.2 --> .. --> sr#853600
• [x] python-joblib --> sr#853473

[mcepl]

%python_exec is broken with multiple python3 flavours #72

[bnavigator]

70 already fixes that

[bnavigator]

Wrong shebang replacement

• [x] python-psutil -- sr#845689 --> sr#845690

@mcepl, how do we deal with this one? The fix will require the new $python expansion, so I cannot just submit it to d:l:p when python-rpm-macros is stuck in Staging:N

@mcepl inline reply: For now a private definition of the macro in the package itself, possibly conditional, i.e. something like:

%{?!python_module:%define python_module() python-%{**} python3-%{**}}

We can always remove it later.

Edit: Could you please post a separate reply rather than editing my post?

[bnavigator]

Conflicting files outside of %python_site(lib|arch)

• [x] python-tqdm -- sr#845737 --> sr#845787
• [x] python-hotdoc -- sr#845773 --> sr#847050
• [x] python-wheezy.template -- sr#845963 --> sr#846099
• [x] translate-toolkit (also %python_subpackages without proper %files headers) -- sr#853407 --> sr#853598

[mcepl]

Conflicting files outside of %python_site(lib|arch)

* [ ]  [python-tqdm](https://build.opensuse.org/package/show/openSUSE:Factory:Staging:N/python-tqdm)

I think this clearly calls for alternatives, however (and I guess Sphinx as well).

[scarabeusiv]

Why not just split bash completion file into subpkg, that would be the most proper solution.

[bnavigator]

I found a way that works with the old and new python_expand: sr#845689

[mcepl]

I found a way that works with the old and new python_expand: sr#845689

??? This is not python-tqdm. ???

[bnavigator]

No it is psutil. Granted the history of https://github.com/openSUSE/python-rpm-macros/issues/66#issuecomment-721016747 makes it confusing :)

tqdm is in the making.

[bnavigator]

python-tqdm-bash-completion: Approach for the Supplements: tag in sr#845737 is a bit hacky. Maybe we need to enhance %python_module or create a new macro for this kind of rich dependency specification.

[bnavigator]

(and I guess Sphinx as well).

python-Sphinx is a case of #5.

[bnavigator]

duplicate files found by %find_lang

• [x] python-Sphinx -- ~sr#845821 --> sr#845822~ sr#845960 --> sr#846078
• [x] python-Sphinx2 -- sr#845824 --> sr#845826

.. only tick off when the fix has landed in Staging:N

[bnavigator]

• [x] many packages unresolvable because they look for python3-dateutil -- #77

[bnavigator]

Duh, the new python-sip had a wrong files section header.

• [x] python-sip -- sr#848979

[bnavigator]

importlib_metadata is missing. Python38 provides the stdlib importlib.metadata, but python36 does need the extra package which is already provided in SLE and Leap.

• [x] python-importlib-metadata -- sr#849805 --> sr#850139
• [x] python-importlib-resources -- sr#849806 --> sr#850589
• [x] python38 provides change -- sr#849807 --> sr#850380

[bnavigator]

After providing python36-importlib-metadata, pytest is usable. Some pytest runs in my branched staging:N failed because they found tests in _build.python36 --> #78

[bnavigator]

Additional provides of python3-othername is not enough:

• [x] python-pycups provides python3-cups -- sr#849822 --> sr#849838
• [x] python-python-xlib -- sr#855004 --> sr#855356
• [x] python-Mako -- sr#855003 --> sr#855358

[bnavigator]

Incorrect expansion of $python in %python_expand

• [x] python-dbus-python -- sr#850243 sr#853418, #83
• [X] python-pyqt-rpm-macros: %pyqt_install_examples macro. -- sr#850603
• [x] python-pybind11 -- sr#853336 #83
• [x] python-Sphinx:test -- sr#853306

[mcepl]

Incorrect expansion of $python in %python_expand

Could you please elaborate on what’s wrong? Is the linked SR trying to fix the problem or is it a reproducer?

[bnavigator]

Incorrect expansion of $python in %python_expand

Could you please elaborate on what’s wrong? Is the linked SR trying to fix the problem or is it a reproducer?

It is trying to fix the problem. Without it %configure PYTHON="/usr/bin/$python" expands to /usr/bin/usr/bin/python3.8

[mcepl]

It is trying to fix the problem. Without it %configure PYTHON="/usr/bin/$python" expands to /usr/bin/usr/bin/python3.8

And how many Spec files sets $PYTHON? Wouldn’t it be easier just to change them to %configure PYTHON="$python" and be done with it? I see two: python-dbus-python and gtk-doc.

[bnavigator]

IIRC I did not test it. But my assumption was that there is a reason the variable for %configure needs to specify the absolute filepath. Until the update for python-rpm-macros will make it out of Staging:N, $python is replaced to python3 without the bindir. $(which $python) will always return the correct absolute path, before and after the update.

[mcepl]

IIRC I did not test it. But my assumption was that there is a reason the variable for %configure needs to specify the absolute filepath. Until the update for python-rpm-macros will make it out of Staging:N, $python is replaced to python3 without the bindir. $(which $python) will always return the correct absolute path, before and after the update.

I see $PYTHON used only in contexts like `PYTHON_SITE_PKG=$PYTHON -c "import distutils.sysconfig; print (distutils.sysconfig.get_python_lib(0,0));"``` and then full path to the binary should work just fine.

[bnavigator]

full path or basename depending on correct $PATH?

[bnavigator]

I see two: python-dbus-python and gtk-doc.

gtk-doc does not use %python_expand. No problem there.

[bnavigator]

With %configure PYTHON="$python":

[   32s] + ../configure --host=x86_64-suse-linux-gnu --build=x86_64-suse-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-dependency-tracking PYTHON=python3
...
[   36s] checking for python... no
[   36s] configure: error: Cannot find python in your system path
[   36s] error: Bad exit status from /var/tmp/rpm-tmp.wWaBES (%build)
[   36s] 

[mcepl]

OK, so you are right. Just why to both set up the environmental variable $PYTHON and do it once more on %configure command line?

[bnavigator]

I don't know. It was introduced here: https://build.opensuse.org/request/show/620627. @scarabeusiv maybe just forgot to remove the export?

[bnavigator]

python-dbus-python SR updated: https://build.opensuse.org/request/show/850243

[mcepl]

python-dbus-python SR updated: https://build.opensuse.org/request/show/850243

Oh, damn. I cannot approve that submit request.

[bnavigator]

Let's see if a maintainer from the Base:System project is willing to. This is not urgent.

[mcepl]

Let's see if a maintainer from the Base:System project is willing to. This is not urgent.

Well, not urgent, but not whatever … Staging:N won’t be let through without it.

[bnavigator]

Yeah but there will be enough other packages to fix until this is the last blocker. I don't see it being let through anytime soon.

[bnavigator]

• [x] python-numpy has a weird pytest call with wrong sitearch expansion and a conflicting f2py3 entry-point sr#851543 and #80 --> sr#851900

[bnavigator]

pytest called without macro and thus importing from non-ignored _build directory with wrong flavor

• [x] python-async_generator -- sr#852715 --> sr#852725
• [x] python-typed-ast -- sr#853092 --> sr#853096
• [x] python-cairocffi -- replace pytest-runner sr#853627

[bnavigator]

Requires python > 3.6

• [x] python-ipython with conflicting ipython3 and iptest3 (when python39 will be there) sr#852728 --> sr#852845 :ballot_box_with_check: :test flavor still unresolvable

[bnavigator]

replace %py3_compile with python_compileall

• [x] python-jaraco.functools -- sr#852726 --> sr#852727
• [x] python-jaraco.collections -- sr#85330 --> sr#853440

[bnavigator]

missing %python_expand

• [x] python-scikit-build -- sr#852764 --> sr#852843

[bnavigator]

• [x] python-pyxdg The %python_exec .. $python/ did not expand correctly (/ not added in #67), but a version bump made the whole construct obsolete anyway-- sr#852787 --> sr#852844

[bnavigator]

Failures in python 3.6 build

• [x] python-aiohttp -- sr#853108 --> sr#853123
• [x] python-wrapt -- sr#853000 --> sr#853084
• [x] python-CherryPy --> sr#855878 -- sr#855889
• [x] python-build -- sr#856709 --> sr#856769
• [x] python-mocket:test -- sr#856705 --> sr#856712
• [x] python-pylint -- sr#856768 -- sr#856770

[bnavigator]

Use of python3 flavor

• [x] python-manuel -- sr#853004 --> sr#853080

[bnavigator]

Fix by update or removal of unnecessary BuildRequirements

• [x] python-setproctitle -- sr#853003 --> sr#853085
• [x] python-gunicorn -- sr#853088 --> sr#853097
• [X] python-m2r -- sr#852781 --> sr#852849
• [x] python-cloudpickle -- sr#854293

[bnavigator]

Requires python > 3.6

• [x] python-curio -- sr#852974 --> sr#853082
  • Upstream reverted back to python >= 3.6 for the current version, but the unit tests fail: dabeaz/curio#336. We might need to fix this because the dependency tree is huge.
  • [x] revert sr#854256
  • [x] python-sniffio -- sr#853087 --> [sr#853098] (https://build.opensuse.org/request/show/853098)
  • [x] revert sr#854361
• [x] python-astropy (not in Ring1)
  • [x] python-skyfield
  • [X] python-pytest-astropy-header
  • [X] python-asdf

[bnavigator]

Missing dependency to libsqlite now:

• [x] breezy

I am working on this, but it is BS … nobody should ever need pysqlite2 ever (when sqlite3 is in the standard library). I will need to investigate it more on Monday. -- @mcepl