Problematic conversion of packages names

[mcyprian]

In many cases it is problem to convert PyPI name to name of Fedora package, because there is not strict uppercase/lowercase rule in naming of Fedora packages:

PyPI	Fedora
Babel	python-babel
MarkupSafe	python-markupsafe
Jinja2	python-jinja2
Sphinx	python-sphinx
Cython	Cython
IPy	python-IPy
PyYAML	PyYAML

[hroncok]

Current guidelines: https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Case_Sensitivity

Perl exception:

The exception to this is for perl module packaging. The CPAN Group and Type should be capitalized in the name, as if they were proper nouns.

Proposal for Python exception:

The exception to this is for Python packages. Python packages (such as packages named as python-<module>) must use lowercase only. In case the module is named with uppercase letters on PyPI, the package should also provide that variant.

Example:

PyPI:     Fiona
Name:     python-fiona
Provides: python-Fiona%{?_isa} = %{version}-%{release}

Note: Only use %{?_isa} here on arched packages.

[mcyprian]

This seems to be one of the most frequent reasons that generated spec file is not capable of being built. @hroncok, @bkabrda couldn't we solve this problem or at least successfully convert more dependency names using DNF API query to check if dependency package exists in Fedora repositories?

Example: we've extracted dependency MarkupSafe

In [68]: query = base.sack.query()
In [70]: [pkg.name for pkg in query if "MarkupSafe".lower() in pkg.name.lower()]
Out[70]: ['python-markupsafe', 'python3-markupsafe']

[hroncok]

This might be the solution, but... How long does this take? If too much time, can we get the lists of Python packages from somewhere else, e.g. from portingdb? Does poritngdb have some sort of API (any sort of machine readable output)?

Back to your proposal: The logic behind this needs to be more clever than simply checking if str is in str. What if the dependency name would be simply A?

I see two possible better ways out (none of them happening any time soon and none of them in the direct hands of pyp2rpm project):

• Wait for the pypi() provides @torsava is working on
• Make the Python packaging guidelines change outlined in my previous comment happen (I've never got any feedback, maybe because I've never explicitly asked for it)

Another solution for the time being might be to get list of all Python packages in Fedora with some uppercase letters in their names (or in any other way problematic names) (there will not be so many of them I guess) and hardcode them in pyp2rpm (or put them in a YAML file, whatever). In other cases, just call .lower() on the Python package name, prepend it with pythonX- and hope for the best. (There might also be a slight issue with python2- prefix, because we want to encourage the use of it, but not all packages use it yet. Back to repoquery/portingdb probing I guess?)

Anyway, if you'll code some logic/magic (even based on hardcoded list), please make it happen as a separate Python module/package, reusable from outside pyp2rpm - this will make it easier to update it and also might help @torsava when filing bugs or when automating the provides generating process.

[mcyprian]

Speed doesn't seems to be problem here. Initialization of Query object takes about 0.3s. Execution of simple script including initialization of Query object, 100 iterations over it and string comparison takes still less than one second. I think it is not a disaster compared to other processes in pyp2rpm (distutils command, installation of package to virtualenv).

The example shows naive implementation to demonstrate how does DNF API query works.

The global solutions you mentioned are fine, but in my opinion name convertor would be good temporary fix of the issue. It seems not to be hard to implement it (I guess it could be ready in a week) and it might help in many cases.

I don't like idea of solution based on hardcoded list very much, so I will try to implement name convertor checking Fedora repositories directly as a separate package. The question is if DNF API or portingdb si the right choice at this moment.

[hroncok]

If DNF repoquery is fast, go for it.

[jwmullally]

+1 for the proposal @hroncok. It seems like the only sensible solution, and its what stdeb and pip2arch does (and already here for Mageia).

From https://www.python.org/dev/peps/pep-0426/#name

All comparisons of distribution names MUST be case insensitive, and MUST consider hyphens and underscores to be equivalent.

... which results in people having all kinds of cases in requirements.txt, setup.py, pkgname arguments etc. Because RPM is case sensitive, its clear lowercase should be used.

This will only become more painful as more Python packages enter Fedora, or more people use great tools like pyp2rpm in their build strategies (containers, VMs...).

---

FYI, with the current release (pyp2rpm 3.2.1) I am still seeing this case-mismatch issue. (don't test any fixes on these PyVISA packages, there are also bugs with Sphinx document generation and module name handling which I'm looking into).

$ sudo dnf install python3-PyVISA-1.8-1.fc25.noarch.rpm python3-PyVISA-py-0.2-1.fc25.noarch.rpm
Error: nothing provides python3-pyvisa >= 1.8 needed by python3-PyVISA-py-0.2-1.fc25.noarch

$ rpm -qp --provides python3-PyVISA-1.8-1.fc25.noarch.rpm
python3-PyVISA = 1.8-1.fc25
python3.5dist(pyvisa) = 1.8
python3dist(pyvisa) = 1.8

$ rpm -qp --requires python3-PyVISA-py-0.2-1.fc25.noarch.rpm
python(abi) = 3.5
python3-pyvisa >= 1.8
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1

[jwmullally]

From the proposal:

In case the module is named with uppercase letters on PyPI, the package should also provide that variant.

Is this really needed? Given the case-insensitivity is effectively set in stone in Python-land, would this part just be baggage/confusion that people may mistakenly use or have to be maintained?

[hroncok]

Not really.

[ncoghlan]

Would it be possible for this to move forward based on the pythonXYdist(name) dependencies that are now in Fedora?

[mcyprian]

Absolutely, this issue was one of the reasons to add PyPI provides to RPM.

[mcyprian]

PR #127 was merged. There is a command line option to enable/disable the convertor based on automatic provides. It is disabled by default at this moment. I guess we can start using it by default for Fedora after F27 mass rebuild.

[hroncok]

I see I was assigned here, sorry for not taking any action. I don't have free cycles to work on pyp2rpm :(