Installing a downloaded tar.gz file fails because of setuptools-scm

[zerothi]

First Time Issue Code

Not applicable.

Description

I cannot install lmfit when downloading the tar.gz file from here

A Minimal, Complete, and Verifiable example

Basically:

# WORKS:
python3 -m pip install lmfit
# DOES NOT WORK
python3 -m pip install lmfit-1.0.3.tar.gz

I don't have problems doing this with other packages, perhaps an pyproject.toml file would help in this case. But I am not sure.

The problem is that some HPC installations does not have internet access and thus requires manual copying of files for installation.

Error message:

  ERROR: Command errored out with exit status 1:
   command: /home/software/test-venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-f0joy_ek/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-f0joy_ek/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-somcw8fu
       cwd: /tmp/pip-req-build-f0joy_ek/
  Complete output (29 lines):
  /opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools/__init__.py:148: SetuptoolsDeprecationWarning: setup_requires is deprecated. Supply build dependencies using PEP 517 pyproject.toml build-requires.
    warnings.warn(
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-f0joy_ek/setup.py", line 23, in <module>
      setup(name='lmfit',
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools/__init__.py", line 159, in setup
      return distutils.core.setup(**attrs)
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/distutils/core.py", line 108, in setup
      _setup_distribution = dist = klass(attrs)
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools/dist.py", line 453, in __init__
      _Distribution.__init__(
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/distutils/dist.py", line 292, in __init__
      self.finalize_options()
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools/dist.py", line 831, in finalize_options
      ep(self)
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools/dist.py", line 852, in _finalize_setup_keywords
      ep.load()(self, ep.name, value)
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools_scm/integration.py", line 65, in version_keyword
      dist.metadata.version = _get_version(config)
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools_scm/__init__.py", line 177, in _get_version
      parsed_version = _do_parse(config)
    File "/opt/gnu/11.2.0/python/3.9.7/lib/python3.9/site-packages/setuptools_scm/__init__.py", line 135, in _do_parse
      raise LookupError(
  LookupError: setuptools-scm was unable to detect version for '/tmp/pip-req-build-f0joy_ek'.

  Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

  For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
  ----------------------------------------
WARNING: Discarding file:///home/software/bash-build/.archives/lmfit-1.0.3.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Version information

Link(s)

[reneeotten]

why don't you use python setup.py install --user instead? I am not sure whether one is supposed to be able to install using pip from an archive, have never tried that. I can take a look and see if that can be fixed, for now installing it the "usual" way using setup.py will get you up-and-running.

[reneeotten]

actually, it works fine for me.... provided that you first have the dependencies installed, the lmfit will install fine when using pip and the tarball from PyPI:

% python -m pip install lmfit-1.0.3.tar.gz
Processing ./lmfit-1.0.3.tar.gz
  Preparing metadata (setup.py) ... done

<snip>

Successfully built lmfit asteval future
Installing collected packages: numpy, future, uncertainties, scipy, asteval, lmfit
Successfully installed asteval-0.9.25 future-0.18.2 lmfit-1.0.3 numpy-1.21.4 scipy-1.7.3 uncertainties-3.1.6

[zerothi]

which setuptools (58.4.0) and setuptools_scm (3.2.3) do you use (mine)?

[reneeotten]

which setuptools (58.4.0) and setuptools_scm (3.2.3) do you use (mine)?

I am using 59.4.0 and 6.3.2 - having recent versions is always preferred... So just make sure you've updated setuptools, pip, and setuptools-scm to their latest versions before installing anything else. Or use python setup.py install --user as I suggested above.

[zerothi]

Sorry my, setuptools_scm is as yours (6.3.2). So that cannot be the cause.

The problem is the python build-community. In pep 517 and 621 it is recommended to never call python setup.py and instead rely on pip for installing stuff...
So this is now my goto pipeline.

I am just as puzzled about this here...

I'll try and post this in setuptools-scm to see if this is an upstream problem...

[reneeotten]

I'll try and post this in setuptools-scm to see if this is an upstream problem...

Probably not... given that it works for me, it has likely something to do with your system - what version of pip are you using? If you want it to work right now, just do python setup.py - I don't really care whether they recommend it or not, it works.

[zerothi]

Yeah, that is definitely the easiest. :)

Anyway, thanks. I'll ping you if the topic I created finds anything.

[newville]

@reneeotten @zerothi Thanks for looking into this.

The problem is the python build-community. In pep 517 and 621 it is recommended to never call python setup.py and instead rely on pip for installing stuff...

Yeah, I think I sort of agree with this sentiment. Having lived through distutils, easy_install, eggs, and seemingly endless churn in tools for installing and distributing packages, I am not 100% convinced that these PEPs really make things better. But I guess we should add a pyproject.toml file.

On setuptools_scm itself, I'm sort of unsure. I've seen similar problems with setuptools_scm in other projects. We basically never had problems with versioneer. Maybe https://github.com/jbweston/miniver is a reasonable alternative? No strong opinion...

[zerothi]

But you are not using versioneer now, no? Perhaps just adding this one again would solve possible problems? I don't know... In the linked issue you can see they recommend adding the pyproject.toml file.

I agree I can't figure out the zoo of packaging/installation methods, and so in my packages I try to enable everything I know ;) Yeah, not the best... But the eco-system sort of requires it.

[reneeotten]

I don't want to go back to using versioneer; pretty much everyone uses setuptools_scm nowadays and so that should work fine (again I cannot reproduce this specific problem, for me it works). I can/will look into adding a pyproject.toml file so that the "recommended" build tools will understand what to do with it.

[zerothi]

I don't want to go back to using versioneer; pretty much everyone uses setuptools_scm nowadays and so that should work fine (again I cannot reproduce this specific problem, for me it works). I can/will look into adding a pyproject.toml file so that the "recommended" build tools will understand what to do with it.

Ok, perfectly understandable! Thanks!

[newville]

@reneeotten I don't think that setuptools_scm is somehow more official than anything else (it is not part of setuptools, for example). If we need a third-party package for installation to work, I think we have to be pretty confident in that package and in everyone being able to install that. With versioneer or miniver, we just add a couple of files but do not add an external dependency that must be installed prior to the installation. I guess that setuptools_scm should be listed in requirements.txt and in install_requires.

Also, making a conda-forge package for lmfit required SETUPTOOLS_SCM_PRETEND_VERSION to be set. That was non-trivial.

I'm not opposed to keeping setuptools_scm, but there are costs. This issue shows one of them.

[reneeotten]

@reneeotten I don't think that setuptools_scm is somehow more official than anything else (it is not part of setuptools, for example).

True, but it's at least maintained by the "Python Packaging Authority" under which umbrella pretty much all popular packaging tools are developed.

If we need a third-party package for installation to work, I think we have to be pretty confident in that package and in everyone being able to install that.

Agreed, but this is the first issue about it that's reported (and I cannot reproduce it as it works fine for me). Additionally, we list two "supported" ways of installing lmfit: either using pip install from the PyPI or with python setup.py. The fact that a third way of using pip with the a downloaded archive is rumored not to work (which is surprising as it should be the same as doing pip install lmfit where the exact same file is downloaded from PyPI) could be considered a bug or one could just say "pick one of the two other options that do work".

With versioneer or miniver, we just add a couple of files but do not add an external dependency that must be installed prior to the installation. I guess that setuptools_scm should be listed in requirements.txt and in install_requires.

The advantage of using an external dependency is that we do not need to take care of (updating) it. The setuptools_scm dependency is listed in setup.py under setup_requires where it should be, I don't think one needs to list it additionally in requirements.txt.

Also, making a conda-forge package for lmfit required SETUPTOOLS_SCM_PRETEND_VERSION to be set. That was non-trivial.

As I said in another Issue, I have no experience with the conda recipes so I cannot comment on this.

I'm not opposed to keeping setuptools_scm, but there are costs. This issue shows one of them.

I don't think the issue here can be blamed on the use of setuptools_scm, it's just that recent versions of setuptools and pip are really trying hard to have people define their dependencies the new PEP-whatever-number. As I said above I will look into making this happen...

[newville]

@reneeotten I don't think that setuptools_scm is somehow more official than anything else (it is not part of setuptools, for example).

True, but it's at least maintained by the "Python Packaging Authority" under which umbrella pretty much all popular packaging tools are developed.

Well, so why is setuptools_scm a separate package then? Yet another package that has to be installed from PyPI. Is it because these are the people who want to make packaging "important".

Are these the same folks who decided that setup.cfg was needed, and now have pyproject.toml -- also a config file but with a different name and format. And is that replacing requirements.txt too? I honestly do not know. It seems like they are trying to prove that packaging is hard and that they are working on it. Well....

I'm certainly OK with following normal "best practices", but I also think it is completely fair to question the unrelenting churn in packaging and setup tools. I do not see why a config file is better than a python file. I do not see why "python setup.py" is now "never recommended".

Agreed, but this is the first issue about it that's reported (and I cannot reproduce it as it works fine for me)

Well, second if you count the problem with conda packaging, which is really just installing in a dedicated environment.

The fact that a third way of using pip with the a downloaded archive is rumored not to work (which is surprising as it should be the same as doing pip install lmfit where the exact same file is downloaded from PyPI) could be considered a bug or one could just say "pick one of the two other options that do work".

Yeah, I think the recommendation is to do pip install . in the source tree. I don't think pip install XXX.tar.gz makes sense.

The advantage of using an external dependency is that we do not need to take care of (updating) it. The setuptools_scm dependency is listed in setup.py under setup_requires where it should be, I don't think one needs to list it additionally in requirements.txt.

If I understand the report here correctly, that did not work.

I don't think the issue here can be blamed on the use of setuptools_scm, it's just that recent versions of setuptools and pip are really trying hard to have people define their dependencies the new PEP-whatever-number. As I said above I will look into making this happen...

Well, if we did not use setuptools_scm, I think we would not have had these issues. Anyway, being tied too closely with very limited range of versions for these tools would cause endless problems -- that cannot be right.

Personally, I think we are forced to spend way too much time and effort of these topics, at least in part because the packaging tools keep changing.

[zerothi]

@reneeotten I don't think that setuptools_scm is somehow more official than anything else (it is not part of setuptools, for example).

True, but it's at least maintained by the "Python Packaging Authority" under which umbrella pretty much all popular packaging tools are developed.

Well, so why is setuptools_scm a separate package then? Yet another package that has to be installed from PyPI. Is it because these are the people who want to make packaging "important".

This is partially described here: https://www.python.org/dev/peps/pep-0517/#id20

Point is that setuptools is too hard to fix and any kind of change to setuptools will break backwards compatibility. So the idea is to have an upper level package which uses what-ever backend the user requests (as far as I understand it).

Are these the same folks who decided that setup.cfg was needed, and now have pyproject.toml -- also a config file but with a different name and format. And is that replacing requirements.txt too? I honestly do not know. It seems like they are trying to prove that packaging is hard and that they are working on it. Well....

I don't think requirements.txt will be replaced. As far as I have seen its format is very well defined.

However, regarding setup.cfg, see here: https://www.python.org/dev/peps/pep-0518/#id40 The setup.cfg is not well-defined and thus required restructuring.

I'm certainly OK with following normal "best practices", but I also think it is completely fair to question the unrelenting churn in packaging and setup tools. I do not see why a config file is better than a python file. I do not see why "python setup.py" is now "never recommended".

As far as I understand it, then the setup.py could use tools that were not present at install time and thus break building the package without the user knowning (consider importing package x at the top-level, and then in setup(..., build_requires='x'). This would not work for this reason. And also getting out of the monkeypatched setup.py all-over would be another goal (albeit extremely difficult to reach, if reachable at all).

The fact that a third way of using pip with the a downloaded archive is rumored not to work (which is surprising as it should be the same as doing pip install lmfit where the exact same file is downloaded from PyPI) could be considered a bug or one could just say "pick one of the two other options that do work".

Yeah, I think the recommendation is to do pip install . in the source tree. I don't think pip install XXX.tar.gz makes sense.

I agree that in this case it seems very odd. And perhaps it is just something I do, or have configured such that it interprets file installs differently from pypi installed sources.
In any case I find that XXX.tar.gz installs are very important, say for systems that for security reasons cannot have an internet connection.

The advantage of using an external dependency is that we do not need to take care of (updating) it. The setuptools_scm dependency is listed in setup.py under setup_requires where it should be, I don't think one needs to list it additionally in requirements.txt.

If I understand the report here correctly, that did not work.

Agreed, it should not be listed there. But it should be in the pyproject.toml file such that it knows in advance (as far as I understand it)...

I don't think the issue here can be blamed on the use of setuptools_scm, it's just that recent versions of setuptools and pip are really trying hard to have people define their dependencies the new PEP-whatever-number. As I said above I will look into making this happen...

Well, if we did not use setuptools_scm, I think we would not have had these issues. Anyway, being tied too closely with very limited range of versions for these tools would cause endless problems -- that cannot be right.

Personally, I think we are forced to spend way too much time and effort of these topics, at least in part because the packaging tools keep changing.

I completely agree. I am battling this my-self and have tried to really spend some time on it. And all I can find is that there is no good solution that just works. Hacks all-around, and then still there are some corner cases when users have different versions etc.

[newville]

On Thu, Dec 2, 2021 at 4:32 AM Nick Papior @.***> wrote:

@reneeotten https://github.com/reneeotten I don't think that setuptools_scm is somehow more official than anything else (it is not part of setuptools, for example).

True, but it's at least maintained by the "Python Packaging Authority" under which umbrella pretty much all popular packaging tools are developed.

Well, so why is setuptools_scm a separate package then? Yet another package that has to be installed from PyPI. Is it because these are the people who want to make packaging "important".

This is partially described here: https://www.python.org/dev/peps/pep-0517/#id20

Point is that setuptools is too hard to fix and any kind of change to setuptools will break backwards compatibility. So the idea is to have an upper level package which uses what-ever backend the user requests (as far as I understand it).

I'm having a hard time understanding this as different from change for the sake of change. Documentation and rationale about tools for packaging has been uniformly poor.

They are simply making it up as they go along. That's OK, just don't bother trying to convince me that they know what they are doing and that really, unlike all the previous changes, this time they are getting it right.

Are these the same folks who decided that setup.cfg was needed, and now

have pyproject.toml -- also a config file but with a different name and format. And is that replacing requirements.txt too? I honestly do not know. It seems like they are trying to prove that packaging is hard and that they are working on it. Well....

I don't think requirements.txt will be replaced. As far as I have seen its format is very well defined.

However, regarding setup.cfg, see here: https://www.python.org/dev/peps/pep-0518/#id40 The setup.cfg is not well-defined and thus required restructuring.

... so they added pyproject.toml, but it is not a replacement for or supersedes the poorly-defined setup.cfg? Looks like churn.

I'm certainly OK with following normal "best practices", but I also think

it is completely fair to question the unrelenting churn in packaging and setup tools. I do not see why a config file is better than a python file. I do not see why "python setup.py" is now "never recommended".

As far as I understand it, then the setup.py could use tools that were not present at install time and thus break building the package without the user knowning (consider importing package x at the top-level, and then in setup(..., build_requires='x'). This would not work for this reason. And also getting out of the monkeypatched setup.py all-over would be another goal (albeit extremely difficult to reach, if reachable at all).

The fact that a third way of using pip with the a downloaded archive is rumored not to work (which is surprising as it should be the same as doing pip install lmfit where the exact same file is downloaded from PyPI) could be considered a bug or one could just say "pick one of the two other options that do work".

Yeah, I think the recommendation is to do pip install . in the source tree. I don't think pip install XXX.tar.gz makes sense.

I agree that in this case it seems very odd. And perhaps it is just something I do, or have configured such that it interprets file installs differently from pypi installed sources. In any case I find that XXX.tar.gz installs are very important, say for systems that for security reasons cannot have an internet connection.

But tar.gz does not specify a Python packaging format. pip install XXX.tar.gz assumes that the tarball contains a folder with python code. OTOH, unpacking a tarball into a folder, moving into that folder and doing pip install . does make sense.

The advantage of using an external dependency is that we do not need to

take care of (updating) it. The setuptools_scm dependency is listed in setup.py under setup_requires where it should be, I don't think one needs to list it additionally in requirements.txt.

If I understand the report here correctly, that did not work.

Agreed, it should not be listed there. But it should be in the pyproject.toml file such that it knows in advance (as far as I understand it)...

I don't think the issue here can be blamed on the use of setuptools_scm, it's just that recent versions of setuptools and pip are really trying hard to have people define their dependencies the new PEP-whatever-number. As I said above I will look into making this happen...

Well, if we did not use setuptools_scm, I think we would not have had these issues. Anyway, being tied too closely with very limited range of versions for these tools would cause endless problems -- that cannot be right.

Personally, I think we are forced to spend way too much time and effort of these topics, at least in part because the packaging tools keep changing.

I completely agree. I am battling this my-self and have tried to really spend some time on it. And all I can find is that there is no good solution that just works. Hacks all-around, and then still there are some corner cases when users have different versions etc.

If the recommendation was to replace the existing and working versioneer with setuptools_scm and then setuptools_scm does not work or requires extra steps to make work, then it is completely reasonable to question that advice (and maybe even the motivation of that advice). We've seen such problems. Yes, setuptools_scm has some advantages. But our commitment is to making installation robust and easy. We are currently using setuptools_scm, but we are not committed or obligated to use it.