search

pypa / flit

Simplified packaging of Python modules
https://flit.pypa.io/
BSD 3-Clause "New" or "Revised" License
2.15k stars 131 forks source link

Cannot build even most simple packages #560

Open swaldhoer opened 2 years ago

swaldhoer commented 2 years ago

Problem

I cannot build the most simple packages using flit, e.g., flit_core/flit_core/tests/samples/with_data_dir. The reason ist, that pip install . is failing.

Details

Versions

PS C:\Users\stefa\Documents> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      19043  0
PS C:\Users\stefa\Documents> python --version; python -m flit --version; python -m pip --version
Python 3.8.2
Flit 3.7.1
pip 22.1.2 from C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pip (python 3.8)

Test case

PS C:\Users\stefa\Documents> git clone https://github.com/pypa/flit.git
PS C:\Users\stefa\Documents> cd .\flit\
PS C:\Users\stefa\Documents\flit> git rev-parse HEAD
048c87c380ac41efc4b26222114e54f6581c64f6

Test

PS C:\Users\stefa\Documents\flit> cd flit_core\flit_core\tests\samples\with_data_dir
PS C:\Users\stefa\Documents\flit\flit_core\flit_core\tests\samples\with_data_dir> flit install
Extras to install for deps 'all': {'.none'}             I-flit.install
Das System kann den angegebenen Pfad nicht finden.
Processing c:\users\stefa\documents\flit\flit_core\flit_core\tests\samples\with_data_dir
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: docutils in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (0.16)
Requirement already satisfied: requests>=2.18 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (2.25.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (1.26.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2020.12.5)
Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2.10)
Building wheels for collected packages: module1
  Building wheel for module1 (pyproject.toml) ... done
  Created wheel for module1: filename=module1-0.1-py3-none-any.whl size=1740 sha256=677808e82c4ec2b88721c46911a499381b6782328fdff551231d4aef488d75cd
  Stored in directory: C:\Users\stefa\AppData\Local\Temp\pip-ephem-wheel-cache-co9e6l2k\wheels\ee\72\a1\0576b8767488b740bf50fcca82f18326360c973eac1cd1fddf
Successfully built module1
Installing collected packages: module1
  Attempting uninstall: module1
    Found existing installation: module1 0.1
    Uninstalling module1-0.1:
      Successfully uninstalled module1-0.1
Successfully installed module1-0.1
Traceback (most recent call last):
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\Scripts\flit.exe\__main__.py", line 7, in <module>
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flit\__init__.py", line 192, in main
    Installer.from_ini_path(args.ini_file, user=args.user, python=python,
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flit\install.py", line 432, in install
    self.install_with_pip()
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\site-packages\flit\install.py", line 370, in install_with_pip
    check_call(cmd, shell=shell)
  File "C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\stefa\\AppData\\Local\\Programs\\Python\\Python38-32\\python.exe', '-m', 'pip', 'install', '.']' returned non-zero exit status 1.
swaldhoer commented 2 years ago

I should mention that, python -m pip install . works for the test case:

PS C:\Users\stefa\Documents\flit\flit_core\flit_core\tests\samples\with_data_dir> python -m pip install .
Processing c:\users\stefa\documents\flit\flit_core\flit_core\tests\samples\with_data_dir
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests>=2.18 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (2.25.1)
Requirement already satisfied: docutils in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from module1==0.1) (0.16)
Requirement already satisfied: idna<3,>=2.5 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (4.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (1.26.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\stefa\appdata\local\programs\python\python38-32\lib\site-packages (from requests>=2.18->module1==0.1) (2020.12.5)
Building wheels for collected packages: module1
  Building wheel for module1 (pyproject.toml) ... done
  Created wheel for module1: filename=module1-0.1-py3-none-any.whl size=1740 sha256=677808e82c4ec2b88721c46911a499381b6782328fdff551231d4aef488d75cd
  Stored in directory: C:\Users\stefa\AppData\Local\Temp\pip-ephem-wheel-cache-apq1sz90\wheels\ee\72\a1\0576b8767488b740bf50fcca82f18326360c973eac1cd1fddf
Successfully built module1
Installing collected packages: module1
  Attempting uninstall: module1
    Found existing installation: module1 0.1
    Uninstalling module1-0.1:
      Successfully uninstalled module1-0.1
Successfully installed module1-0.1
PS C:\Users\stefa\Documents\flit\flit_core\flit_core\tests\samples\with_data_dir> Get-Command foo

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     foo.exe                                            0.0.0.0    C:\Users\stefa\AppData\Local\Programs\Python\Python38-32\Scripts\foo.exe
takluyver commented 2 years ago

The output looks like pip installs it correctly, but then returns a non-zero exit code, which means something has gone wrong. The traceback is just saying that pip indicated an error.

I notice that there's a random German error message ("Das System kann den angegebenen Pfad nicht finden.") mixed in with the English output. I don't know where that's coming from, but maybe it's related somehow.

swaldhoer commented 2 years ago

I have some idea. How does flit invoke the python -m pip ...? Does it use shell=True?

bpabel commented 2 years ago

I'm not sure if this is the cause, but your pyproject.toml lists a console script entrypoint for module1:main, but that function does not exist in your module.

takluyver commented 2 years ago

I have some idea. How does flit invoke the python -m pip ...? Does it use shell=True?

On Windows it does, yes:

https://github.com/pypa/flit/blob/3edbd2f3e3b97a057fc785519c1617e2d78114fa/flit/install.py#L369-L370

Edit: this was changed in 414571d7fd5ccc7ac852a8449edcaaf59f407902, with a comment saying "On Windows, shell needs to be True to pick up our local PATH when finding the Python command."

swaldhoer commented 2 years ago

So first of all, thanks @takluyver you pointed me in the right direction.

So this particular issue was all about a wrong setup on my second machine.

However, I do not really understand why flit needs to use shell=True for subprocesses on Windows? I had a look at the commit and Pull request, but I don't really get it. Would it not be sufficent to use just the env-keyword of the subprocess package and by thgat use the correct PATH?

nanonyme commented 1 year ago

It most definitely is cleaner to use shutil.which to resolve executable path than to use shell=True. It is available on Python 3.3 and newer.

takluyver commented 1 year ago

I prefer not to use shell=True if possible, and Python 3.5 was already out when I made that commit, so I assume there was a reason I didn't use shutil.which(). But sadly my notes at the time weren't sufficient to figure out what that was. I vaguely remember something odd about the way that virtualenvs behaved on Windows, but I don't remember either the details or whether it was related to this specific change.

We have Windows CI, so if you want to make a PR to use shutil.which(), then if it passes CI, I'm happy to go with it and see if any complaints turn up.

swaldhoer commented 1 year ago

It most definitely is cleaner to use shutil.which to resolve executable path than to use shell=True. It is available on Python 3.3 and newer.

I don't understand why shutil.which() would fix the problem? The problem seemed to be, that the enviroment variables are not updated as the process would inheriate from the parent process and therefore the problem would still exist?

Using shutil.which() as well as an updated version of PATH passed to the subprocess should be the correct solution.

connorferster commented 6 months ago

Just wanted to add that I had a similar problem. I went into the Registry Editor and updated the key suggested by @swaldhoer. In my case it just some partial string in there like "if ever" or something (don't recall exactly). I deleted it leaving an empty field (the default setting) and it fixed the problem.

nanonyme commented 6 months ago

It most definitely is cleaner to use shutil.which to resolve executable path than to use shell=True. It is available on Python 3.3 and newer.

I don't understand why shutil.which() would fix the problem? The problem seemed to be, that the enviroment variables are not updated as the process would inheriate from the parent process and therefore the problem would still exist?

Using shutil.which() as well as an updated version of PATH passed to the subprocess should be the correct solution.

The core part of shutil.which() approach was getting rid of the cmd.exe in between that may mess things up.