Possible issue on windows with wrong environment content

[FlorianWilhelm]

Hi @juftin, thanks for your cool work on hatch-pip-compile.

I just started to migrate some of my hatch projects to hatch + hatch-pip-compile but on Github CI Windows systems I am running into the problem that hatch seems to build different venvs compared to Linux/Macos, most pkgs are missing. Find the details here, especially hatch run pip list command in the Run tests and track coverage step.

I tried already many things like installing hatch with pipx and installing hatch with pip, which leads on Windows to an error

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'c:\\hostedtoolcache\\windows\\python\\3.11.9\\x64\\scripts\\hatch.exe' -> 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pip-uninstall-zc7cmf0j\\hatch.exe'

Find the details here.

Do you have any idea what the problem might be and where to look into? Everything is fine on Linux and MacOS and I am running out of ideas here. Thank you :-)

[juftin]

Dang, that's a tricky one. I will dig into this and see if I can't figure it out (I'll jump into this as soon as I can). The missing dependencies is certainly strange to me. I don't have access to a Windows machine personally, so I'll probably begin by increasing the hatch and hatch-pip-compile verbosity to see if I can't detect an obvious error.

[juftin]

Your project wouldn't be affected since it's not using constraint environments, but I have seen some regression issues on hatch >= 1.10.x. I also wonder what happens if you try to use hatch~=1.9.7.

I'm trying to resolve these issues as fast as I can and get testing for 1.10 and 1.11 (both less than 3 weeks old at this point), but if that resolves your issue it would certainly make a fix more urgent.

[FlorianWilhelm]

Thanks for your quick response. I set hatch to hatch~=1.9.7 but still the same problem occurs. Since @ofek develops on Windows, as far as I know, I would be quite surprised if there are any problems with hatch itself. My guess would be that pip-tools is doing something strange here, why is it trying to uninstall hatch anyway? My interpretation is that somehow it breaks out of the virtualenv created by hatch on Windows, which would at least explain what we are seeing.

[juftin]

Yeah, that's definitely the case here. You have pip-compile-installer set to pip-sync so it's trying to uninstall everything from the environment that doesn't match its input. And you're right, it's escaping your default virtual environment and instead operating on hatch's own environment.

Behind the scenes hatch-pip-compile calls python -m piptools sync ... but there should be a "safe activation" that prevents this from escaping the virtual environment (and it seems like this isn't working right in your windows test environment):

with self.environment.safe_activation():
    self.environment.install_dependencies()

As a side note, using pip-sync as your installer can have some unintended consequences when you are installing the same lockfile across platforms. I wrote a little bit about this here: https://juftin.com/posts/2024-01-09-hatch-pip-compile/#platform--python-version-specific-requirements

[FlorianWilhelm]

Okay, thanks. So I changed the pip-compile-installer to the default pip, deleted all lock files, ran hatch env prune and

❯ hatch run ls; hatch run lint:ls; hatch run docs:ls

I get these errors but the command nonetheless completes successfully:

pre-commit installed at .git/hooks/pre-commit
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ultimate-notion 0.4.post1.dev35+ge3c9320.d20240524 requires emoji~=2.8, which is not installed.
ultimate-notion 0.4.post1.dev35+ge3c9320.d20240524 requires mistune~=3.0, which is not installed.
ultimate-notion 0.4.post1.dev35+ge3c9320.d20240524 requires notion-client~=2.2, which is not installed.
ultimate-notion 0.4.post1.dev35+ge3c9320.d20240524 requires numpy~=1.24, which is not installed.
ultimate-notion 0.4.post1.dev35+ge3c9320.d20240524 requires pandas~=2.0, which is not installed.
ultimate-notion 0.4.post1.dev35+ge3c9320.d20240524 requires pydantic~=2.0, which is not installed.
ultimate-notion 0.4.post1.dev35+ge3c9320.d20240524 requires tabulate>=0.9, which is not installed.

Any idea why those errors are shown?

But now Github CI/CD works :-)

Thanks a lot for your help!

[juftin]

Any idea why those errors are shown?

Yes! I do actually know this one, it's related to this: https://github.com/juftin/hatch-pip-compile/issues/46 (it's a non-issue but annoying nonetheless)

Basically the hatch workflow is to install the package first and then the dependencies. This is a little tricky with hatch-pip-compile because when you pip install ultimate-notion it tells pip that pandas~=2.0 is an acceptable requirement so maybe it installs pandas==2.3.0 (let's pretend this is the latest version). Then it moves onto your lockfile which actually says that pandas==2.2.2 should be installed - so it would install pandas twice (and maybe leave behind some undesired dependencies). So instead of doing that I tell hatch-pip-compile to run pip install ultimate-notion --no-deps which results in pip complaining with your attached message, and then follow up with pip install -r lockfile which resolves pip's previous concerns.

[juftin]

The fact that pip-sync fails as the installer in windows is a little concerning. This is telling me that the safe_activation isn't working as intended for this. I think I can actually resolve this by explicitly passing the --python-executable argument to pip-sync https://pip-tools.readthedocs.io/en/stable/cli/pip-sync/

[FlorianWilhelm]

Yes! I do actually know this one, it's related to this: https://github.com/juftin/hatch-pip-compile/issues/46 (it's a non-issue but annoying nonetheless)

Thanks, got it. So this is completely intended and you could suppress it by setting stderr to subprocess.DEVNULL?