Cannot install Pants > 2.19 if home directory in `/etc/passwd` is a symlink.

[originalrkk]

Describe the bug The pants CLI works fine in all of our environments at version 2.19. When we change the version to 2.20 or 2.21 or 2.22, we see one of two errors, although it's not clear why some environments trigger one and some another, given that they're all on the same Ubuntu (perhaps quirks of the user setup):

$ pants --keep-sandboxes=on_failure --changed-since=origin/main lint
Bootstrapping Pants 2.20.0                                                                                                                                                                                                                                                                  
Installing pantsbuild.pants==2.20.0 into a virtual environment at /home/<REDACTED>/.cache/nce/60b<REDACTED>/bindings/venvs/2.20.0                                                                                                              
Failed to create Pants virtual environment.                                                                                                                                                                                                                                                 
Error: Command '['/bulk_data/home/<REDACTED>/.cache/nce/60b<REDACTED>/bindings/pex_root/venvs/591<REDACTED>/561<REDACTED>/bin/python', '/tmp/tmpp276ax9e.pex', 'venv', '--prompt', 'Pants
 2.20.0', '--compile', '--pip', '--collisions-ok', '--no-emit-warnings', '--disable-cache', '/home/<REDACTED>/.cache/nce/60b<REDACTED>/bindings/venvs/2.20.0']' returned non-zero exit status 1., output:                                      
-----                                                                                                                                                                                                                                                                                       
b'Traceback (most recent call last):\n  File "/bulk_data/home/<REDACTED>/.cache/nce/679<REDACTED>/cpython-3.9.18+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/runpy.py", line 197, in _run_module_as_main\n    re
turn _run_code(code, main_globals, None,\n  File "/bulk_data/home/<REDACTED>/.cache/nce/679<REDACTED>/cpython-3.9.18+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/runpy.py", line 87, in _run_code\n    exec(code
, run_globals)\n  File "/home/<REDACTED>/.pex/unzipped_pexes/9c1<REDACTED>/__main__.py", line 105, in <module>\n    from pex.pex_bootstrapper import bootstrap_pex\nModuleNotFoundError: No module named \'pex\'\n'                                                    
-----

Or

Bootstrapping Pants 2.22.0
Installing pantsbuild.pants==2.22.0 into a virtual environment at /home/<REDACTED>/.cache/nce/60b<REDACTED>/bindings/venvs/2.22.0
Failed to fetch https://github.com/pantsbuild/pants/releases/download/release_2.22.0/pants.2.22.0-cp39-linux_x86_64.pex: [22] HTTP response code said error (The requested URL returned error: 404)
Wasn't able to fetch the Pants PEX at https://github.com/pantsbuild/pants/releases/download/release_2.22.0/pants.2.22.0-cp39-linux_x86_64.pex.

Check to see if the URL is reachable (i.e. GitHub isn't down) and if pants.2.22.0-cp39-linux_x86_64.pex asset exists within the release. If the asset doesn't exist it may be that this platform isn't yet supported. If that's the case, please reach out on Slack: https://www.pantsbuild.org/docs/getting-help#slack or file an issue on GitHub: https://github.com/pantsbuild/pants/issues/new/choose.

Exception:

Command '['/home/<REDACTED>/.cache/nce/226<REDACTED>/ptex-linux-x86_64', 'https://github.com/pantsbuild/pants/releases/download/release_2.22.0/pants.2.22.0-cp39-linux_x86_64.pex']' returned non-zero exit status 1.
Error: Failed to establish atomic directory /home/<REDACTED>/.cache/nce/60b<REDACTED>/locks/install-ab9<REDACTED>. Population of work directory failed: Boot binding command failed: exit status: 1

Isolates your Pants from the elements.

Please select from the following boot commands:

<default> (when SCIE_BOOT is not set in the environment)  Detects the current Pants installation and launches it.
bootstrap-tools                                           Introspection tools for the Pants bootstrap process.
update                                                    Update scie-pants.

You can select a boot command by setting the SCIE_BOOT environment variable.

Pants version 2.19 - 2.22

OS Linux:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
PRETTY_NAME="Ubuntu 22.04.3 LTS"

Additional info We've tried a bit of flailing to fix this:

• Attempting different Pants versions.
• ./get-pants.sh
• SCIE_BOOT=update pants (yields No new releases of scie-pants were found.)

We tried to install Pex directly into the virtual environment using pip, but got

raise MetadataError(\npex.dist_metadata.MetadataError: Failed to determine project name and version for distribution at /bulk_data/home/<REDACTED>/.pex/unzipped_pexes/ba7<REDACTED>/.deps/PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.\n'

FWIW, https://github.com/pantsbuild/pants/releases/download/release_2.22.0/pants.2.22.0-cp39-linux_x86_64.pex is not an accessible address as far as I can tell.

[cburroughs]

The second error is because 2.22 has not been released yet, that is https://github.com/pantsbuild/pants/releases/tag/release_2.22.0 will also 404. (There have been several RCs and we hope it is really close, but it is not out yet!)

I am unsure about the first error. When you say "all of our environments", do you mean that it happens in a variety of environments ( every developers workstation) or in a standardized environment (that is on with your/bulk_data` mount).

I hesitate to recommend this, but is deleting the cache (.cache/nce) among the things you have tried?

[originalrkk]

Ah, yes, that might have been a miscommunication. Forget about 2.22...

The main thing is: Tried tearing down .cache/nce, actually all of .cache and .pex and /tmp/*pants*, the whole box, but no go. 2.19.0 installs fine, 2.20 and 2.21 fail, always with the same ModuleNotFoundError.

Just to be sure it wasn't some quirk of config in our repo, we even cleared everything out including any Pants binary and did the following:

curl --proto '=https' --tlsv1.2 -fsSL https://static.pantsbuild.org/setup/get-pants.sh | bash
mkdir test-pants
cd test-pants
echo '[GLOBAL]' > pants.toml
echo 'pants_version = "2.21.0"' >> pants.toml
pants repl .

And still see the same error on these boxes.

Any ideas how we could get this to spit out some more useful logs about what's happening perhaps?

[cburroughs]

Can you conform what version of the scie-pants bootloader this box has?

$ PANTS_BOOTSTRAP_VERSION=report pants

Any ideas how we could get this to spit out some more useful logs about what's happening perhaps?

I am not sure they would have much more that the earlier output, but you can try what is in find ~/.cache/nce/ -iname '*log' (ex: install.log)

[originalrkk]

This is what we see for the bootloader:

$ PANTS_BOOTSTRAP_VERSION=report pants
0.12.0

Only an install.log (no configure.log or pants-install.log on this box like we see on a healthy system), and it just reiterates what's in the dump.

[originalrkk]

Okay, a bit more context: We were able to reproduce this on a completely clean Ubuntu image on AWS with a new Pants repo: ami-0d486650b94f4c69b. Not sure whether that's pointing to some networking configuration around it (perhaps it's failing to fetch something silently?).

[originalrkk]

We found the source of the issue. The Pants installer isn't correctly handling a symlinked home directory. In particular, in /etc/passwd, the home directory was listed as a symlink /home/<user> instead of /bulk_data/home/<user>.

[cburroughs]

That looks frustratingly deep to debug; glad you found it!