frostming / unearth

A utility to fetch and download python packages
https://unearth.readthedocs.io
MIT License
142 stars 18 forks source link

Ignoring .netrc errors leads to silent failure #124

Closed lambda closed 4 months ago

lambda commented 5 months ago

Describe the bug In #121 failures to parse .netrc are skipped. But this leads to hard to diagnose failures when you are using .netrc for authentication

Additionally, when it falls back, it falls back to KDE wallet for some reason, despite the fact that I'm on Gnome and my main keyring is Gnome Keyring.

To Reproduce Steps to reproduce the behavior:

  1. Create a .netrc file in your home directory on Linux or macOS, with authentication credentials for a local PyPI index
  2. Save it with the default permissions, which are generally 0644 (or chmod 0644 ~/.netrc if not)
  3. Set up a pyproject.toml that uses tool.pdm.source with the URL for your local PyPI index, with name = "pypi"
  4. Run pdm install

Expected behavior I expect it to either use my .netrc, or give me an error why it can't be used. This used to work fine with the bad permissions (which are only really an issue on shared systems, which most developer systems are not). Newer versions of the netrc module seem to throw errors for the wrong permissions, but as long as they throw an error and I see it, I can fix it.

However, when updating my PDM environment with unearth 0.15.4, I simply get a silent failure to parse .netrc, and then it falls back to trying to use KDE Wallet, and if I cancel out of that it tries to prompt for auth credentials in the terminal. This is very confusing, and I had to downgrade unearth to actually get the error message to figure out why this was failing.

Screenshots If applicable, add screenshots to help explain your problem.

System (please complete the following information):

Additional context Add any other context about the problem here.

frostming commented 5 months ago

This is the original issue: https://github.com/pdm-project/pdm/issues/2916

I don't accept the solution to add a switch for this. pip/requests ignores permission errors so i think it's okay to keep the same

lambda commented 5 months ago

I used PDM because it's better than pip. Providing a useful error message is better than ignoring the file. The single hardest part of getting set up in our environment is setting up authentication to our internal package registry, in part because of how particular it is about how the .netrc file is formatted (I use .netrc because it works across all platforms, otherwise I'd have to support a whole host of different ways to configure keychains on different platforms), but it makes it much harder to debug errors with .netrc if we never see them.

frostming commented 5 months ago

/cc @ZacEllis if you have any comment on this.

iburakov commented 4 months ago

+1 for at least a debug-level log message when something goes wrong with loading creds from .netrc. It could save me a couple of hours today.

I understand why raising might not be acceptable. Maybe, a verbose log message would be a great solution that will be consistent with pip/requests, non-breaking and still very helpful when it comes to debugging .netrc-related now-silent failures.


Just for context: I gave known-to-be-valid creds in .netrc for pdm sync to use in GitLab CI, but it kept ignoring them as if they didn't exist. Enabling -vv output didn't help – exactly this is what could be improved here, imho. It looked quite meaningless, especially for maximum verbosity:

<...>
unearth.auth: Found index url <private-pypi-url>
User for <private-pypi-netloc>: Traceback (most recent call last):
<...>
    username = input(f"User for {netloc}: ")
EOFError: EOF when reading a line

and took, being not aware of pdm implementation details, a solid 2+ hours to untangle. Critical missing piece of the puzzle was simply silenced here.

The actual error could be logged like this:

<...>
unearth.auth: Found index url <private-pypi-url>
unearth.utils: Couldn't get credentials from netrc because of NetrcParseError: ~/.netrc access too permissive: access permissions must restrict access to only the owner (/root/.netrc, line 1)
User for <private-pypi-netloc>: Traceback (most recent call last):
<...>
    username = input(f"User for {netloc}: ")
EOFError: EOF when reading a line

As you see, many things can go unexpected here, so such a log entry would be a huge time-saver, especially for new users not aware of pdm/unearth implementation details.

As a workaround, for the time being, I had to, in addition to finding this all out, manually call get_netrc_auth-like-code in the same CI environment to get my traceback and the error above. Posting it in case it's helpful for someone else.

python -c "import os; from netrc import netrc; netrc(os.getenv('NETRC')).authenticators('<private-pypi-netloc>')"

On a general note, thanks for great tools! Keep up the good work.