netrc module does not handle multiple entries for a single host

[e70c471c-1989-4988-938b-e8b69b72d197]

BPO	11416
Nosy	@rhettinger, @merwok, @bitdancer, @briancurtin, @berkerpeksag, @arsyee
PRs	python/cpython#17823
Files	netrc.patch netrc-doc.patch issue11416.diff netrc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['3.7', '3.8', 'type-feature', 'library'] title = 'netrc module does not handle multiple entries for a single host' updated_at = user = 'https://bugs.python.org/saffroy' ``` bugs.python.org fields: ```python activity = actor = 'Frew Schmidt' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'saffroy' dependencies = [] files = ['22195', '22204', '34485', '42510'] hgrepos = [] issue_num = 11416 keywords = ['patch', 'needs review'] message_count = 27.0 messages = ['130193', '130618', '130686', '130689', '132421', '132422', '132434', '137218', '137267', '137276', '137277', '137278', '137279', '137280', '137282', '137283', '137288', '137289', '137341', '137449', '138071', '213229', '213247', '213955', '263699', '358788', '359297'] nosy_count = 9.0 nosy_names = ['rhettinger', 'eric.araujo', 'r.david.murray', 'brian.curtin', 'saffroy', 'berker.peksag', 'Yuri.Bochkarev', 'luc', 'arsyee'] pr_nums = ['17823'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue11416' versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8'] ```

[e70c471c-1989-4988-938b-e8b69b72d197]

I have a netrc file with two entries for the same host. The netrc module only returns the last entry.

$ cat > .netrc
machine host.com
        login foo
        password foo

machine host.com login bar password bar

$ python -c 'import netrc; print netrc.netrc()'
machine host.com
        login 'bar'
        password 'bar'

My Linux ftp clients (ftp, lftp) always use the first entry for a given host. Also lftp can use the password from the proper entry if I supply the host and login.

With the netrc module in Python 2.6.6 (as tested on Debian Squeeze), I can only retrieve the last entry.

[merwok]

Thanks for the report. Would you like to work on a patch? Guidelines are found at http://docs.python.org/devguide

[e70c471c-1989-4988-938b-e8b69b72d197]

I could spend some time on a small patch.

Here are the changes I would consider:

• netrc.authenticators() could return the first entry matching a given host (now it returns the last)
• it could also accept an extra parameter (with default=None), a login to select the right account/password for a given host
• it could also return a list of entries (tuples) matching the supplied host (now it always returns only one tuple), or a new method could do that

IMHO the last change would make the overall behaviour more consistent (BTW note the plural for "authenticators"), but it could also break existing apps.

Comments?

[bitdancer]

I think whatever change we make it should be backward compatible, even though that is a bit annoying (especially given the current method name). So either a parameter that selects the new behavior and defaults to the old, or a new method that returns a list, IMO.

[e70c471c-1989-4988-938b-e8b69b72d197]

So I finally cooked a little patch for netrc.py in python 2.6. The patch extends netrc.authenticators() with an extra parameter to select a login name, but otherwise the behaviour remains the same (still returns the last entry for a given host).

Lightly tested, works for me.

[briancurtin]

Can you add your tests to Lib/test/test_netrc.py?

[e70c471c-1989-4988-938b-e8b69b72d197]

Good that you mentioned the official tests, they let me see that netrc.hosts is actually part of the API, and my first patch broke it.

Here is an updated patch, with extra tests.

[e70c471c-1989-4988-938b-e8b69b72d197]

Ping? A patch is available for review.

[merwok]

Looks good. One remark: instead of using keys() + getitem in Lib/netrc.py:109, you can use this: for host, attrs in self.allhosts.items()

[e70c471c-1989-4988-938b-e8b69b72d197]

You are suggesting something like this, I suppose?

--- a/Lib/netrc.py
+++ b/Lib/netrc.py
@@ -105,8 +105,8 @@ class netrc:
     def __repr__(self):
         """Dump the class data in the format of a .netrc file."""
         rep = ""
-        for host in self.allhosts.keys():
-            for attrs in self.allhosts[host]:
+        for (host, attrlist) in self.allhosts.items():
+            for attrs in attrlist:
                 rep = rep + "machine "+ host + "\n\tlogin " + repr(attrs[0]) + "\n"
                 if attrs[1]:
                     rep = rep + "account " + repr(attrs[1])

[merwok]

Exactly, minus the parens (not needed).

[e70c471c-1989-4988-938b-e8b69b72d197]

Patch slightly updated after Eric's comments.

[merwok]

I’m surprised self.assert_ does not produce a DeprecationWarning (in favor of assertTrue). The tests should use assertEqual(expected, computed) anyway, to let developers see a useful diff when the test fails. Could you change this in the tests?

[e70c471c-1989-4988-938b-e8b69b72d197]

Patch formatting changed to be more review-friendly (looks like MQ-style patch isn't?), otherwise same as 2011-05-30 16:14.

[e70c471c-1989-4988-938b-e8b69b72d197]

Eric: yes I can look into the asserts, but note I generated and tested my patch from a checkout of 2.6 (see my first report), so maybe that's why I didn't see any warning.

[merwok]

Ah, okay. Your patch needs to apply to 2.7 or 3.2, which use assertEqual and not assert_.

[e70c471c-1989-4988-938b-e8b69b72d197]

Here is a patch against 2.7.

[merwok]

Great! Could you expand the docstrings and reST documentation to mention the new behavior?

[e70c471c-1989-4988-938b-e8b69b72d197]

Additional patch for docstrings and documentation. Applies on top of previous patch.

[merwok]

Looks good, thanks.

[bitdancer]

This should not be applied until after the patch for bpo-12009 is applied, at which point the test will need to be adjusted. bpo-12009 completely refactors the netrc test suite.

I wonder if the 'hosts' attribute should be deprecated, but let's leave that discussion for another day :)

Since this requires an API change, it is a feature and can only go into 3.3. I've adjusted 'versions' accordingly.

[ffe14577-f712-4e8f-a5e2-7e999d8ff51a]

What is the status of this issue?

On version 3.3 I still can not get more than one entry per host. The stopping issue bpo-12009 seems to be closed.

(Sorry, if I violate some etiquette by bumping this, just tell me.)

[merwok]

This patch hasn’t been committed. The comment before yours says that it needs to be edited a bit.

[berkerpeksag]

I've updated the tests to match the changes in bpo-12009 and documentation a bit.

[d457d479-8fd8-4931-944b-73fd0f2524ae]

This issue is causing a problem in OfflineIMAP. I wrote a patch that I think is a little simpler than the existing one, including tests and docs, though honestly I don't care one way or another. See attached. Note that the patch is for Python 2.7 but it mostly applies to 3.5 just fine. Any chance we could get mine or the other one applied?

Thanks

[a73dbb73-bbdc-4987-bf33-a2d78783a83d]

As I understand this issue is there up to 3.8 (and will live into 3.9). As urllib3 supports netrc and requests uses urllib3, this issue can potentially cause a lot of inconsistencies. In case of multile entries (with no user provided), curl and wget takes the first entry, while cpython netrc takes the last. I see there is a proposed implementation, but I couldn't find it's pull request on github. Is there any chance this will be resolved in the near future? Alternatively, I can create a pull request, but I only bother if there is an actual intention to fix the current behavior.

[a73dbb73-bbdc-4987-bf33-a2d78783a83d]

Just for further clarification on how .netrc is supposed to work, I tested some other implementations. Sorry if this comment gets too long, but it seems netrc is not really a standardized solution.

In my last comment I referred curl and wget, but of course the original implementation is ftp (which points to the netrc manual). The manual itself doesn't talk about priorities, but all the mentioned 3 tools are in common in that they take the first entry if the same machine is in the netrc file multiple times.

When trying to force using the second entry, though, breaks this consistency quickly: none of the above tools seem to support a query with host and login name provided. ftp: doesn't support providing a user name from shell (and doesn't use netrc when opening a connection interactively) curl: if username is provided, it interactively asks for a password and uses that (which may be an issue on my end, as source code suggested there is a login-based lookup) wget: if username is provided without a password, it doesn't send a auth information at all (also wget doesn't send default auth information for http, which behavior can be altered somehow)

Actually these results suprised me, as I remembered I used host+login based lookup previously. And the answer was git clone via https: git's implementation is exactly what I would expect:

• if only host is provided, the first entry is used
• if both host and login is provided, it looks up the password for the appropriate login
• if no matching machine entry can be found, it uses the default entry
• it also supports login matching for the default entry

I'm about the send a pull request with my understanding on how netrc should work. Of course all comments are welcome, and I'm happy to alter the implementation (or break it up to smaller pieces if necessary). My priorities were:

• Not to break backward compatibility.
• Provide a way to consistently pick up the first entry, without the need to make modifications in all of the dependent libraries (see doc change).
• __repr__() had a bug: for the default entry, it gives 'machine default', instead of 'default'.
• another bug: 'machine default' is actually not supported, as the netrc.hosts dictionary cannot make a difference between the default entry and a machine called default (this has a partial fixed)

As r.david.murray mentioned, the hosts dictionary should probably be deprecated.