socket.getfqdn() doesn't cope properly with purely DNS-based setups

[53553675-8906-4186-8b7e-3bb3c40d5d7f]

BPO	5004
Nosy	@loewis, @tiran, @mcepl, @bitdancer, @thomaswaldmann, @jan-hudec, @shoop
Files	python2.7-socket-getfqdn.patch python5004-test.c: test C program showing various gethost function output on my system python2.7-socket-getfqdn.patch: Updated patch, originally by Stijn.Hoop

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', 'library', '3.9', '3.10', '3.11'] title = "socket.getfqdn() doesn't cope properly with purely DNS-based setups" updated_at = user = 'https://bugs.python.org/dfranke' ``` bugs.python.org fields: ```python activity = actor = 'mcepl' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'dfranke' dependencies = [] files = ['29919', '29921', '50390'] hgrepos = [] issue_num = 5004 keywords = ['patch'] message_count = 24.0 messages = ['80216', '109635', '109647', '162508', '173987', '187065', '187098', '187234', '187237', '187238', '187253', '187256', '187350', '187355', '297435', '300593', '308979', '372583', '404775', '404780', '404796', '404806', '404826', '405364'] nosy_count = 14.0 nosy_names = ['loewis', 'christian.heimes', 'mcepl', 'r.david.murray', 'dfranke', 'devurandom', 'mcjeff', 'ankitoshniwal', 'Thomas.Waldmann', 'Stijn.Hoop', 'bulb', 'James Shewey', 'richard.security.consultant', 'shoop'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue5004' versions = ['Python 3.9', 'Python 3.10', 'Python 3.11'] ```

[53553675-8906-4186-8b7e-3bb3c40d5d7f]

On Linux and presumably on other POSIX-like systems, socket.getfqdn() doesn't work if a system resolves its own FQDN using DNS rather than /etc/hosts.

My system's FQDN is 'fugue.tank.wellohorld.com'. My /etc/hosts is empty except for loopback entries, and /etc/resolv.conf contains the line 'domain tank.wellohorld.com'. This is sufficient for 'hostname -f' to do the Right Thing, but socket.getfqdn() simply returns 'fugue':

dfranke@fugue:~/Python-2.6.1$ hostname
fugue
dfranke@fugue:~/Python-2.6.1$ hostname -f
fugue.tank.wellohorld.com
dfranke@fugue:~/Python-2.6.1$ ./python
Python 2.6.1 (r261:67515, Jan 19 2009, 13:56:59)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'fugue'
>>>
dfranke@fugue:~/Python-2.6.1$ echo -e '$a\n172.17.0.120
fugue.tank.wellohorld.com fugue\n.\nwq' | sudo ed /etc/hosts
305
350
dfranke@fugue:~/Python-2.6.1$ ./python
Python 2.6.1 (r261:67515, Jan 19 2009, 13:56:59)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'fugue.tank.wellohorld.com'
>>>
dfranke@fugue:~/Python-2.6.1$

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

Would someone with appropriate knowledge please take a look to see if this is still an issue.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

I think anybody willing to invest the time could acquire the appropriate knowledge, at least to determine whether it's still an issue (i.e. trying to reproduce it). To fix it, one would then need to read the source code of hostname, and find out what they do differently; strace might be sufficient already.

[1ebc1928-cbb7-41eb-8605-4924cc9468fe]

I cannot reproduce this issue. I just tested this on my mac.

atoshniw@prusev-mn:~/Documents/code/python-dev/bin #hostname -f
prusev-mn.helloworld.com
atoshniw@prusev-mn:~/Documents/code/python-dev/bin #python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'prusev-mn.helloworld.com'

[8c8022e7-f081-4d49-be03-926cedb53a7a]

Gave this a go myself...

$ ./python
Python 3.4.0a0 (default:57a33af85407, Oct 27 2012, 21:26:30) 
[GCC 4.4.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'host.domain.com'
>>> 

$ hostname -f
host.domain.com

$ cat /etc/hosts
127.0.0.1       localhost.localdomain   localhost

# The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters

Linux host 3.5.2-x86_64 #1 SMP Wed Aug 15 14:31:07 EDT 2012 x86_64 GNU/Linux

According to strace, both rely on DNS:

recvfrom(3, "Wj\201\200\0\1\0\1\0\5\0\0\00219\003134\003230\003173\7in-a"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.2.3.4")}, [16]) = 176

Same behavior on both 2.6 & hg tip. I think this is a non-issue.

[873da39a-4b88-4a57-9e7b-1e2c6db7beba]

Still seeing this on Fedora 18 / Python 2.7.3.

I only have loopback in /etc/hosts

[TUE\shoop@pclin281] \<~> cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

I search in .campus.tue.nl and .win.tue.nl:

[TUE\shoop@pclin281] \<~> grep search /etc/resolv.conf search campus.tue.nl. win.tue.nl.

Hostname -f reliably returns .campus.tue.nl as it should

[TUE\shoop@pclin281] \<~> hostname -f pclin281.campus.tue.nl [TUE\shoop@pclin281] \<~> hostname -f
pclin281.campus.tue.nl

But socket.getfqdn disagrees, even with itself when run multiple times:

[TUE\shoop@pclin281] <~> python
Python 2.7.3 (default, Aug  9 2012, 17:23:57) 
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'pclin281'
>>> socket.getfqdn()
'pclin281.win.tue.nl'
>>> socket.getfqdn()
'pclin281'
>>> socket.getfqdn()
'pclin281.win.tue.nl'
>>> 

Note that pclin281.win.tue.nl is in fact also a valid DNS entry, but not one that I would expect the function to ever return given the search order.

[bitdancer]

Note that socket.getfqdn is a wrapper around a couple of socket calls that are just wrappers of OS level socket calls. If you take a look at socket.py you'll see the definition. As Martin said earlier, if you (or anyone else) can figure out what hostname does differently and suggest how to patch our getfqdn method to behave similarly, I'm sure the patch will be welcome.

Unfortunately there won't be any good way to write a test for this.

[873da39a-4b88-4a57-9e7b-1e2c6db7beba]

OK, fair enough.

From reading sources, it appears that hostname is using getaddrinfo(3) on kernelhostname with hints->ai_flags & AI_CANONNAME, while Lib/socket.py simply uses gethostbyaddr(kernelhostname), and falls back on kernelhostname in case of errors.

Unfortunately I am not entirely sure who is "correct" here, as I don't know the intent of socket.getfqdn().

In my case, kernelhostname is set to 'pclin281' e.g. without the dots. I believe this to be correct, but I know that this is already "controversial" as in there exists software that expects an FQDN there, and internet folklore is split about 50/50 about this necessity.

Then, apparently, there is confusion about AI_CANONNAME and what it actually should do. glibc upstream does address lookups but Fedora patches this out. See this recent glibc bug for more pointers:

http://sourceware.org/bugzilla/show_bug.cgi?id=15218

As mentioned in that bug, a lot of software runs on Fedora and works using that definition of AI_CANONNAME.

However, switching Lib/socket.py / getfqdn from gethostbyaddr to getaddrinfo might have more implications than just fixing this case. I can try to write a patch, but is this the right direction?

[873da39a-4b88-4a57-9e7b-1e2c6db7beba]

Attached is a very lightly tested patch that matches hostname -f behaviour on my system. I suspect this should be OK but it definitely needs more testing than just my system...

[bitdancer]

The problem with your patch is that it changes the (effective) meaning of the 'name' parameter. Before the patch, name can be an IP address. After the patch, that will fail on Fedora. (It also fails on my Gentoo system).

It is interesting to note, as well, that the documentation for gethostbyaddr says that it is obsolete and getaddrinfo should be used instead.

Could we use the getaddrinfo call if we don't get an FQDN back from gethostbyaddr? It doesn't look like that would completely solve your problem, though, given your example output. Have you figured out why that is happening?

Alternatively, perhaps we could fall back to gethostbyaddr if we don't get an fqdn from the getaddrinfo call.

However, given that the documentation actually specifies the algorithm used by getfqdn, I'm not sure if we can make either change in a bugfix version.

[873da39a-4b88-4a57-9e7b-1e2c6db7beba]

OK, dumping my current findings here, as I'm still not sure what the expected results should be.

First of all, Lib/socket.py calls gethostbyaddr with a name. As in, gethostby _ADDR_ with a name.

This works because Modules/socketmodule.c internally uses setipaddr() to resolve the name to an address. setipaddr() does this using a call to getaddrinfo() with hints.ai_family == AF_UNSPEC and no further flags.

On my system (confirmed using the test program attached) this results in SIX entries, and this is the part that confused me.

Due to virtualization I have a virtual bridge virbr0 configured with an internal IP address 192.168.122.1, as well as my LAN-connected bridge br0 with IP address 131.155.71.8. Both of these addresses are returned in the call to getaddrinfo() (each one 3 times), but NOT ALWAYS IN THE SAME ORDER.

And this is the clue as to why python's socket.getfqdn() does not behave consistently. For 192.168.122.1 does not resolve to anything, hence it will return "pclin281". And 131.155.71.8 will backwards resolve to pclin281.win.tue.nl as the PTR record points to that entry.

Now, again, I'm not entirely sure what to do here. I agree that this is not a simple bugfix. I also think that, apart from the weirdness of getaddrinfo() return order, socket.getfqdn() is doing it's documented job of returning /an/ FQDN for a given host.

But in case of the guaranteed LOCAL canonical hostname, another function is warranted, imho.

Does this make sense?

For the record, output of a given run on my system:

[TUE\shoop@pclin281] \<~/tmp> ./test gai canon result 0: pclin281.campus.tue.nl 192.168.122.1 gai canon result 1: (null) 131.155.71.8 gai result 0: (null) 131.155.71.8 gai result 1: (null) 131.155.71.8 gai result 2: (null) 131.155.71.8 gai result 3: (null) 192.168.122.1 gai result 4: (null) 192.168.122.1 gai result 5: (null) 192.168.122.1 ghbn result 0 h_name: pclin281.campus.tue.nl ghbn result 0 halias: \_NONE ghbn result 1 h_name: pclin281.campus.tue.nl ghbn result 1 halias: \_NONE ghbn result 2 h_name: pclin281.campus.tue.nl ghbn result 2 halias: \_NONE ghbn result 3 h_name: pclin281.campus.tue.nl ghbn result 3 halias: \_NONE ghbn result 4 h_name: pclin281.campus.tue.nl ghbn result 4 halias: \_NONE ghbn result 5 h_name: pclin281.campus.tue.nl ghbn result 5 halias: \_NONE ghbn result 6 h_name: pclin281.campus.tue.nl ghbn result 6 halias: \_NONE ghbn result 7 h_name: pclin281.campus.tue.nl ghbn result 7 halias: \_NONE ghbn result 8 h_name: pclin281.campus.tue.nl ghbn result 8 halias: \_NONE ghbn result 9 h_name: pclin281.campus.tue.nl ghbn result 9 halias: \_NONE

[bitdancer]

Yeah, a new function was a thought that had crossed my mind as well. getfqdnbyname, maybe? Or gethostnamefqdn? Then deprecate calling getfqdn without an argument.

I agree that gethostbyaddr accepting a non-IP is weird. I have no idea why it was implemented that way, much less why it is *used* that way. It's been that way for a long time, though.

[873da39a-4b88-4a57-9e7b-1e2c6db7beba]

So after a good nights sleep: does it not make sense to use the canonical hostname iff the name argument is not present / empty? Otherwise, fall back to the documented steps? That way extra API is avoided, and I can't think of a case where you would rather have my weird results vs "the output of hostname -f".

[bitdancer]

That is an interesting proposal, yes. I suppose someone that needs the getaddrinfo semantics for something other than the local host can just call it directly.

Now, do we add the fact that we are doing this to the current alogarithmic documention? :)

[391fc774-70cf-42f2-b96f-42d5ae950a7d]

According to the man page for gethostbyaddr "The gethostbyname() and gethostbyaddr() functions are obsolete. Applications should use getaddrinfo(3) and getnameinfo(3) instead." - so perhaps using the correct API call might be a good start to resolving this issue, but I found that in my case, I needed to chase the problem upstream instead of downstream. On my Red Hat box, the kernel.hostname value with sysctl was incorrect. I had to re-set it with a sysctl kernel.hostname=hostname.example.com. This overrides /etc/hosts, so I suspect this is probably not an issue on other distros that do not use sysctl.

The moral of the story being garbage in, garbage out.

[1562ced0-0aea-4dbd-a287-b8a12bfca88a]

In my case, /etc/hostname, /proc/sys/kernel/hostname, uname -n, hostname -f all show the same FQDN, but python -c 'import socket ; print(socket.getfqdn())' still prints the short hostname. /etc/hosts is empty except for localhost. /etc/nsswitch.conf contains: hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname

[73104530-ea09-48eb-8e89-fbef1c14f7e7]

Embarassing as always to stumble over some stuff and then find a 9y old ticket here, where it is discussed and even (almost) solved.

Our ticket: https://github.com/borgbackup/borg/issues/3471

Fixed getfqdn we use now instead of socket.getfqdn():

https://github.com/borgbackup/borg/pull/3484/commits/9b0d0f3127289fc3bbe620e8adce8038ed594e9f#diff-4b53f84e19a3bb376bf2202371ed269aR188

Note: no "else: name = hostname" at the end, that is a bug in the patch attached to this ticket (hostname is undefined after applying the patch).

[9e1628db-f7d6-4574-b28b-2e6e8284f5a4]

Confirming the fixed version linked in previous comment by Thomas Waldmann is correct and matches what hostname -f does.

[440611a1-b6fc-46e9-a123-2bc6a47e067f]

I just ran into this 12 year old issue. Can this be merged please?

[tiran]

Could you or somebody else please create a PR with patch and a test case?

[440611a1-b6fc-46e9-a123-2bc6a47e067f]

Here is the updated patch. Is python5004-test.c enough as a test case?

[tiran]

We no longer accept patches. Contributors have to create a PR on GitHub, so we can record contributions and verify the contributor license agreement.

[440611a1-b6fc-46e9-a123-2bc6a47e067f]

In that case Stijn Hope should create the PR since he wrote the patch. Anyone else could get in trouble for using his code without proper permission.

[d5e308dd-4a3a-4332-8ef2-a55ec9cea928]

I hereby put my patch in the public domain and/or under any desired copyright license as required by the Python project to accept it.

Regards,

Stijn Hoop

On Fri, 22 Oct 2021 21:03:26 +0000 Richard van den Berg \report@bugs.python.org\ wrote:

Richard van den Berg \richard.security.consultant@gmail.com\ added the comment:

In that case Stijn Hope should create the PR since he wrote the patch. Anyone else could get in trouble for using his code without proper permission.

----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue5004\

---