nmap fails to obtain mac address in 2023.2.x

[scottsweb]

The problem

The nmap integration is failing to detect devices in Home Assistant 2023.2.x. I have tested all versions beyond 2023.1.7 and the issue is present.

I am running Home Assistant in a container with rootless podman. My compose file looks like this:

  homeassistant:
    image: homeassistant/home-assistant:2023.1.7 #stable
    container_name: homeassistant
    network_mode: host
    privileged: true
    restart: always
    group_add:
      - dialout
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "./data/homeassistant:/config:z"
      - "/run/user/1000/podman/podman.sock:/var/run/docker.sock:z"

And my nmap configuration is as follows:

-F -T4 --min-rate 10 --host-timeout 5s --unprivileged

With a scan interval of 20 and 450 second wait until a device is home.

What version of Home Assistant Core has the issue?

2023.2.2

What was the last working version of Home Assistant Core?

2023.1.7

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Nmap Tracker

Link to integration documentation on our website

https://www.home-assistant.io/integrations/nmap_tracker/

Diagnostics information

The only nmap failure in the logs is:

2023-02-07 11:30:46.166 INFO (MainThread) [homeassistant.components.nmap_tracker] No MAC address found for 192.168.1.200

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2023-02-07 10:58:52.975 INFO (MainThread) [homeassistant.components.nmap_tracker] No MAC address found for 192.168.1.200

New versions complain that the MAC address cannot be found.

On older versions, getting inside the container and running:

nmap 192.168.1.200 -F -T4 --min-rate 10 --host-timeout 5s --unprivileged --exclude 192.168.1.4 --reason -v

Gives me:

Starting Nmap 7.92 ( https://nmap.org ) at 2023-02-07 11:24 CET
Initiating Ping Scan at 11:24
Scanning 192.168.1.200 [2 ports]
Completed Ping Scan at 11:24, 0.23s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 11:24
Completed Parallel DNS resolution of 1 host. at 11:24, 0.00s elapsed
Initiating Connect Scan at 11:24
Scanning 192.168.1.200 [100 ports]
Completed Connect Scan at 11:24, 0.03s elapsed (100 total ports)
Nmap scan report for 192.168.1.200
Host is up, received conn-refused (0.0096s latency).
All 100 scanned ports on 192.168.1.200 are in ignored states.
Not shown: 100 closed tcp ports (conn-refused)

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.35 seconds

On the newer version the result is:

Starting Nmap 7.92 ( https://nmap.org ) at 2023-02-07 11:29 CET
Initiating Ping Scan at 11:29
Scanning 192.168.1.200 [2 ports]
Completed Ping Scan at 11:29, 0.17s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 11:29
Completed Parallel DNS resolution of 1 host. at 11:29, 0.00s elapsed
Initiating Connect Scan at 11:29
Scanning 192.168.1.200 [100 ports]
Completed Connect Scan at 11:29, 0.03s elapsed (100 total ports)
Nmap scan report for 192.168.1.200
Host is up, received conn-refused (0.011s latency).
All 100 scanned ports on 192.168.1.200 are in ignored states.
Not shown: 100 closed tcp ports (conn-refused)

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.31 seconds

I don't see any immediate difference here and given its the same version of nmap, it seems strange that something breaks.

Additional information

The only thing I can think of at this stage is that home assistant cannot be run rootless any longer, but I cannot see anything in recent change logs that may have impacted this.

[home-assistant[bot]]

nmap_tracker documentation nmap_tracker source

[Zasari]

Same for me, looking at the used python package getmac - Example from an IP that is 'up' for me:

#> python -m getmac -v -dddd --ip 192.168.200.20

DEBUG    Initializing 'ip4' method cache (platform: 'linux')
DEBUG    27 methods available: CtypesHost, ArpingHost, ArpFile, SysIfaceFile, FcntlIface, UuidArpGetNode, UuidLanscan, GetmacExe, IpconfigExe, WmicExe, ArpExe, DarwinNetworksetupIface, ArpFreebsd, ArpOpenbsd, IfconfigWithIfaceArg, IfconfigEther, IfconfigOther, IpLinkIface, NetstatIface, IpNeighborShow, ArpVariousArgs, DefaultIfaceLinuxRouteFile, DefaultIfaceIpRoute, DefaultIfaceRouteCommand, DefaultIfaceRouteGetCommand, DefaultIfaceOpenBsd, DefaultIfaceFreeBsd
DEBUG    9 type-filtered methods for 'ip4': CtypesHost, ArpingHost, ArpFile, UuidArpGetNode, ArpExe, ArpFreebsd, ArpOpenbsd, IpNeighborShow, ArpVariousArgs
DEBUG    5 platform-filtered methods for 'linux' (method_type='ip4'): ArpingHost, ArpFile, UuidArpGetNode, IpNeighborShow, ArpVariousArgs
DEBUG    5 tested methods for 'ip4': ArpingHost, ArpFile, UuidArpGetNode, IpNeighborShow, ArpVariousArgs
DEBUG    Current method cache: {'ip4': 'ArpingHost', 'ip6': 'None', 'iface': 'None', 'default_iface': 'None'}
DEBUG    Current fallback cache: {'ip4': '[<getmac.getmac.ArpFile object at 0x7ff76490e9e0>, <getmac.getmac.UuidArpGetNode object at 0x7ff76490eb60>, <getmac.getmac.IpNeighborShow object at 0x7ff76490ea70>, <getmac.getmac.ArpVariousArgs object at 0x7ff76490eb00>]', 'ip6': '[]', 'iface': '[]', 'default_iface': '[]'}
DEBUG    Finished initializing 'ip4' method cache
DEBUG    Not sending UDP packet, using network request method 'ArpingHost' instead
DEBUG    Attempting get() (method='ArpingHost', method_type='ip4', arg='192.168.200.20')
DEBUG    Running: '/usr/sbin/arping --ridiculous-garbage-string'
DEBUG    Running: '/usr/sbin/arping -r -C 1 -c 1 192.168.200.20'
DEBUG    Method 'ArpingHost' failed for 'ip4' lookup
DEBUG    Raw MAC found: None
DEBUG    getmac took 0.0256 seconds

It seems to work, when setting the platform to other:

# python -m getmac -v -dddd --ip 192.168.200.20 --override-platform other
DEBUG    Initializing 'ip4' method cache (platform: 'linux')
WARNING  Platform override is set, using 'other' as platform instead of detected platform 'linux'
DEBUG    27 methods available: CtypesHost, ArpingHost, ArpFile, SysIfaceFile, FcntlIface, UuidArpGetNode, UuidLanscan, GetmacExe, IpconfigExe, WmicExe, ArpExe, DarwinNetworksetupIface, ArpFreebsd, ArpOpenbsd, IfconfigWithIfaceArg, IfconfigEther, IfconfigOther, IpLinkIface, NetstatIface, IpNeighborShow, ArpVariousArgs, DefaultIfaceLinuxRouteFile, DefaultIfaceIpRoute, DefaultIfaceRouteCommand, DefaultIfaceRouteGetCommand, DefaultIfaceOpenBsd, DefaultIfaceFreeBsd
DEBUG    9 type-filtered methods for 'ip4': CtypesHost, ArpingHost, ArpFile, UuidArpGetNode, ArpExe, ArpFreebsd, ArpOpenbsd, IpNeighborShow, ArpVariousArgs
DEBUG    3 platform-filtered methods for 'other' (method_type='ip4'): UuidArpGetNode, IpNeighborShow, ArpVariousArgs
DEBUG    3 tested methods for 'ip4': UuidArpGetNode, IpNeighborShow, ArpVariousArgs
DEBUG    Current method cache: {'ip4': 'UuidArpGetNode', 'ip6': 'None', 'iface': 'None', 'default_iface': 'None'}
DEBUG    Current fallback cache: {'ip4': '[<getmac.getmac.IpNeighborShow object at 0x7fb581d469b0>, <getmac.getmac.ArpVariousArgs object at 0x7fb581d469e0>]', 'ip6': '[]', 'iface': '[]', 'default_iface': '[]'}
DEBUG    Finished initializing 'ip4' method cache
DEBUG    Attempting to populate ARP table with UDP packet to 192.168.200.20:55555
DEBUG    Attempting get() (method='UuidArpGetNode', method_type='ip4', arg='192.168.200.20')
DEBUG    Raw MAC found: F0:2F:7B:19:8F:71
DEBUG    getmac took 0.0092 seconds
f0:2f:7b:19:8f:71

Not sure, which method was used/working in the past, but the method ArpingHost is not using a compatible 'arping' binary, as the one from busybox is not supporting those options, just:

#> arping
BusyBox v1.35.0 (2022-08-01 15:14:44 UTC) multi-call binary.

Usage: arping [-fqbDUA] [-c CNT] [-w TIMEOUT] [-I IFACE] [-s SRC_IP] DST_IP

Send ARP requests/replies

    -f      Quit on first ARP reply
    -q      Quiet
    -b      Keep broadcasting, don't go unicast
    -D      Exit with 1 if DST_IP replies
    -U      Unsolicited ARP mode, update your neighbors
    -A      ARP answer mode, update your neighbors
    -c N        Stop after sending N ARP requests
    -w TIMEOUT  Seconds to wait for ARP reply
    -I IFACE    Interface to use (default eth0)
    -s SRC_IP   Sender IP address
    DST_IP      Target IP address

and as expected listed in the system arp-table:

#> arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.200.20           ether   f0:2f:7b:19:8f:71   C                     enp0s3

[Zasari]

So an (bit ugly) option for a quick-fix might be to force the detection method in this integration before calling the get_mac_address method:

import getmac
...
getmac.getmac.FORCE_METHOD="UuidArpGetNode"
...

[scottsweb]

Just tested here too @Zasari

On the old working container, running: python -m getmac -v -dddd --ip 192.168.1.200

DEBUG    Trying: '_read_arp_file' (to_find: '192.168.1.200')
DEBUG    Result: {mac address}
DEBUG    Raw MAC found: {mac address}

And on the newer containers:

DEBUG    Initializing 'ip4' method cache (platform: 'linux')
DEBUG    27 methods available: CtypesHost, ArpingHost, ArpFile, SysIfaceFile, FcntlIface, UuidArpGetNode, UuidLanscan, GetmacExe, IpconfigExe, WmicExe, ArpExe, DarwinNetworksetupIface, ArpFreebsd, ArpOpenbsd, IfconfigWithIfaceArg, IfconfigEther, IfconfigOther, IpLinkIface, NetstatIface, IpNeighborShow, ArpVariousArgs, DefaultIfaceLinuxRouteFile, DefaultIfaceIpRoute, DefaultIfaceRouteCommand, DefaultIfaceRouteGetCommand, DefaultIfaceOpenBsd, DefaultIfaceFreeBsd
DEBUG    9 type-filtered methods for 'ip4': CtypesHost, ArpingHost, ArpFile, UuidArpGetNode, ArpExe, ArpFreebsd, ArpOpenbsd, IpNeighborShow, ArpVariousArgs
DEBUG    5 platform-filtered methods for 'linux' (method_type='ip4'): ArpingHost, ArpFile, UuidArpGetNode, IpNeighborShow, ArpVariousArgs
DEBUG    5 tested methods for 'ip4': ArpingHost, ArpFile, UuidArpGetNode, IpNeighborShow, ArpVariousArgs
DEBUG    Current method cache: {'ip4': 'ArpingHost', 'ip6': 'None', 'iface': 'None', 'default_iface': 'None'}
DEBUG    Current fallback cache: {'ip4': '[<getmac.getmac.ArpFile object at 0x7fee8457e860>, <getmac.getmac.UuidArpGetNode object at 0x7fee8457e9e0>, <getmac.getmac.IpNeighborShow object at 0x7fee8457e8f0>, <getmac.getmac.ArpVariousArgs object at 0x7fee8457e980>]', 'ip6': '[]', 'iface': '[]', 'default_iface': '[]'}
DEBUG    Finished initializing 'ip4' method cache
DEBUG    Not sending UDP packet, using network request method 'ArpingHost' instead
DEBUG    Attempting get() (method='ArpingHost', method_type='ip4', arg='192.168.1.200')
DEBUG    Running: '/usr/sbin/arping --ridiculous-garbage-string'
DEBUG    Running: '/usr/sbin/arping -r -C 1 -c 1 192.168.1.200'
DEBUG    Method 'ArpingHost' failed for 'ip4' lookup
DEBUG    Raw MAC found: None
DEBUG    getmac took 0.0040 seconds

Will see if your temp fix works for me too

[gajosu]

Same here, after update to last version, nmap stops working

[scottsweb]

Following the tips from @Zasari, I have something working.

1. Get into the container podman exec -it homeassistant bash
2. Head to this folder cd /usr/src/homeassistant/homeassistant/components/nmap_tracker/
3. Install nano with apk add nano
4. Edit the init file nano __init__.py
5. Replace this from getmac import get_mac_address with

import getmac
from getmac import get_mac_address
getmac.getmac.FORCE_METHOD="ArpFile"

6. exit and restart the container

The ArpFile method was what the old version used for me, will be interesting to see if this works for others. If it does I will open a PR.

[mark007]

Thanks @scottsweb I have added your workaround/fix into a pull request for the kef integration which is also affected. It does work, however it generates warnings in the HA logs like below, and not just for one integration, for all integrations using getmac.

I'm thinking now should this fix go at HA level, possibly silencing this warning. It seems fixing this at a single integration, affects other integrations also.

Logger: getmac
Source: /usr/local/lib/python3.10/site-packages/getmac/getmac.py:1615
First occurred: 15:33:31 (4 occurrences)
Last logged: 15:33:43

Forcing method 'ArpFile' to be used for 'ip4' lookup (arg: '192.168.8.162')
Forcing method 'ArpFile' to be used for 'ip4' lookup (arg: '192.168.8.1')
Forcing method 'ArpFile' to be used for 'ip4' lookup (arg: '192.168.8.121')

[scottsweb]

@mark007 - it looks like a log level can be set for getmac. Details here: https://pypi.org/project/getmac/ and here: https://docs.python.org/3/library/logging.html#logging.basicConfig

There are also newer versions of getmac we might want to test. 3 released in the the last few weeks.

[Zasari]

Indeed I guess the nicest solution would be having support from getmac based on the limited arping used here - or having a test by using a more featured version of arping in HA.

[mark007]

I have tried 0.9.2 in the manifest.json of my copy of the kef integration within my custom_components directory, rebooted HA and it didn't help. I'm not sure if that's the correct way to update the version. Maybe we should involve the getmac Devs? As it feels wrong to have to force override the method doesn't it.

[Zasari]

Perhaps the easiest fix would simply to install a full arping in the HA container - that should at least be a quick-fix for those having the issues right now without extra log messages from getmac:

apk add arping

But idk the process on how HA decides which packages are to be added into the base image...

[scottsweb]

apk add arping

Didn't seem to fix it for me unfortuantely. The original patch is still good though.

[mark007]

As requested by the getmac developer I have created an issue here as he thinks he might know the cause and a quick fix.

https://github.com/GhostofGoes/getmac/issues/83

[Zasari]

apk add arping

Didn't seem to fix it for me unfortuantely.

@scottsweb : Indeed, it seems to not work reliably with the arping version - it seems to fail when devices (like Android smartphones) go into some deeper sleep stages - as this phone was resting on battery the whole night at home:

[Zasari]

I've checked for other python packages that could be used - and it seems a pretty stable tool is 'scapy' that would provide the same function:

#> pip install scapy

Ex.:

#> python
>>> import scapy.all as scapy
>>> scapy.getmacbyip( ip="192.168.200.150" )
'12:34:56:67:89:0a'

Of course a much bigger pip package, but seems to be pretty stable.

Update: I did a quick test, but it seems always the router's mac address is returned 🤷 So not an option, if I didn't miss something obvious...

[gajosu]

Following the tips from @Zasari, I have something working.

1. Get into the container podman exec -it homeassistant bash
2. Head to this folder cd /usr/src/homeassistant/homeassistant/components/nmap_tracker/
3. Install nano with apk add nano
4. Edit the init file nano __init__.py
5. Replace this from getmac import get_mac_address with

import getmac
from getmac import get_mac_address
getmac.getmac.FORCE_METHOD="ArpFile"

6. exit and restart the container

The ArpFile method was what the old version used for me, will be interesting to see if this works for others. If it does I will open a PR.

I made these changes and they work, I created a custom component to temporarily override the native nmap_tracker component.

I leave here attached the custom_component with the change in case someone needs it nmap_tracker.zip

[carefulcomputer]

Thanks @scottsweb for folks running in HA containers, i found the file in this path

/usr/lib/python3.10/site-packages/homeassistant/components/nmap_tracker/__init__.py

[mark007]

I have created this pull request, currently linked to the kef integration issue but should also fix this issue if it was merged. Instead of overriding the getmac method, the proposed approach is to fix the conflict between some integrations still using the old 'get-mac' library and other integrations using the newer 'getmac' library, both of which conflict with eachother (essentially getmac is overridden by its older get-mac counterpart, as internally it uses the same python package name, and I guess as alphabetically it comes first, python chooses it instead of the newer getmac).

https://github.com/home-assistant/core/pull/87712

Any help with this patch is welcome as I'm unsure how to test it any further than simply making the changes to the requirements files, the constraints file (to prevent future integrations bringing back the old get-mac again) and updating the only integration in the core that I see bringing in get-mac (vilfo-api-client) to a newer version, which did actually swap out get-mac for the new getmac.

[scottsweb]

Noting this is still an issue as of 2023.4.2

[jocamane]

too many time to resolve !!!! Looks like this addon died

[freeDom-]

I have created this pull request, currently linked to the kef integration issue but should also fix this issue if it was merged. Instead of overriding the getmac method, the proposed approach is to fix the conflict between some integrations still using the old 'get-mac' library and other integrations using the newer 'getmac' library, both of which conflict with eachother (essentially getmac is overridden by its older get-mac counterpart, as internally it uses the same python package name, and I guess as alphabetically it comes first, python chooses it instead of the newer getmac).

87712

Any help with this patch is welcome as I'm unsure how to test it any further than simply making the changes to the requirements files, the constraints file (to prevent future integrations bringing back the old get-mac again) and updating the only integration in the core that I see bringing in get-mac (vilfo-api-client) to a newer version, which did actually swap out get-mac for the new getmac.

Can someone confirm that the issue no longer exists? If so the issue can be closed, since the mentioned PR is merged already.

[scottsweb]

@freeDom- as of 2023.5.4 it was still an issue for me. I am not sure when the merged PR will be released? perhaps 2023.6?

[freeDom-]

It was merged last week, so it is currently only available on the dev environment and should be included in the next update

[GhostofGoes]

This issue should be resolved with getmac 0.9.4. I'm not familiar with developing for Home Assistant and am unable to test if it fixes it for sure. If someone would be willing to test this version I'd appreciate it.

side chatter: I'd recommend against using scapy. It's an excellent library for cybersecurity work or constructing custom packets. however, it's quite heavy and takes several seconds just to import it, and some of the util functions are slow (for example on windows it shells out to powershell). getmac was actually originally created to provide as a lighter-weight alternative to scapy for some work i was doing.

[scottsweb]

Confirming this is working again in 2023.6 - hopefully you all see the same results. If not, probably best to open a new issue anyway. Thanks to everyone who pitched in 🙇