Problem updating OUI database with `--oui_update`

[joshschmelzle]

@nimbo78 reports a failure to update the OUI database using --oui_update:

$ sudo profiler  --oui_update
2021-11-01 08:05:32,932 [INFO] update_manuf: manuf file is located at /opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf
2021-11-01 08:05:32,933 [INFO] update_manuf: manuf file last modified at: Sun Oct 31 15:32:31 2021
2021-11-01 08:05:32,934 [INFO] update_manuf: running 'sudo manuf --update'
Traceback (most recent call last):
  File "/opt/wlanpi/pipx/bin/profiler", line 8, in <module>
    sys.exit(main())
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/profiler/__main__.py", line 27, in main
    manager.start(args)
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/profiler/manager.py", line 78, in start
    sys.exit(0) if helpers.update_manuf() else sys.exit(-1)
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/profiler/helpers.py", line 441, in update_manuf
    cp = run_cli_cmd(["sudo", f"{sys.prefix}/bin/manuf", "--update"])
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/profiler/helpers.py", line 52, in run_cli_cmd
    capture_output=True,
  File "/usr/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['sudo', '/opt/wlanpi/pipx/venvs/profiler/bin/manuf', '--update']' returned non-zero exit status 1.

Originally posted by @nimbo78 in https://github.com/WLAN-Pi/wlanpi-profiler/issues/75#issuecomment-956017659

[joshschmelzle]

Can you try updating the OUI database with manuf outside of profiler?

Since it is installed in a virtualenv we need to specify the full path to it like this:

wlanpi@wlanpi:~$ sudo /opt/wlanpi/pipx/venvs/profiler/bin/manuf --update

Then check the last time the manuf db file was updated:

wlanpi@wlanpi:~$ stat /opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf
  File: /opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf
  Size: 1927876     Blocks: 3768       IO Block: 4096   regular file
Device: b301h/45825d    Inode: 269335      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-11-01 13:19:18.364504804 +0000
Modify: 2021-11-01 13:21:48.788012073 +0000
Change: 2021-11-01 13:21:48.788012073 +0000
 Birth: -

Then can you check if updating via profiler --oui_update still fails?

wlanpi@wlanpi:~$ sudo profiler --oui_update
2021-11-01 13:32:50,516 [INFO] update_manuf: manuf file is located at /opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf
2021-11-01 13:32:50,518 [INFO] update_manuf: manuf file last modified at: Mon Nov  1 13:26:28 2021
2021-11-01 13:32:50,518 [INFO] update_manuf: running 'sudo manuf --update'
2021-11-01 13:32:59,832 [INFO] update_manuf: completed process return code is non-zero
2021-11-01 13:32:59,833 [INFO] update_manuf: manuf file last modified at: Mon Nov  1 13:32:56 2021

[joshschmelzle]

Possibly related to #46

[nimbo78]

i think there's some service problem:

sudo /opt/wlanpi/pipx/venvs/profiler/bin/manuf --update
Traceback (most recent call last):
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 144, in update
    response = urlopen(Request(manuf_url, headers={'User-Agent': 'Mozilla'}))
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Service Temporarily Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/wlanpi/pipx/venvs/profiler/bin/manuf", line 8, in <module>
    sys.exit(main())
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 324, in main
    parser = MacParser(manuf_name=args.manuf, update=args.update)
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 70, in __init__
    self.update()
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 146, in update
    raise URLError("Failed downloading OUI database")
urllib.error.URLError: <urlopen error Failed downloading OUI database>

[joshschmelzle]

By default in this version of manuf, it pulls the data from wireshark.org, which I see now happens to redirect to gitlab.

Line 64 of manuf.py contains: https://code.wireshark.org/review/gitweb?p=wireshark.git;a=blob_plain;f=manuf as the MANUF_URL.

Looks like Cloudflare CDN is used, and when I browse to said link, I see URL redirection occurs to https://gitlab.com/wireshark/wireshark/raw/master/manuf.

Are you able to wget https://gitlab.com/wireshark/wireshark/raw/master/manuf successfully? Seems like possibly a reachability issue.

[nimbo78]

wget https://gitlab.com/wireshark/wireshark/raw/master/manuf
--2021-11-01 17:33:00--  https://gitlab.com/wireshark/wireshark/raw/master/manuf
Resolving gitlab.com (gitlab.com)... 172.65.251.78, 2606:4700:90:0:f22e:fbec:5bed:a9b9
Connecting to gitlab.com (gitlab.com)|172.65.251.78|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1916655 (1.8M) [text/plain]
Saving to: ‘manuf’

manuf                                  100%[============================================================================>]   1.83M  --.-KB/s    in 0.06s

2021-11-01 17:33:00 (32.9 MB/s) - ‘manuf’ saved [1916655/1916655]

[joshschmelzle]

Thanks for confirming. Can you try sudo /opt/wlanpi/pipx/venvs/profiler/bin/manuf --update again? From the same network you've been testing on, are you able to browse to https://code.wireshark.org/review/gitweb?p=wireshark.git;a=blob_plain;f=manuf and does the redirect work?

[nimbo78]

same device, same network, without poweroff from first message here:

wlanpi@wlanpi:~$ sudo /opt/wlanpi/pipx/venvs/profiler/bin/manuf --update
[sudo] password for wlanpi:
Traceback (most recent call last):
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 144, in update
    response = urlopen(Request(manuf_url, headers={'User-Agent': 'Mozilla'}))
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Service Temporarily Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/wlanpi/pipx/venvs/profiler/bin/manuf", line 8, in <module>
    sys.exit(main())
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 324, in main
    parser = MacParser(manuf_name=args.manuf, update=args.update)
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 70, in __init__
    self.update()
  File "/opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf/manuf.py", line 146, in update
    raise URLError("Failed downloading OUI database")
urllib.error.URLError: <urlopen error Failed downloading OUI database>

from browser ok, but there's cloudflare captcha\antiddos before redirecting. may be root of the problen in clouflare antiddos?

[joshschmelzle]

I suspect the captcha/antiddos is the problem, which I don't see from my network from browser. I took a quick glance at the manuf code, and I do not think it would handle that.

For manually updating the OUI database for profiler, something like this should do the trick:

cd /opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf
sudo rm manuf
sudo wget https://gitlab.com/wireshark/wireshark/raw/master/manuf

[nimbo78]

I suspect the captcha/antiddos is the problem, which I don't see from my network from browser. I took a quick glance at the manuf code, and I do not think it would handle that.

For manually updating the OUI database for profiler, something like this should do the trick:

cd /opt/wlanpi/pipx/venvs/profiler/lib/python3.7/site-packages/manuf
sudo rm manuf
sudo wget https://gitlab.com/wireshark/wireshark/raw/master/manuf

this trick works well, thnx

[joshschmelzle]

It seems like others have run into a similar issue using Python to retrieve files behind a Cloudflare CDN. https://stackoverflow.com/questions/62684468/pythons-requests-triggers-cloudflares-security-while-urllib-does-not

We either need to work on getting a fix in the upstream package or roll the maintenance of the OUI database into our codebase.

[joshschmelzle]

Fixed upstream in manuf by https://github.com/coolbho3k/manuf/pull/30.

[joshschmelzle]

Addressed in 8dc0b03cc7ed080dbcf12e74e5cd22e4cef29c13 where we bump manuf from 1.1.1 to 1.1.3 which has the fix. For reference https://github.com/coolbho3k/manuf/issues/29