Migrate from iwscan/iwlist to iw-nl80211 + several tweaks and improvements

[DL6ER]

IMPORTANT: Please take note of the below checklist, especially the first two items.

Pull Request Checklist

• [X] All pull requests must include the Contributor License Agreement (see below).
• [X] Code should conform to the following:
  • [X] pep8 compliant with some exceptions (see pytest.ini)
  • [ ] 100% test coverage with pytest (with valid tests). If you have difficulty writing tests for the code, feel free to ask for help or submit the PR without tests.
  • [X] Complete, correctly-formatted documentation for all classes, functions and methods.
  • [ ] documentation has been rebuilt with tox -e docs
  • [X] All modules should have (and use) module-level loggers.
  • [X] Commit messages should be meaningful, and reference the Issue number if you're working on a GitHub issue (i.e. "issue #x - "). Please refrain from using the "fixes #x" notation unless you are sure that the the issue is fixed in that commit.
  • [X] Git history is fully intact; please do not squash or rewrite history.

Contributor License Agreement

By submitting this work for inclusion in wifi-survey-heatmap, I agree to the following terms:

• The contribution included in this request (and any subsequent revisions or versions of it) is being made under the same license as the wifi-survey-heatmap project (the Affero GPL v3, or any subsequent version of that license if adopted by wifi-survey-heatmap).
• My contribution may perpetually be included in and distributed with wifi-survey-heatmap; submitting this pull request grants a perpetual, global, unlimited license for it to be used and distributed under the terms of wifi-survey-heatmap's license.
• I have the legal power and rights to agree to these terms.

---

This PR adds a number of improvements and tweaks:

wifi-survey

1. Migrate from iwconfig to iw-nl80211 (communicate directly with the kernel), fixes #1 ~at least partially~~ Existing solutions don't give direct access to scanning so I left iwscan in there for now, I may revisit this later done
2. Add new option to control duration of iperf3 test runs. In my experience, 10 seconds is often not enough when mapping far away from the router, it needs some "ramp up" time.
3. Show progress of scan in currently active survey circle (actively counting towards 100% while a survey is being performed)

wifi-heatmap

1. Update JSON parser to read new format It is a bit unfortunate that the current JSON format stores iwconfig and iwscan objects. The iwconfig object is now gone and replaced by generous fields. This breaks backwards compatibility with older JSON files but seems unavoidable without a (probably difficult to maintain) converter
2. Add option wifi-heatmap --show-points to show points. They are hidden now by default as they are distracting without providing additional information on sufficiently dense measurements.
3. Add frequency_TITLE.png heatmap to show band-steering effects. (no band switching was done in my quick test walking)
4. Add channel_bitrate_TITLE.png heatmap to show negotiated maximum bandwidth on the current channel
5. Print RSS (received signal strength) with correctly scaled units (assuming the wireless card is calibrated sufficiently well)
6. Change headers from TCP/reverse-TCP/UDP to more user-friendly Download (TCP) / Upload (TCP) / Download (UDP)
7. Reenabled Download (UDP) (former reverse-UDP) because if works just fine with iperf3 (was a problem with iperf2)
8. Add option -c <colormap name> to specify other colormaps (e.g. to highlight differences, white = average, red = better than average, blue = less than average) (this is -c bwr)

General

1. Add Quick Start to README (pointing more prominently towards docker) and other minor tweaks for the README
2. docker container has been updated and tested

---

I have neither touched any tests nor modified the documentation (outside of adding a lot more comments to the code itself).

If these changes are accepted (and wanted), I have some more in the pipeline to hopefully improve the user experience and feeling of this project. Especially also for the final representation of the maps and UI tweaks such as floor plan scaling to best fit onto a given screen.

[DL6ER]

I do not feel responsible for the failed Travis builds:

wifi_survey_heatmap/vendor/iwlib/__pycache__/_cffi__xed268f7bx65d109ba.c:265:19: fatal error: iwlib.h: No such file or directory

compilation terminated.

They should probably run in the very same docker container that are provided for the one-click deployments.

[DL6ER]

Several real-life tests this evening showed that PyRIC but also iwscan aren't very reliable. I will port the entire code to use netlink (nl80211) sockets directly over the weekend. This will give us the best solution. Implementing this takes a bit of time and proper testing so stay tuned.

[DL6ER]

Migration from iwscan/iwconfig to using only libnl is now complete

I had to fix two issues in libnl as well to get it python3.8-compatible and also to ensure scanning always returns all available data. Hence, please use my fixed fork

ip3 install --upgrade --user git+https://github.com/DL6ER/libnl

for installing the library right now. PRs towards the main library are already set up.

[DL6ER]

Illustration of what the last commit added:

Points are always stored in absolute coordinates of the floorplan.

[jantman]

@DL6ER This is, frankly, amazing. I'm stunned that you'd put this much work into it, and embarrassed that you had to deal with a pretty messy codebase that I really only intended on using once. I've never used this project for work, so I just use it when I move or help someone set up a new AP.

Everything here looks fine to me, so I'm more than happy to merge this even though I haven't run the new code myself. I'm even more flattered to have a contribution from a developer of a really well-known piece of software! I'm more than happy to merge anything else you want to contribute, especially since I doubt I'll be using this code again for quite some time.

[DL6ER]

That's the power of open source :-) I made some more changes yesterday. My main focus is on making it easy to use so people without much experience can use it more easily.

[jantman]

That's the power of open source :-) I made some more changes yesterday. My main focus is on making it easy to use so people without much experience can use it more easily.

Wonderful, thanks! All improvements will be gladly merged, especially since I have no idea when I'll be using this again.