search

nitred / nr-wg-mtu-finder

MIT License
202 stars 22 forks source link

Errors at end of test run. #3

Closed zandr closed 2 years ago

zandr commented 2 years ago

I've been trying to dial in a WG connection between two sites that are about 200ms apart, with one end using PPPoE. I'm running Ubuntu in a Parallels VM on the peer side and in Boot Camp on an old mac on the server side.

I seem to get a good test run, with the right number of rows written to the .csv, but at the end of the run both sides throw errors. On the peer I get:

  File "/usr/local/bin/nr-wg-mtu-finder", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.8/dist-packages/nr_wg_mtu_finder/main.py", line 115, in run
    MTUFinder(**args.dict())
  File "/usr/local/lib/python3.8/dist-packages/nr_wg_mtu_finder/mtu_finder.py", line 58, in __init__
    self.run_peer_mode()
  File "/usr/local/lib/python3.8/dist-packages/nr_wg_mtu_finder/mtu_finder.py", line 294, in run_peer_mode
    self.server_mtu, server_status = self.__peer_mode__wait_for_server_init()
  File "/usr/local/lib/python3.8/dist-packages/nr_wg_mtu_finder/mtu_finder.py", line 232, in __peer_mode__wait_for_server_init
    server_mtu, server_status = resp.json()["server_mtu"], resp.json()["server_status"]
  File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

and on the server I get

[2022-06-11 23:14:26,486] ERROR in app: Exception on /server/status [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/local/lib/python3.8/dist-packages/nr_wg_mtu_finder/sync_server.py", line 34, in server_status
    shutdown_server()
  File "/usr/local/lib/python3.8/dist-packages/nr_wg_mtu_finder/sync_server.py", line 18, in shutdown_server
    raise RuntimeError("Not running with the Werkzeug Server")
RuntimeError: Not running with the Werkzeug Server
10.2.0.2 - - [11/Jun/2022 23:14:26] "GET /server/status HTTP/1.1" 500 -

It looks like the data is there, though. Is there a way to manually process the CSV into a heatmap?

zandr commented 2 years ago

Thank you, btw, for a great tool. This is fantastic for automating an annoying problem.

nitred commented 2 years ago

@zandr Thank you very much for using it! Sorry you're facing issues. I don't have a lot of time today to go through and fix the underlying problem. My best guess is that the problem occurs because I haven't pinned any of the project dependencies. Anyway, it's very good that you have the CSV and here's how you can generate the heatmap.

If you have already installed nr-wg-mtu-finder using pip, then you can create a my_plot.py file with the following contents. Then run the file with python my_plot.py

# my_plot.py
from nr_wg_mtu_finder.plot import plot_log
plot_log(
    log_filename='/path/to/my_log.csv',
    plot_filename='/path/to/my_plot.png'
)

If that doesn't work either, then you can have a look at the issue #2 for how the generation of heatmap works and feel free to clone the git repo and alter the source code of nr_wg_mtu_finder/plot.py to generate your heatmap.

I will keep this issue open to find the root cause of the issue later and also to make sure to create a CLI tool for heatmap generation in case anyone else faces the same issue.

zandr commented 2 years ago

Let me know if I can help with versions of any of the deps. Both machines are 20.04.4 LTS, the peer was installed fresh today, the server was updated today. nr-wg-mtu-finder was installed on both ends today from github.

The snippet works a treat for manually generating the heatmap. Thanks!

nitred commented 2 years ago

Thanks for the offer! But, I think I have an old environment for the project somewhere which contains a working version of the project and I'll just get the versions of all the dependencies from there in one go 👍

nitred commented 2 years ago

@zandr If possible, would you be able to share your heatmap (only if you want to). I am really curious about how MTUs behave. I have a created an issue where you can share it - #4

zandr commented 2 years ago

Running a full sweep now, will post heatmaps in a few hours.

krstik commented 2 years ago

Hi everybody, have the same problem... How to resolve it? and create heatmap?

`reating log file: wg_mtu_finder_peer_20220615T180040.csv: SUCCESS

Pinging server to establish connection : SUCCESS Send peer ready for next loop to server : SUCCESS Pinging server to establish connection : SUCCESS Waiting for server init and status : FAILED, SERVER_STATUS: NOT_INITIALIZED, Retrying... Waiting for server init and status : Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.8/http/client.py", line 1256, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 951, in send self.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xffff99e75760>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.2.0.1', port=5000): Max retries exceeded with url: /server/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff99e75760>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/parallels/.local/bin/nr-wg-mtu-finder", line 8, in sys.exit(run()) File "/home/parallels/.local/lib/python3.8/site-packages/nr_wg_mtu_finder/main.py", line 115, in run MTUFinder(args.dict()) File "/home/parallels/.local/lib/python3.8/site-packages/nr_wg_mtu_finder/mtu_finder.py", line 58, in init self.run_peer_mode() File "/home/parallels/.local/lib/python3.8/site-packages/nr_wg_mtu_finder/mtu_finder.py", line 294, in run_peer_mode self.server_mtu, server_status = self.peer_mode__wait_for_server_init() File "/home/parallels/.local/lib/python3.8/site-packages/nr_wg_mtu_finder/mtu_finder.py", line 226, in peer_mode__wait_for_server_init resp = requests.get( File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.2.0.1', port=5000): Max retries exceeded with url: /server/status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff99e75760>: Failed to establish a new connection: [Errno 111] Connection refused')) `

krstik commented 2 years ago

And this is log from server

Serving Flask app 'nr_wg_mtu_finder.sync_server' (lazy loading)

nitred commented 2 years ago

@krstik I'm currently on vacation and will fix the underlying problem once I'm back.

However it appears you were able to successfully create the log file on the peer according to the logs you pasted:

creating log file: wg_mtu_finder_peer_20220615T180040.csv: SUCCESS

This means your run has finished successfully but failed only at the final heatmap generation phase. You can generate the heatmap manually by following the instructions I wrote in an earlier comment on this issue. The instructions worked for Zandar.

If you happen to be successful in generating the heatmap, I'd appreciate it if you could upload the heatmap image with some addtional details to issue #4.

nitred commented 2 years ago

I was able to replicate the error that was reported @zandr @krstik . I have a released a new version (0.2.0) that fixes this issue by pinning the dependencies.

You can now install the project directly from Pypi now with pip install nr-wg-mtu-finder==0.2.0 --upgrade

I have also added a new script nr-wg-mtu-finder-heatmap which will allow you to generate heatmaps from partial log files in case the original script (i.e. nr-wg-mtu-finder) crashes mid way and doesn't finish before the heatmap is generated.

Thanks for reporting the issue! I will close it for now.