Pathspider not terminating with dscp

[nstudach]

Pathspider will not terminate on its own. works with: pspdr measure -i eth0 --input webtest.ndjson --output sgGxn-ams3-ecn -w 20 ecn Reproducing error: 1: pspdr measure -i eth0 --input webtest.ndjson --output NHJ6q-location:fra1-dscp -w 20 dscp also with:

2. pspdr measure -i eth0 --input webtest.ndjson --output NHJ6q-fra1-dscp -w 20 dscp

Terminal output: 1:

ERROR:pathspider:exception occurred. terminating.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 403, in exception_wrapper
    target(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 410, in run_flow_enqueuer
    f = self._next_flow()
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 320, in _next_flow
    if not self._next_packet():
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 183, in _next_packet
    keep_flow = keep_flow and fn(rec, self._pkt.ip6, rev=rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 102, in ip6
    return self._dscp_extract(rec, ip, rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 112, in _dscp_extract
    if ip.pkt_len == (ip.hdr_len + ip.tcp.doff) * 4: # No payload
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
INFO:pathspider:terminating pathspider
Process observer:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 403, in exception_wrapper
    target(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 410, in run_flow_enqueuer
    f = self._next_flow()
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 320, in _next_flow
    if not self._next_packet():
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 183, in _next_packet
    keep_flow = keep_flow and fn(rec, self._pkt.ip6, rev=rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 102, in ip6
    return self._dscp_extract(rec, ip, rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 112, in _dscp_extract
    if ip.pkt_len == (ip.hdr_len + ip.tcp.doff) * 4: # No payload
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 410, in exception_wrapper
    self.terminate()
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 588, in terminate
    self.merger_thread.join()
AttributeError: 'NoneType' object has no attribute 'join'

2:

INFO:pathspider:activating spider...
INFO:pathspider:starting pathspider
INFO:pathspider:Creating observer
ERROR:pathspider:exception occurred. terminating.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 403, in exception_wrapper
    target(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 410, in run_flow_enqueuer
    f = self._next_flow()
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 320, in _next_flow
    if not self._next_packet():
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 183, in _next_packet
    keep_flow = keep_flow and fn(rec, self._pkt.ip6, rev=rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 102, in ip6
    return self._dscp_extract(rec, ip, rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 112, in _dscp_extract
    if ip.pkt_len == (ip.hdr_len + ip.tcp.doff) * 4: # No payload
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
INFO:pathspider:terminating pathspider
Process observer:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 403, in exception_wrapper
    target(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 410, in run_flow_enqueuer
    f = self._next_flow()
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 320, in _next_flow
    if not self._next_packet():
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/observer.py", line 183, in _next_packet
    keep_flow = keep_flow and fn(rec, self._pkt.ip6, rev=rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 102, in ip6
    return self._dscp_extract(rec, ip, rev)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/chains/dscp.py", line 112, in _dscp_extract
    if ip.pkt_len == (ip.hdr_len + ip.tcp.doff) * 4: # No payload
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 410, in exception_wrapper
    self.terminate()
  File "/usr/local/lib/python3.6/dist-packages/pathspider-2.0.0.dev0-py3.6.egg/pathspider/base.py", line 588, in terminate
    self.merger_thread.join()
AttributeError: 'NoneType' object has no attribute 'join'
INFO:feeder:job_feeder: all jobs added, waiting for spider to finish
INFO:pathspider:opening output file NHJ6q-fra1-dscp
INFO:pathspider:beginning shutdown

[mirjak]

yes there is no hdr_len in ipv6… can you try to remove the IPv6 entry in webtests and see if it works…?

[nstudach]

You are right. I deleted the one IPv6 adress and it worked. Am I wrong or should it not also work with IPv6?

[mirjak]

It should work now. I push a patch yesterday.

[irl]

Yes, and then I fixed your patch.

[mirjak]

Thanks! And sorry, I did this in a bit a rush yesterdays evening