search

teto / pymptcpanalyzer

Tool to help analyze mptcp pcaps
21 stars 5 forks source link

AttributeError: 'MpTcpAnalyzerCmdApp' object has no attribute 'colorize' #12

Closed deshmukhrajvardhan closed 5 years ago

deshmukhrajvardhan commented 5 years ago

Output of mptcpanalyzer -dDEBUG:

Setting log level to DEBUG
DEBUG:root:Starting in folder /home/raj
DEBUG:root:Pandas version: 0.23.4
DEBUG:root:cmd2 version: 0.9.7
An error happened:
'MpTcpAnalyzerCmdApp' object has no attribute 'colorize'
Displaying backtrace:

Traceback (most recent call last):
  File "/home/raj/.local/lib/python3.7/site-packages/mptcpanalyzer/cli.py", line 1216, in main
    analyzer = MpTcpAnalyzerCmdApp(config, **vars(args))
  File "/home/raj/.local/lib/python3.7/site-packages/mptcpanalyzer/cli.py", line 142, in __init__
    self.prompt = self.colorize("Ready>", "blue")
AttributeError: 'MpTcpAnalyzerCmdApp' object has no attribute 'colorize'

how to reproduce

Link to the incriminated pcap (if possible/relevant):

teto commented 5 years ago

0.9.7 removed the colorize feature, https://github.com/python-cmd2/cmd2/blob/master/CHANGELOG.md i will update the package shortly.

teto commented 5 years ago

I hope to complete the fix tomorrow meanwhile you can downgrade cmd2 to 0.9.6 via changing this line https://github.com/teto/mptcpanalyzer/blob/7df9d127d0d200483d393bb952f4870d3b41f1e4/setup.py#L135 to cmd2==0.9.6 and then work in develop mode via "python3 setup.py develop". Feel free to ask should you have any question about usage.

deshmukhrajvardhan commented 5 years ago 
* tshark version ($ tshark --version): TShark (Wireshark) 2.6.6 (Git v2.6.6 packaged as 2.6..4 packaged as 2.6.4-2~ubuntu16.04.0)
sudo python3.5 setup.py develop
[sudo] password for rajvardhan: 
running develop
Checking .pth file support in /usr/local/lib/python3.5/dist-packages/
/usr/bin/python3.5 -E -c pass
TEST PASSED: /usr/local/lib/python3.5/dist-packages/ appears to support .pth files
running egg_info
writing entry points to mptcpanalyzer.egg-info/entry_points.txt
writing dependency_links to mptcpanalyzer.egg-info/dependency_links.txt
writing requirements to mptcpanalyzer.egg-info/requires.txt
writing mptcpanalyzer.egg-info/PKG-INFO
writing top-level names to mptcpanalyzer.egg-info/top_level.txt
reading manifest file 'mptcpanalyzer.egg-info/SOURCES.txt'
writing manifest file 'mptcpanalyzer.egg-info/SOURCES.txt'
running build_ext
Creating /usr/local/lib/python3.5/dist-packages/mptcpanalyzer.egg-link (link to .)
mptcpanalyzer 0.3 is already the active version in easy-install.pth
Installing mptcpanalyzer script to /usr/local/bin

Installed /media/rajvardhan/0346f3bf-703e-41d8-8470-222066422528/recent_exp/mptcpanalyzer
Processing dependencies for mptcpanalyzer==0.3
Searching for cmd2==0.9.6
Best match: cmd2 0.9.6
Processing cmd2-0.9.6-py3.5.egg
cmd2 0.9.6 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/cmd2-0.9.6-py3.5.egg
Searching for pandas==0.24.1
Best match: pandas 0.24.1
Processing pandas-0.24.1-py3.5-linux-x86_64.egg
pandas 0.24.1 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/pandas-0.24.1-py3.5-linux-x86_64.egg
Searching for matplotlib==3.0.2
Best match: matplotlib 3.0.2
Processing matplotlib-3.0.2-py3.5-linux-x86_64.egg
matplotlib 3.0.2 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/matplotlib-3.0.2-py3.5-linux-x86_64.egg
Searching for stevedore==1.30.0
Best match: stevedore 1.30.0
Processing stevedore-1.30.0-py3.5.egg
stevedore 1.30.0 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/stevedore-1.30.0-py3.5.egg
Searching for wcwidth==0.1.7
Best match: wcwidth 0.1.7
Processing wcwidth-0.1.7-py3.5.egg
wcwidth 0.1.7 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/wcwidth-0.1.7-py3.5.egg
Searching for attrs==18.2.0
Best match: attrs 18.2.0
Processing attrs-18.2.0-py3.5.egg
attrs 18.2.0 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/attrs-18.2.0-py3.5.egg
Searching for colorama==0.4.1
Best match: colorama 0.4.1
Processing colorama-0.4.1-py3.5.egg
colorama 0.4.1 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/colorama-0.4.1-py3.5.egg
Searching for pyperclip==1.7.0
Best match: pyperclip 1.7.0
Processing pyperclip-1.7.0-py3.5.egg
pyperclip 1.7.0 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/pyperclip-1.7.0-py3.5.egg
Searching for numpy==1.16.1
Best match: numpy 1.16.1
Adding numpy 1.16.1 to easy-install.pth file
Installing f2py3.5 script to /usr/local/bin
Installing f2py3 script to /usr/local/bin

Processing python_dateutil-2.8.0-py3.5.egg
python-dateutil 2.8.0 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/python_dateutil-2.8.0-py3.5.egg
Searching for pyparsing==2.0.3
Best match: pyparsing 2.0.3
pyparsing 2.0.3 is already the active version in easy-install.pth

Using /usr/lib/python3/dist-packages
Searching for kiwisolver==1.0.1
Best match: kiwisolver 1.0.1
Processing kiwisolver-1.0.1-py3.5-linux-x86_64.egg
kiwisolver 1.0.1 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/kiwisolver-1.0.1-py3.5-linux-x86_64.egg
Searching for cycler==0.10.0
Best match: cycler 0.10.0
Processing cycler-0.10.0-py3.5.egg
cycler 0.10.0 is already the active version in easy-install.pth

Using /usr/local/lib/python3.5/dist-packages/cycler-0.10.0-py3.5.egg
Searching for six==1.10.0
Best match: six 1.10.0
six 1.10.0 is already the active version in easy-install.pth

Using /usr/lib/python3/dist-packages
Searching for pbr==5.1.2
Best match: pbr 5.1.2
Processing pbr-5.1.2-py3.5.egg
pbr 5.1.2 is already the active version in easy-install.pth
Installing pbr script to /usr/local/bin

Using /usr/local/lib/python3.5/dist-packages/pbr-5.1.2-py3.5.egg
Searching for setuptools==20.7.0
Best match: setuptools 20.7.0
setuptools 20.7.0 is already the active version in easy-install.pth
Installing easy_install script to /usr/local/bin

Using /usr/lib/python3/dist-packages

Output of mptcpanalyzer -dDEBUG:


  File "/usr/local/bin/mptcpanalyzer", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)```
teto commented 5 years ago

I've added support for 0.9.7 in https://github.com/teto/mptcpanalyzer/commit/11d6d9a3c2a1f730c9ec84ac885fbfe6a065f064 . Could you try please ? Let me know if you have trouble with the software. I would like to start make it useful to the public but it might still be a bit "raw".

deshmukhrajvardhan commented 5 years ago

Hi teto,

 # ran python3 -mpip install mptcpanalyzer
Requirement already satisfied: mptcpanalyzer in /home/raj/.local/lib/python3.7/site-packages (0.3)
Requirement already satisfied: cmd2>=0.9 in /home/raj/.local/lib/python3.7/site-packages (from mptcpanalyzer) (0.9.8)
Requirement already satisfied: pandas>=0.23 in /home/raj/anaconda3/lib/python3.7/site-packages (from mptcpanalyzer) (0.23.4)
Requirement already satisfied: stevedore in /home/raj/.local/lib/python3.7/site-packages (from mptcpanalyzer) (1.30.0)
Requirement already satisfied: matplotlib in /home/raj/anaconda3/lib/python3.7/site-packages (from mptcpanalyzer) (3.0.2)
Requirement already satisfied: attrs>=16.3.0 in /home/raj/anaconda3/lib/python3.7/site-packages (from cmd2>=0.9->mptcpanalyzer) (18.2.0)
Requirement already satisfied: colorama in /home/raj/anaconda3/lib/python3.7/site-packages (from cmd2>=0.9->mptcpanalyzer) (0.4.1)
Requirement already satisfied: wcwidth>=0.1.7 in /home/raj/anaconda3/lib/python3.7/site-packages (from cmd2>=0.9->mptcpanalyzer) (0.1.7)
Requirement already satisfied: pyperclip>=1.5.27 in /home/raj/.local/lib/python3.7/site-packages (from cmd2>=0.9->mptcpanalyzer) (1.7.0)
Requirement already satisfied: python-dateutil>=2.5.0 in /home/raj/anaconda3/lib/python3.7/site-packages (from pandas>=0.23->mptcpanalyzer) (2.7.5)
Requirement already satisfied: pytz>=2011k in /home/raj/anaconda3/lib/python3.7/site-packages (from pandas>=0.23->mptcpanalyzer) (2018.7)
Requirement already satisfied: numpy>=1.9.0 in /home/raj/anaconda3/lib/python3.7/site-packages (from pandas>=0.23->mptcpanalyzer) (1.15.4)
Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /home/raj/.local/lib/python3.7/site-packages (from stevedore->mptcpanalyzer) (5.1.2)
Requirement already satisfied: six>=1.10.0 in /home/raj/anaconda3/lib/python3.7/site-packages (from stevedore->mptcpanalyzer) (1.12.0)
Requirement already satisfied: cycler>=0.10 in /home/raj/anaconda3/lib/python3.7/site-packages (from matplotlib->mptcpanalyzer) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/raj/anaconda3/lib/python3.7/site-packages (from matplotlib->mptcpanalyzer) (1.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/raj/anaconda3/lib/python3.7/site-packages (from matplotlib->mptcpanalyzer) (2.3.0)
Requirement already satisfied: setuptools in /home/raj/anaconda3/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->mptcpanalyzer) (40.6.3)

but, i think the path is not recognized, as 

# running: mptcpanalyzer --load http1_no_reorder.pcap                                             
# oh my zsh shell it gives output
zsh: command not found: mptcpanalyzer
# on bash shell gives output
mptcpanalyzer: command not found

can you help me set the write path?

Thank you

teto commented 5 years ago

Hum it's strange. I never used anaconda but it seems to be in your $HOME. Did you run pip with a --user flag ? Maybe check in ~/.local/bin if you see mptcpanalyzer. Else i would try develop mode; like clone the repository then from the directory run python3 -mpip develop. See https://setuptools.readthedocs.io/en/latest/setuptools.html?highlight=development%20mode#develop-deploy-the-project-source-in-development-mode

deshmukhrajvardhan commented 5 years ago

Hi @teto , i was able to add the path using export PATH="/home/raj/.local/bin:$PATH"

But it the colorize issue isn't solved (even after doing the below suggested fix)

Line 135 in 7df9d12 'cmd2>=0.9', # to improve cmd capabilities to cmd2==0.9.6 and then work in develop mode via "python3 setup.py develop".

mptcpanalyzer --load http1_trace0.pcap                               
Setting log level to ERROR
An error happened:
'MpTcpAnalyzerCmdApp' object has no attribute 'colorize'
Displaying backtrace:

Traceback (most recent call last):
  File "/home/raj/.local/lib/python3.7/site-packages/mptcpanalyzer/cli.py", line 1216, in main
    analyzer = MpTcpAnalyzerCmdApp(config, **vars(args))
  File "/home/raj/.local/lib/python3.7/site-packages/mptcpanalyzer/cli.py", line 142, in __init__
    self.prompt = self.colorize("Ready>", "blue")
AttributeError: 'MpTcpAnalyzerCmdApp' object has no attribute 'colorize'
teto commented 5 years ago

I mean I haven't pushed yet a new release, might do next week would like to fix the autocompletion that seems broken with cmd2 0.9.7 before. Until then, you have to use the dev version (aka clone and use develop mode). Sorry for the inconvenience.

deshmukhrajvardhan commented 5 years ago

Hi @teto, I used python3 setup.py develop. I still get the 'colorize' error. Can you provide me with instructions after the python3 setup.py develop develop step?

Thanks

teto commented 5 years ago

If I grep for colorize, I don't find any reference. Have you updated the repository ? From your mptcpanalyzer repository, run:

git checkout master
git pull
python3 setup.py develop

you can check with git log the different commits. your repo should be at df376c4b0bcba724392a9d253637db7cae44bbd2 Next time you have an error, post the stacktrace.

deshmukhrajvardhan commented 5 years ago

Hi @teto , There were mptcpanalyzer folders in my .local directory (had to manually remove them) and /lib/python3.7/site-packages/ directory. (had to remove python3 -mpip uninstall mptcpanalyzer).

Then used 

git checkout master
git pull
python3 setup.py develop

My local repo is at df376c4

Now i get another error (tshark version related):

 mptcpanalyzer --load ../mptcp_1path_http1_2_ipc_11Feb/interface_1_7M_2M_mptcp_1path_ipc_http1_11Feb.pcap
Setting log level to ERROR
WARNING: mptcpanalyzer may require a custom wireshark. Check github for mptcp patches streaming.
ran cmd ['tshark', '-E', 'header=y', '-r', '/home/raj/harddisk/recent_exp/mptcp_1path_http1_2_ipc_11Feb/interface_1_7M_2M_mptcp_1path_ipc_http1_11Feb.pcap', '-E', 'separator=|', '-o', 'gui.column.format:"Time","%At","ipsrc","%s","ipdst","%d"', '-o', 'tcp.analyze_sequence_numbers:True', '-o', 'mptcp.analyze_mappings:True', '-o', 'mptcp.relative_sequence_numbers:True', '-o', 'mptcp.intersubflows_retransmission:True', '-o', 'mptcp.analyze_mptcp:True', '-2', '-R', 'mptcp or tcp and not icmp', '-T', 'fields', '-e', 'frame.number', '-e', 'frame.time_relative', '-e', 'frame.time_epoch', '-e', '_ws.col.ipsrc', '-e', '_ws.col.ipdst', '-e', 'ip.src_host', '-e', 'ip.dst_host', '-e', 'tcp.stream', '-e', 'tcp.srcport', '-e', 'tcp.dstport', '-e', 'tcp.window_size', '-e', 'tcp.flags', '-e', 'tcp.option_kind', '-e', 'tcp.seq', '-e', 'tcp.len', '-e', 'tcp.ack', '-e', 'tcp.options.timestamp.tsval', '-e', 'tcp.options.timestamp.tsecr', '-e', 'mptcp.expected_token', '-e', 'mptcp.stream', '-e', 'tcp.options.mptcp.sendkey', '-e', 'tcp.options.mptcp.recvkey', '-e', 'tcp.options.mptcp.recvtok', '-e', 'tcp.options.mptcp.datafin.flag', '-e', 'tcp.options.mptcp.subtype', '-e', 'tcp.options.mptcp.rawdataseqno', '-e', 'tcp.options.mptcp.rawdataack', '-e', 'tcp.options.mptcp.subflowseqno', '-e', 'tcp.options.mptcp.datalvllen', '-e', 'tcp.options.mptcp.addrid', '-e', 'mptcp.rawdsn64', '-e', 'mptcp.ack', '-e', 'mptcp.dsn', '-e', 'mptcp.related_mapping', '-e', 'mptcp.reinjection_of', '-e', 'mptcp.reinjected_in']
stderr= tshark: Some fields aren't valid:
    mptcp.reinjected_in
    mptcp.reinjection_of

An error happened:
tshark: Some fields aren't valid:
    mptcp.reinjected_in
    mptcp.reinjection_of

Displaying backtrace:

Traceback (most recent call last):
  File "/home/raj/harddisk/recent_exp/mptcpanalyzer/mptcpanalyzer/cli.py", line 1248, in main
    analyzer.onecmd("load_pcap %s" % args.input_file)
  File "/home/raj/.local/lib/python3.7/site-packages/cmd2/cmd2.py", line 2003, in onecmd
    stop = func(statement)
  File "/home/raj/.local/lib/python3.7/site-packages/cmd2/cmd2.py", line 255, in cmd_wrapper
    args = argparser.parse_args(lexed_arglist)
  File "/home/raj/anaconda3/lib/python3.7/argparse.py", line 1749, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/home/raj/harddisk/recent_exp/mptcpanalyzer/mptcpanalyzer/parser.py", line 548, in parse_known_args
    res = super().parse_known_args(args, namespace)
  File "/home/raj/anaconda3/lib/python3.7/argparse.py", line 1781, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/home/raj/.local/lib/python3.7/site-packages/cmd2/argparse_completer.py", line 1329, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/home/raj/.local/lib/python3.7/site-packages/cmd2/argparse_completer.py", line 1285, in consume_positionals
    take_action(action, args)
  File "/home/raj/.local/lib/python3.7/site-packages/cmd2/argparse_completer.py", line 1184, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/raj/harddisk/recent_exp/mptcpanalyzer/mptcpanalyzer/parser.py", line 77, in __call__
    df = load_into_pandas(values, self.loader)
  File "/home/raj/harddisk/recent_exp/mptcpanalyzer/mptcpanalyzer/data.py", line 388, in load_into_pandas
    raise Exception(stderr)
Exception: tshark: Some fields aren't valid:
    mptcp.reinjected_in
    mptcp.reinjection_of

my tshark version details:

tshark --version 
TShark (Wireshark) 2.6.6 (Git v2.6.6 packaged as 2.6.6-1~ubuntu18.04.0)

Copyright 1998-2019 Gerald Combs <gerald@wireshark.org> and contributors.
License GPLv2+: GNU GPL version 2 or later <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) with libpcap, with POSIX capabilities (Linux), with libnl 3,
with GLib 2.56.2, with zlib 1.2.11, with SMI 0.4.8, with c-ares 1.14.0, with Lua
5.2.4, with GnuTLS 3.5.18, with Gcrypt 1.8.1, with MIT Kerberos, with MaxMind DB
resolver, with nghttp2 1.30.0, with LZ4, with Snappy, with libxml2 2.9.4.

Running on Linux 4.15.0-45-generic, with Intel(R) Core(TM) i7-8750H CPU @
2.20GHz (with SSE4.2), with 15742 MB of physical memory, with locale
en_US.UTF-8, with libpcap version 1.8.1, with GnuTLS 3.5.18, with Gcrypt 1.8.1,
with zlib 1.2.11, binary plugins supported (13 loaded).

Built using gcc 7.3.0.

how do i update to the required version?

Thanks

teto commented 5 years ago

Maybe I should make it more explicit from the README (https://github.com/teto/mptcpanalyzer#required-wireshark-version) but you need to get a wireshark master, most likely by compiling it yourself. It's necessary to analyze MPTCP reinjections as otherwise wireshakr is buggy and they haven't made any release with my patch yet.

teto commented 5 years ago

Any news ? I saw wireshark 3.0 had a release candidate/beta so maybe you could find a precompiled package for ubuntu if you don't want to compile it yourself.

deshmukhrajvardhan commented 5 years ago

Hi @teto , Sorry for the delay. I have built wireshark with the latest commit till today. I have the binary in the /home/raj/harddisk/recent_exp/wireshark-ninja/run/tshark

But the mptcp-analyzer is using the path /usr/bin/tshark Can you help me set the tshark path variable?

Thanks

teto commented 5 years ago

I used to have a setting to set which wireshark to use, but in the end I think it's simpler to adjust your PATH environment variable so that your tshark appears before the system one. Either you install your custom wireshark (make install) and then it should go into /usr/local/bin which has precedence over /usr/bin or you adjust your PATH with sthg like export PATH=/home/raj/harddisk/recent_exp/wireshark-ninja/run:$PATH Note that wireshark needs an extra environment variable when not installed: export WIRESHARK_RUN_FROM_BUILD_DIRECTORY=1

deshmukhrajvardhan commented 5 years ago

Hi @teto , I adjusted the path and it works fine. I get till the below output:

[142184 rows x 40 columns]}, input_file='mptcp_1path_http1_2_ipc_11Feb/interface_1_7M_2M_mptcp_1path_ipc_http1_11Feb.pcap')
Loading mptcp_1path_http1_2_ipc_11Feb/interface_1_7M_2M_mptcp_1path_ipc_http1_11Feb.pcap

Press ? to list the available commands and `help <command>` or `<command> -h`
for a detailed help of the command

interface_1_7M_2M_mptcp_1path_ipc_http1_11Feb.pcap>

Can you help me get timestamp and MPTCP sequence number? My goal is to find packet retransmissions and packet reordering.

Thanks, Raj

teto commented 5 years ago

If you type help in the interpreter you can see what commands are available.

You should find the id of your mptcp connection first via list_mptcp_connections, then you can call summary on it.

If you have pcaps from both the client and the server, you can do deeper analysis of the reinjection with map_mptcp_connections and classify_reinjections.

The dynamic typing of python/pandas has been the source of lots of bugs so it's very possible that you find bugs. I am now working to polish the software/debug eventual problems/add tests etc.

In last resort, if only to find the reinjections, you can load the pcap and use the following filters: mptcp.reinjection_of and mptcp.reinjected_in

teto commented 5 years ago

I pushed a new release which should be more robust. Let me know if there are any issues.