somenath-sinha
commented
1 month ago Complete Log Files and Output (expected vs received on Netmiko).
datapath_sess_truncated.txt datapath_sess_expected.txt test.log
Closed somenath-sinha closed 1 month ago
somenath-sinha
commented
1 month ago datapath_sess_truncated.txt datapath_sess_expected.txt test.log
somenath-sinha
commented
1 month ago @ktbyers / Kirk, Your tool has been invaluable to me but I just can't figure out why it almost randomly stops accepting data even though it's clearly being sent by the device, and is in the log!
Please check this out as soon as you can, and let me know?!
ktbyers
commented
1 month ago It doesn't look like it is truncated from your test.log file:
DEBUG:netmiko:read_channel:
172.31.248.83 172.31.248.1 17 8209 8494 1/0 0 0 0 tunnel 3301 4 0 0 FYCI 13
10.148.124.254 10.236.0.160 6 22 45900 0/0 0 0 1 pc2 15 4 233 F 11
172.31.248.1 172.31.251.120 17 8222 8209 0/0 0 0 1 tunnel 2834 19 0 0 FYI 12
172.31.248.1 172.31.250.213 17 17110 8209 0/0 0 0 0 tunnel 2384 4 0 0 FYI 11
172.27.0.3 224.0.0.102 17 1985 1985 0/0 0 48 0 pc2 9222 32471 2597680 FC 14
10.147.22.119 224.0.0.252 17 62513 5355 0/0 0 24 0 tunnel 878 f 2 96 FTC 13
149.154.167.50 172.27.8.8 6 443 47060 0/0 0 0 0 tunnel 1650 11 3 169 F 15
10.250.1.43 10.148.124.254 6 61377 21 1/15784 0 0 1 pc2 17 2 80 FCI 11
10.147.22.119 23.215.205.230 6 52203 443 0/0 0 0 0 tunnel 878 4b 13 1550 FCi 13
172.31.249.5 172.31.248.1 17 8209 8494 0/0 0 0 0 tunnel 2432 e 0 0 FYCI 14
172.31.248.1 172.31.248.233 17 4500 4500 0/0 0 0 0 pc2 6 0 0 FY 12
172.31.248.1 172.31.248.250 17 8419 8209 0/0 0 0 0 tunnel 1221 0 0 0 FYI 12
172.27.9.153 172.217.194.188 6 52388 5228 0/0 0 0 2 tunnel 1816 1a41 162 8817 C 14
172.31.249.182 172.31.248.1 17 8209 8222 0/0 0 0 1 tunnel 1974 1a 0 0 FYCI 15
172.31.248.1 172.31.249.221 47 0 0 0/0 2 24 0 pc2 9aa4 31585372 1089898200 F 12
10.148.124.254 172.31.249.150 17 8515 8211 1/0 0 0 0 local 9 1 242 FCI 11
172.31.248.178 10.148.124.254 17 8211 8224 0/0 0 0 0 local 0 0 0 FYI 11
172.27.5.122 46.232.210.90 17 52683 6881 0/0 0 0 0 tunnel 2293 d 2 96 FC 11
(INMUM-02274-MB01SVR-WC02) *[mynode] #The prompt comes back at the end there.
Also your session_log files don't look like Netmiko's standard session_log files?
Also what do you mean you don't see an error--does this mean you terminate the program yourself or are catching the error (or something else)?
somenath-sinha
commented
1 month ago @ktbyers - I agree, the test.log file has the complete output, but the part that send_command() or send_command_expect() shows as the output is truncated - which is where the confusion lies. For example, this is the output when I run the script now. As you'll see, the number of entries is much smaller, and the last line doesn't even complete before send_command_expect() presents the output.
cd 'c:\Users\scripts\Aruba_SessionTracer'; & 'c:\Users\scripts\Aruba_SessionTracer\env\Scripts\python.exe' 'c:\Users\.vscode\extensions\ms-python.debugpy-2024.6.0-win32-x64\bundled\libs\debugpy\adapter/../..\debugpy\launcher' '53671' '--' 'C:\Users\scripts\Aruba_SessionTracer\tracker.py'
Connecting to: <device 1>
Connecting to: <device 2>
<device 2> is the primary controller - running commands on this.
Datapath Session Table Entries
------------------------------
Flags: F - fast age, S - src NAT, N - dest NAT
D - deny, R - redirect, Y - no syn
H - high prio, P - set prio, T - set ToS
C - client, M - mirror, V - VOIP
Q - Real-Time Quality analysis
u - Upstream Real-Time Quality analysis
I - Deep inspect, U - Locally destined
E - Media Deep Inspect, G - media signal
r - Route Nexthop, h - High Value
A - Application Firewall Inspect
i - Session classified on first packet
B - Permanent, O - Openflow
L - Log, o - Openflow config revision mismatched
Source IP or MAC Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge Packets Bytes Flags CPU ID
----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- -------
172.31.251.145 172.31.248.1 17 8209 17110 0/0 0 0 1 tunnel 1307 10 0 0 FYCI 12
172.27.9.50 103.228.155.199 6 56397 443 0/0 0 0 1 tunnel 2536 3975 5809 458983 C 15
10.148.124.254 172.31.250.221 17 8222 8211 0/0 0 0 1 local 4 3 402 FCI 11
172.31.249.237 172.31.248.1 17 8209 17110 0/0 0 0 1 tunnel 491 f 0 0 FYCI 14
212.93.114.10 172.27.5.122 6 54039 59904 0/0 0 0 2 tunnel 2293 1a 1 44 11
172.31.248.1 172.31.249.168 17 8494 8209 0/0 0 0 1 tunnel 3247 9 0 0 FYI 15
172.31.251.76 172.31.248.1 17 8209 8222 0/0 0 0 0 tunnel 327 2 0 0 FYCI 15
162.247.243.24 172.27.13.41 6 443 46634 0/0 0 0 5 tunnel 707 bf 13 2070 11
172.31.251.35 10.148.124.254 17 8211 8419 0/0 0 0 2 local 19 0 0 FYI 11
172.31.248.1 172.31.248.233 47 0 0 0/0 2 16 0 pc2 c3fd 29084259 1088271364 F 11
172.31.250.193 172.31.248.1 17 8209 8494 0/0 0 0 1 tunnel 2977 9 0 0 FYCI 13
3.64.42.113 10.147.19.105 6 443 53560 0/0 0 16 4 tunnel 3058 830 96 24558 h 15
10.147.22.120 20.212.88.117 6 53997 443 0/0 0 0 12 tunnel 3166 58d 25 11721 Cih 14
52.108.50.37 10.147.22.99 6 443 51617 0/0 0 24 1 tunnel 1651 1482 415 44264 i 15
172.31.248.187 172.31.248.1 17 8209 17110 0/0 0 0 1 tunnel 143 6 0 0 FYCI 11
172.31.248.1 172.31.251.198 17 4500 4500 0/0 0 0 2 pc2 86 2 253 F 11
172.31.248.98 172.31.248.1 17 8209 8421 0/0 0 0 1 tunnel 2945 10 0 0 FYCI 13
142.250.67.150 172.27.13.248 17 443 60631 0/0 0 46 1 tunnel 3356 c 11 7099 F 15
10.148.124.254 172.31.251.247 17 8494 8211 0/0 0 0 1 local 12 1 129 FCI 11
106.70.168.210 172.27.5.122 6 6881 59787 0/0 0 0 0 pc2 1 1 40 FC 14
10.148.124.254 172.31.251.68 17 8419 8211 0/0 0 0 1 local 10 2 1815 FCI 11
172.31.248.1 172.31.251.102 17 8494 8209 0/0 0 0 0 tunnel 3005 1 0 0 FYI 15
52.59.167.127 10.147.16.213 6 443 53400 0/0 0 16 3 tunnel 848 44e 55 13918 h 12
172.31.250.169 172.31.248.1 17 8209 8494 1/0 0 0 0 tunnel 2360 3 0 0 FYCI 11
110.42.235.42 172.27.5.122 6 49155 59472 0/0 0 0 2 tunnel 2293 2772 171 6984 12
198.244.190.234 172.27.5.122 6 6881 59827 0/0 0 11 3 tunnel 2293 2b 2 92 13
10.148.124.254 172.31.249.129 17 8224 8211 0/0 0 0 1 local d 2 1975 FCI 11
172.31.248.1 172.31.251.179 17 8494 8209 0/0 0 0 1 tunnel 10 4 0 0 FYI 12
172.31.248.1 172.31.251.30 47 0 0 0/0 4 32 0 pc2 c41c 58505658 3512790498 F 14
10.147.24.224 52.108.240.79 6 53273 443 0/0 0 24 0 tunnel 2248 176c 457 26854 TCi 15
205.185.114.24 172.27.5.122 6 6898 59934 0/0 0 10 1 tunnel 2293 14 4 172 F 12
172.31.248.1 172.31.249.110 17 8222 8209 0/0 0 0 0 tunnel 53 2 0 0 FYI 15
10.147.24.9 165.225.120.45 6 62913 443 0/0 0 24 1 tunnel 1815 da9 244 17651 TC 14
10.148.124.254 172.31.250.168 17 8419 8211 0/0 0 0 1 local d 2 1107 FCI 11
f8:04:2e:00:d4:40 806 0/0 0 0 0 tunnel 1021 508e 137599 281802752 F 15
172.31.248.1 172.31.251.43 17 8209 8209 0/0 0 0 8 tunnel 2589 83 0 0 FYI 11
142.250.70.118 172.27.5.37 6 443 47860 0/0 0 0 3 tunnel 1477 32 8 6282 15
10.148.124.254 172.31.251.182 17 8421 8211 0/0 0 0 1 local 7 1 104 FCI 11
172.31.248.1 172.31.251.203 47 0 0 0/0 4 24 0 pc2 c41d 87986536 134239262 F 14
172.31.248.1 172.31.250.255 17 4500 4500 0/0 0 0 2 pc2 1f 1 29 F 11
172.31.248.1 172.31.250.95 17 8494 8209 0/0 0 0 1 tunnel 2662 9 0 0 FYI 14
172.31.251.47 172.31.248.1 17 8209 8494 0/0 0 0 1 tunnel 2260 8 0 0 FYCI 12
52.108.240.79 10.147.22.119 6 443 59197 0/0 0 24 1 tunnel 878 1f2e 634 62776 i 14
10.148.124.254 172.31.251.49 17 8222 8211 1/0 0 0 0 local 0 1 177 FCI 11
172.31.248.1 172.31.249.10 17 8515 8209 0/0 0 0 0 tunnel 1419 3 0 0 FYI 14
8.8.8.8 172.27.0.101 6 443 52754 0/0 0 46 1 tunnel 2957 26c 1222 224389 i 12
54.254.133.151 172.27.9.138 6 2957 35460 0/0 0 0 0 tunnel 1717 2201 3048 159506 13
172.31.248.1 172.31.251.160 17 8222 8209 0/0 0 0 1 tunnel 1555 6 0 0 FYI 13
20.38.108.228 172.27.9.138 6 443 44194 0/0 0 0 1 tunnel 1717 9 306 24924 Fi 12
172.31.249.141 172.31.248.1 17 8209 8444 0/0 0 0 1 tunnel 2660 e 0 0 FYCI 11
172.31.251.41 172.31.248.1 17 8209 17110 0/0 0 0 1 tunnel 2508 b 0 0 FYCI 13
172.31.248.1 172.31.249.5 47 0 0 0/0 4 32 0 pc2 c41d 45590077 1832812700 F 14
10.148.124.254 172.31.251.228 17 8222 8211 0/0 0 0 1 local 4 1 177 FCI 11
172.27.5.122 146.70.204.168 6 60059 64386 0/0 0 0 0 tunnel 2293 2 2 104 YC 11
172.27.13.228 142.250.183.202 6 45436 443 0/0 0 0 0 tunnel 1477 0 9 1151 NCIi 13
45.57.41.1 172.27.5.17 6 443 56518 0/0 0 46 1 tunnel 502 29d 175 38880 11
172.27.13.41 163.70.143.15 6 41482 443 0/0 0 0 12 tunnel 707 bf 15 4478 C 12
172.31.251.71 172.31.248.1 17 8209 8494 0/0 0 0 1 tunnel 2348 9 0 0 FYCI 12
172.31.250.121 172.31.248.1 17 8209 8494 1/0 0 0 0 tunnel 1614 2 0 0 FYCI 14
172.31.251.244 172.31.248.1 17 8209 8222 0/0 0 0 1 tunnel 1131 6 0 0 FYCI 11
172.31.248.1 172.31.249.48 17 8209 8209 0/0 0 0 2 tunnel 2757 14 0 0 FYI 13
172.31.249.206 172.31.248.1 17 8209 8209 0/0 0 0 0 tunnel 1360 8 7 1361 FCI 14
172.31.248.1 172.31.251.151 17 8209 8209 0/0 0 0 20 tunnel 2161 14a 0 0 FYI 15
172.27.0.101 116.119.74.209 6 57215 443 0/0 0 0 1 tunnel 2957 1da 149 23275 Ci 14
172.31.248.240 172.31.248.1 17 8209 8209 0/0 0 0 1 tunnel 2384 2a 22 13122 FCI 14
52.123.137.139 172.27.8.68 6 443 54204 0/0 0 0 1 tunnel 2004 6e92 724 72999 i 12
172.31.248.1 172.31.251.120 17 8515 8209 0/0 0 0 1 tunnel 1463 c 0 0 FYI 12
172.31.251.178 172.31.248.1 17 8209 8209 0/0 0 0 1 tunnel 2391 2d 19 6776 FCI 12
10.147.24.224 165.225.120.44 6 53319 443 0/0 0 24 1 tunnel 2248 174c
PS C:\Users\scripts\Aruba_SessionTracer> And finally the "session logs" are just the output from the device (expected) and the terminal when the program is run (truncated).
P.S. Really appreciate you taking the time to help solve this!
somenath-sinha
commented
1 month ago @ktbyers - also a weird thing I've noticed it - the data from the AMC kind of "stutters", and basically when the period of watch for the next chunk is a bit long is when the output just stops. Maybe that's what's causing the incomplete output?
I'm still not sure why send_command_expect() won't wait for the prompt to come back though - especially since I tried setting the read_timeout to 300 seconds.
Please note that this command takes a lil time to run, just like show tech-support from Cisco's OSs, and takes about 40s to complete.
somenath-sinha
commented
1 month ago I've since figured out:
ktbyers
commented
1 month ago For some reason, even though the entire data is available in the logs, it just chooses the first 9K bytes
or so during printing.Meaning your print(op) isn't printing the entire output?
somenath-sinha
commented
1 month ago Yep - that seems to be the case @ktbyers . I even tried with Paramiko, but basically a similar / same output:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname, username=config.get("connection", "uid"), password=config.get("connection", "upass"))
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(LOOKUP['datapath_table'])
ssh_stderr="".join(ssh_stderr.readlines())
ssh.close()
print(ssh_stderr)
Datapath Session Table Entries
------------------------------
Flags: F - fast age, S - src NAT, N - dest NAT
D - deny, R - redirect, Y - no syn
H - high prio, P - set prio, T - set ToS
C - client, M - mirror, V - VOIP
Q - Real-Time Quality analysis
u - Upstream Real-Time Quality analysis
I - Deep inspect, U - Locally destined
E - Media Deep Inspect, G - media signal
r - Route Nexthop, h - High Value
A - Application Firewall Inspect
i - Session classified on first packet
B - Permanent, O - Openflow
L - Log, o - Openflow config revision mismatched
Source IP or MAC Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge Packets Bytes Flags CPU ID
----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- -------
172.31.251.145 172.31.248.1 17 8209 17110 0/0 0 0 1 tunnel 950 f 0 0 FYCI 12
172.31.248.1 172.31.251.225 17 17110 8209 0/0 0 0 1 tunnel 1455 5 0 0 FYI 14
172.27.9.50 103.228.155.199 6 56397 443 0/0 0 0 1 tunnel 2536 5bd5 9258 715919 C 15
165.225.120.44 10.147.24.224 6 443 61252 0/0 0 16 1 tunnel 2248 15f 34 11049 h 12
172.27.1.46 8.8.4.4 17 47536 53 0/0 0 0 1 tunnel 2862 7 2 148 FCI 13
10.148.124.254 172.27.12.141 6 8080 56753 0/0 0 0 1 tunnel 2536 9 5 661 FSi 11
172.31.248.1 172.31.249.168 17 8494 8209 0/0 0 0 1 tunnel 2405 12 0 0 FYI 15
142.250.192.74 172.27.14.41 6 443 44420 0/0 0 32 11 tunnel 564 b5 26 21873 i 14
[OUTPUT OMITTED]
172.31.249.85 172.31.248.1 17 8209 17110 0/0 0 0 1 tunnel 2626 8 0 0 FYCI 11
172.31.248.178 172.31.248.1 17 8209 8421 0/0 0 0 1 tunnel 2679 8 0 0 FYCI 15
172.31.250.206 172.31.248.1 17 8209 8421 0/0 0 0 1 tunnel 2760 c 0 0 FYCI 15
172.31.248.1 172.31.249.229 17 8209 8209 0/0 0 0 2 tunnel 176 1e 0 0 FYI 13
172.31.251.41 172.31.248.1Netmiko should read all of the data (and print should print out all of the data). Netmiko should generate an exception if it can't read all of the data in read_timeout seconds.
What does your send_command_expect() method look like (since you are wrapping this all in your class)?
somenath-sinha
commented
1 month ago Hey @ktbyers - you Sir, are a genius!
Netmiko is indeed getting all the data, but the print statement's output was being truncated. I tried running the script directly in the terminal - and lo and behold --- I get the entire output!
VSCode just decided that it didn't want to show the entire output, and didn't think it'd be necessary to mention anything about shortening the output!
Thanks a lot mate, @ktbyers -- P.S. Long time user of netmiko, can't even begin to tell you how big a fan I am of your work!
Netmiko truncates the output of
show datapath session tableon Aruba 7210 AMC/WLCs.The full output is supposed to be around 12844 lines long, but
netmiko.send_command()andnetmiko.send_command_expect()both only partially get data till around the 84th line of the output, and then stop. What's even more perplexing is - when debugging is enabled, I can see that the rest of the data does show up in the logs, but not in the output from the commands.Note: Please check https://guides.github.com/features/mastering-markdown/ to see how to properly format your request.
Setup
Netmiko version
(Paste verbatim output from
pip freeze | grep netmikobetween quotes below)Netmiko device_type (if relevant to the issue)
(Paste
device_typebetween quotes below)Steps to Reproduce the Issue
SSH into the device running Aruba OS (WLC/Mobility Controller) and run the
show datapath sessions tablecommand with netmiko. As is general behaviour,no pagingis already in effect, clear from the fact that around the 84th line of output is captured, but it should be 12K+ lines of output.Error Traceback
(Paste the complete traceback of the exception between quotes below)
Relevant Python code
(Please try to essentialize your Python code to the minimum code needed to reproduce the issue) (Paste the code between the quotes below)
datapath_sess_expected.txt Main file:
AMC.py file:
Problem example
Output Log
Debug Log (relevant section only) [lines: 684 - 783]