search

carlmontanari / scrapli

Fast, flexible, sync/async, Python 3.7+ screen scraping client specifically for network devices
https://carlmontanari.github.io/scrapli/
MIT License
575 stars 59 forks source link

Some *long* commands failed to exectute for some reason (scrapli.exceptions.ScrapliTimeout: timed out reading from transport) #341

Closed sirmax123 closed 3 months ago

sirmax123 commented 3 months ago

Describe the bug Some long commands are failed to execute with

scrapli.exceptions.ScrapliTimeout: timed out reading from transport

To Reproduce Steps to reproduce the behavior:

  1. Your script Part of script related to the issue: 
    cmd = "mgmt-ip 10.99.{port}.{onuno} 255.255.255.0 vlan 99 priority 0 route 0.0.0.0 0.0.0.0 10.99.{port}.254 host 1".format(
    port=p,
    onuno=new_onu_number
)

    (actual cmd is: mgmt-ip 10.99.32.1 255.255.255.0 vlan 99 priority 0 route 0.0.0.0 0.0.0.0 10.99.32.254 host 1)

Execution part:

def execute_configuration_commands(register_new_onu_commands, device):
    device_config = device["config"]
    conn = Scrapli(**device_config)
    conn.open()

    for cmd in register_new_onu_commands:
       # for debug!
        print(cmd)
        print(len(cmd))
        if (cmd.startswith("pon-o")):
            conn.comms_prompt_pattern = "(^.*#$)"
        reply_from_olt = conn.send_command(cmd, failed_when_contains=["%Error "])
        print("Reply: {}".format(reply_from_olt.result))

Device configuration

{
    "devices": [
        {
            "name": "ZTE C320",
            "config": {
                "host": "172.31.0.249",
                "port": 23,
                "auth_username": "user",
                "auth_password": "password",
                "platform": "cisco_iosxe",
                "transport": "telnet",
                "timeout_ops": 120,
                "timeout_socket": 30
            }
        }
    ]
}
  1. What you're connecting to (vendor, platform, version) 
    #show version-running
PhyLoc  FileType   VerType    VerTag            BuildTime           VerLength
-------------------------------------------------------------------------------
1/1/1   GTXK       MVR        V2.1.0            2017-07-03 00:28:55 7789380
1/1/1   GTXK       BT         V4.0.14           2017-09-04  0:52:45 524288
1/1/2   GTXK       MVR        V2.1.0            2017-07-03 00:28:55 7789380
1/1/2   GTXK       BT         V4.0.14           2017-09-04  0:52:45 524288
1/1/4   SMXA       MVR        V2.1.0            2017-01-17 01:04:45 24647784
1/1/4   SMXA       BT         V4.0.5            2014-12-10  9:13:26 524288
1/1/4   SMXA       FW         V2.1.0            2017-06-23 03:42:13 1720296
  2. Anything else relevant Shorter command executed w/o error, looks like 55 chars is max :(

Part of script output:

conf t
6
Reply: %Info 20272: Enter configuration commands, one per line. End with CTRL/Z.
interface gpon-olt_1/2/16
25
Reply:
onu 1 type GPON_ONE_ETH_PORT sn ZXIC0656FEBF
44
Reply: .[Successful]
exit
4
Reply:
no interface gpon-onu_1/2/16:1
30
Reply:
interface gpon-onu_1/2/16:1
27
Reply:
description " "
15
Reply:
sn-bind enable sn
<working commands skipped>
Reply:
mgmt-ip 10.99.32.1 255.255.255.0 vlan 99 priority 0 route 0.0.0.0 0.0.0.0 10.99.32.254 host 1
93
Traceback (most recent call last):
<traceback described below>

Expected behavior

Stack Trace

Traceback (most recent call last):
  File "./zte_configure_onu.py", line 289, in <module>
    execute_configuration_commands(register_new_onu_commands, device)
  File "./zte_configure_onu.py", line 229, in execute_configuration_commands
    reply_from_olt = conn.send_command(cmd, failed_when_contains=["%Error "])
  File "/usr/local/lib/python3.8/dist-packages/scrapli/driver/network/sync_driver.py", line 249, in send_command
    response: Response = super().send_command(
  File "/usr/local/lib/python3.8/dist-packages/scrapli/driver/generic/sync_driver.py", line 177, in send_command
    response: Response = self._send_command(
  File "/usr/local/lib/python3.8/dist-packages/scrapli/decorators.py", line 306, in decorate
    result = wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/scrapli/driver/generic/sync_driver.py", line 144, in _send_command
    raw_response, processed_response = self.channel.send_input(
  File "/usr/local/lib/python3.8/dist-packages/scrapli/decorators.py", line 223, in decorate
    return _multiprocessing_timeout(
  File "/usr/local/lib/python3.8/dist-packages/scrapli/decorators.py", line 114, in _multiprocessing_timeout
    return future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/scrapli/channel/sync_channel.py", line 488, in send_input
    _buf_until_input = self._read_until_input(channel_input=bytes_channel_input)
  File "/usr/local/lib/python3.8/dist-packages/scrapli/channel/sync_channel.py", line 105, in _read_until_input
    buf += self.read()
  File "/usr/local/lib/python3.8/dist-packages/scrapli/channel/sync_channel.py", line 69, in read
    buf = self.transport.read()
  File "/usr/local/lib/python3.8/dist-packages/scrapli/decorators.py", line 223, in decorate
    return _multiprocessing_timeout(
  File "/usr/local/lib/python3.8/dist-packages/scrapli/decorators.py", line 109, in _multiprocessing_timeout
    return _handle_timeout(
  File "/usr/local/lib/python3.8/dist-packages/scrapli/decorators.py", line 141, in _handle_timeout
    raise ScrapliTimeout(message)
scrapli.exceptions.ScrapliTimeout: timed out reading from transport

Screenshots If applicable, add screenshots to help explain your problem, but do note that formatted text is much preferred over screenshots!

OS (please complete the following information):

sirmax123 commented 3 months ago

Reproduced with

pip3 list | grep scrapli
scrapli                2024.1.30
carlmontanari commented 3 months ago

guessing this is not really an iosxe device ya? if thats true then you need to either send the command without using the send_command helper stuff (meaning you need to just write using the transport directly, like conn.transport.write) or you need to disable terminal width setting for your platform (or make it really big) like all the other platforms do. you can also try to set comms_roughly_match_inputs -> true -- this thing basically just makes sure all the chars of your input are found in order in the echoed output from your device, in theory this means that when your device sends backspace ansi control chars (or whatever its doing) we should still successfully "read" your input back off the channel and be able to continue on.

sirmax123 commented 3 months ago

@carlmontanari

Thank you for your reply,

guessing this is not really an iosxe device ya?

yes, this is ZTE C320 OLT (Pon network switch), not Cisco IOS-XE

like conn.transport.write

yes, this works for me

I do not know how to see what device sends back, there are no colors in the telnet session output, but it looks like it sends \n\r instead of \n, anyway my issue is solved

I understand that it is not possible to test scrapli with any no-name Chinese switch (or I'd better say "piece of shit" instead of "switch" about this device) so I think we can close the issue as a non-relevant to the scrapli.

The only question I have is: Is it possible to enable "deep logging" to see what was sent to the device and what was got back before parsing?

Thank you, Max

carlmontanari commented 3 months ago

ah I forgot one thing, you can prolly also use this (passed as kwarg to your driver.send_command). its also worth testing that roughly match setting I mentioned the other day.

in general though if you want to use the transport.write you then are going to have to be fully managing the read loop yourself basically. so you can check out the channel package and see how things are look in there and replicate to your need -- basically you just need to also conn.transport.read in a loop until you have consumed what you wrote into the channel and also what the response from the device was and all the way up to the prompt.

I would bet that there is a way to set terminal width on your device though, so I would defo look into that more. I dont think ive seen a device that doesn't let ya set that but who knows!

re the logging q: logging at debug level logs all output right after we read it from the transport, so that should get you what you need I imagine.

anyway, good luck :D