Unable to access system-view (Comware) in v1.0.0

[steelegbr]

While connecting to Comware 5 switches seems to work successfully, entering system-view (configuration) mode seems to "break" netmiko. Specifically, it appears that the change of prompt from <SWITCH-NAME> to [SWITCH-NAME] causes issues.

The problem presents itself as a script hanging before bombing out with the following error: IOError: Search pattern never detected in send_command_expect:

This seems to be a new issue moving between versions as an older copy of netmiko (v0.5.1) doesn't have the problem.

[ktbyers]

@steelegbr Please post your code and the exception that you get.

[steelegbr]

Here's a snippet of the code I'm running:

        try:
            self.connection = ConnectHandler(**connectionSettings)
            self.connected = True
        except:
            ex = sys.exc_info()[0]
            print 'Failed to connect to network stack %s. Reason: %s.' % (self.ipAddress, ex)
            return False

        # Ensure we obtain system level access

        screenResponse = self.sendCommand('screen-length disable')
        systemResponse = self.sendCommand('system-view')
        systemPrompt = self.connection.find_prompt()

And here's the exception in full:

Traceback (most recent call last):
  File "./switch_remote_job.py", line 117, in <module>
    rabbit_channel_in.start_consuming()
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1681, in start_consuming
    self.connection.process_data_events(time_limit=None)
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 656, in process_data_events
    self._dispatch_channel_events()
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 469, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1310, in _dispatch_events
    evt.body)
  File "./switch_remote_job.py", line 58, in rabbitInbound
    if (not stack.connect()):
  File "/usr1/home/netconf/stack.py", line 68, in connect
    systemResponse = self.sendCommand('system-view')
  File "/usr1/home/netconf/stack.py", line 83, in sendCommand
    return self.connection.send_command(command)
  File "/usr/lib/python2.7/site-packages/netmiko/base_connection.py", line 645, in send_command
    search_pattern))
IOError: Search pattern never detected in send_command_expect: \<SWITCH\-NAME\>

Though I do note that calling the config_mode() function instead of running the system-view command myself does work. Looks like I may have been doing it wrong with the older version of the library.

[steelegbr]

Looks like I may have closed this a little early. While the config_mode() call works, actually configuring a port doesn't work. As an example, here's the code I'm calling (after the above):

self.connection.send_command('interface GigabitEthernet %d/0/%d' % (switch, port))

And the error I run in to:

Traceback (most recent call last):
  File "./switch_remote_job.py", line 117, in <module>
    rabbit_channel_in.start_consuming()
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1681, in start_consuming
    self.connection.process_data_events(time_limit=None)
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 656, in process_data_events
    self._dispatch_channel_events()
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 469, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1310, in _dispatch_events
    evt.body)
  File "./switch_remote_job.py", line 71, in rabbitInbound
    stack.configurePort(switch, port, job['config'])
  File "/usr1/home/netconf/stack.py", line 100, in configurePort
    portSelectResponse = self.sendCommand('interface GigabitEthernet %d/0/%d' % (switch, port))
  File "/usr1/home/netconf/stack.py", line 83, in sendCommand
    return self.connection.send_command(command)
  File "/usr/lib/python2.7/site-packages/netmiko/base_connection.py", line 645, in send_command
    search_pattern))
IOError: Search pattern never detected in send_command_expect: \[SWITCH\-NAME\]

As a note, the prompt changes to [SWITCH-NAME-GigabitEthernet1/0/1]

[ktbyers]

For configuration changes, you should be using the send_config_set() pass in a list of configuration commands.

Notwithstanding that, I don't see why the above failed? Can you send me over what this looks like in the CLI when you do this manually?

[steelegbr]

The send_config_set() command works well. Though, I then run into problems running the "save force" command afterwards.

That said, here's an example of what a simple configuration session could look like:

<STACK-NAME>system-view 
System View: return to User View with Ctrl+Z.
[STACK-NAME]interface GigabitEthernet 1/0/1
[STACK-NAME-GigabitEthernet1/0/1]poe enable
[STACK-NAME-GigabitEthernet1/0/1]return 
<STACK-NAME>save force.......................................................................................................
 The current configuration is saved to the active main board successfully.
Slot 2:
 The current configuration is saved successfully.
Slot 3:
 The current configuration is saved successfully.
Slot 4:
 The current configuration is saved successfully.
Slot 5:
 The current configuration is saved successfully.
 Configuration is saved to device successfully.
<STACK-NAME>

[ktbyers]

@steelegbr Can you post your exception that you get when you try to do the 'save force'?

[ktbyers]

@steelegbr Any update on the error you get with save force?

[steelegbr]

Yes, and apologies for taking a while to get back. Here's the error I run in to:

File "./switch_remote_job.py", line 117, in rabbit_channel_in.start_consuming() File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1681, in start_consuming self.connection.process_data_events(time_limit=None) File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 656, in process_data_events self._dispatch_channel_events() File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 469, in _dispatch_channel_events impl_channel._get_cookie()._dispatch_events() File "/usr/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1310, in _dispatch_events evt.body) File "./switch_remote_job.py", line 76, in rabbitInbound stack.save() File "/usr1/home/netconf/stack.py", line 109, in save self.sendCommand('save force') File "/usr1/home/netconf/stack.py", line 76, in sendCommand return self.connection.send_command(command) File "/usr/lib/python2.7/site-packages/netmiko/base_connection.py", line 645, in send_command search_pattern)) IOError: Search pattern never detected in send_command_expect:

And here's the code being called:

connection.send_command('save force')

[ktbyers]

@steelegbr How long roughly does 'save force' take to execute?

[steelegbr]

On the two stacks I just tested, we're looking at 1:40 and 2:15 for save force to complete and replicate across the switches. I'm guessing we're hitting some sort of timeout then?

[ktbyers]

@steelegbr Yes, you will need to add additional delay.

Try the following:

connection.send_command('save force', delay_factor=2)

If delay_factor=2 doesn't work, then you might need to increase this to 4 or 8.

I think 2 might work, however...from memory I think Netmiko send_command will wait about 90 seconds for a command to complete given the default delays. Making it delay_factor of 2 will double this.

[steelegbr]

Yup, that seems to have done the trick. All good now.