replace_confing on Nexus3k not working

[pjw9]

Objective:

• replace config on Nexus3064 using napalm module with nxos-ssh driver (ansible with jinja2 templates).

Issue:

• when I launch my ansible playbook with 'replace_config' on, it returns errors (below)

Troubleshooting with napalm CLI tool shows that my config gets transferred onto the box properly, then using config replace returns error, however config merge works fine:

napalm --vendor nxos_ssh --user user --password 'mypw' HOST configure roles/nxos-3k/files/full_config_test --strategy merge --dry-run

The above works fine and returns the "diff".

napalm --vendor nxos_ssh --user user --password 'mypw' HOST configure roles/nxos-3k/files/full_config_test --strategy replace --dry-run

The above fails and returns error:

================= Traceback =================

Traceback (most recent call last): File "/usr/local/bin/napalm", line 10, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/napalm/base/clitools/cl_napalm.py", line 312, in main run_tests(args) File "/usr/local/lib/python3.7/site-packages/napalm/base/clitools/cl_napalm.py", line 297, in run_tests configuration_change(device, args.config_file, args.strategy, args.dry_run) File "/usr/local/lib/python3.7/site-packages/napalm/base/clitools/cl_napalm.py", line 245, in configuration_change diff = call_compare_config(device) File "/usr/local/lib/python3.7/site-packages/napalm/base/clitools/cl_napalm.py", line 31, in wrapper r = func(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/napalm/base/clitools/cl_napalm.py", line 226, in call_compare_config diff = device.compare_config(args, **kwargs) File "/usr/local/lib/python3.7/site-packages/napalm/nxos/nxos.py", line 209, in compare_config diff = self._get_diff() File "/usr/local/lib/python3.7/site-packages/napalm/nxos/nxos.py", line 191, in _get_diff diff_out.split("Generating Rollback Patch")[1] IndexError: list index out of range

When I log in to the host I can see my config file that I generate gets transferred onto the Nexus host properly:

I can verify my conf file by issuing: # show file bootflash:candidate_config.txt

I can even apply it manually with: configure replace bootflash:candidate_config.txt

At this point I think this might be a bug. I would need assistance to troubleshoot further.

Regards PJW

[ktbyers]

If you execute this manually what do you see?

show diff rollback-patch file sot_file file bootflash:candidate_config.txt

[pjw9]

# show diff rollback-patch file bootflash:sot_file file bootflash:candidate_config.txt
  ERROR: Rollback patch computation failed due to either of the following reasons
  1. The destination checkpoint file was not created using CLI 
  2. The destination checkpoint file was created using nxos version earlier than 4.2.1

[ktbyers]

Okay...that looks like your issue there. Did you look into what it reported here more?

[pjw9]

It doesn't say much to me unfortunately. I looked into both, sot_file and candidate_config.txt... Does that mean my candidate_config is not in a desired format and is not understood, hence can't get the diffs? Couldn't find anything in a code.

[ktbyers]

Is your candidate_config file an NX-OS checkpoint file? That is the most common issue here.

It is NOT just a running-config...it needs to be a checkpoint file.

[pjw9]

That makes perfect sense. Going to test it fully.

[pjw9]

Thanks Kirk, I got _getdiffs and _configreplace working, however now getting "Unable to find prompt" error. Seems it times out. Would changing/increasing delay value or some other timers fix it? Is that a right track?

The full traceback is:
  File "/usr/local/lib/python3.7/site-packages/napalm/nxos/nxos.py", line 229, in commit_config
    self._netmiko_device.set_base_prompt()
  File "/usr/local/lib/python3.7/site-packages/netmiko/base_connection.py", line 1066, in set_base_prompt
    prompt = self.find_prompt(delay_factor=delay_factor)
  File "/usr/local/lib/python3.7/site-packages/netmiko/base_connection.py", line 1107, in find_prompt
    raise ValueError("Unable to find prompt: {}".format(prompt))

fatal: [-> localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "archive_file": null,
            "candidate_file": null,
            "commit_changes": true,
            "config": null,
            "config_file": "/tmp/",
            "dev_os": "nx-os_ssh",
            "diff_file": "../diffs.txt",
            "get_diffs": true,
            "hostname": "",
            "optional_args": null,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "provider": null,
            "replace_config": true,
            "timeout": 60,
            "username": []
        }
    },
    "msg": "cannot install config: Unable to find prompt: "
}

[ktbyers]

Try adding:

  optional_args:
      global_delay_factor: 4

And see if that works.

Let me know.

[pjw9]

I've added the above, no joy, getting the same error.

[ktbyers]

Is there anything unusual about the device's prompt?

Is this a real device or a virtual device?

What are you changing in your set of configuration changes?

[pjw9]

Nothing unusual with the prompt that I noticed (#). This is a real device, nexus 3064. The objective is to deploy the full config (config built using jinja2 templates).

[pjw9]

Upgraded netmiko to v3.0.0. But getting similar thing (with global_delay still on). "msg": "cannot install config: Router prompt not found: '.Executing Rollback Patch'"

[ktbyers]

Can you show me the failing devices full-prompt/hostname?

Can you show me what happens when you do the following from the CLI (include leading and trailing prompts):

rollback running-config file candidate_config.txt

Note, make sure your candidate_config.txt file is correct before you do this. Also make sure you only do this on a test network device (as that command will fully replace the running-config with the candidate_config.txt file).

[pjw9]

SW-02#
SW-02# rollback running-config file bootflash:candidate_config.txt
Collecting Running-Config
Generating Rollback patch for switch profile
Rollback Patch is Empty
Note: Applying config parallelly may fail Rollback verification
Collecting Running-Config
#Generating Rollback Patch
Executing Rollback Patch
..Generating Running-config for verification
Generating Patch for verification
Verification failed, Rolling back to previous configuration
Collecting Running-Config
Cleaning up switch-profile buffer
Generating Rollback patch for switch profile
Executing Rollback patch for switch profiles. WARNING - This will change the configuration of switch profiles and will also affect any peers if configured
Collecting Running-Config
#Generating Rollback Patch
.Executing Rollback Patch
.....Generating Running-config for verification
Generating Patch for verification

Rollback failed. Use 'show rollback log verify' or 'show rollback log exec' to see reasons for failure

ERROR: Rollback Patch is Empty

SW-02# sh rollback log verify
Operation            : Rollback to Checkpoint File
Checkpoint file name : /candidate_config.txt
Scheme               : bootflash
Rollback done By     :user
Rollback mode        : atomic
Verbose              : disabled
Status               : Failed

Verification patch contains the following command:
---------------------------------------------------
!!
logging console
logging module
logging monitor
interface loopback0
  no shutdown
  exit
interface Ethernet1/52/4
  logging event port trunk-status
  logging event port link-status
  switchport
  no shutdown
[...]
no feature ssh
no feature nxapi
Status               : Success

Verification patch contains the following commands:
---------------------------------------------------
!!
! No changes

SW-02# sh rollback log exec
Operation            : Rollback to Checkpoint File
Checkpoint file name : /candidate_config.txt
Scheme               : bootflash
Rollback done By     :
Rollback mode        : atomic
Verbose              : disabled
`config sync `
Rollback Status      : Failed
Restoring Previous Config : Sucess

Executing Patch:
----------------
`config t `
`no logging console`
`no logging module`
`no logging monitor`
`no logging logfile messages 5 size 4194304`
`interface loopback0`
`shutdown`
`exit`
`interface Ethernet1/52/4`
[...]
`interface Ethernet1/46`
`switchport`
Syntax error while parsing 'switchport'
[...]

The below shows me the expected diff: sh diff rollback-patch running-config file bootflash:candidate_config.txt

The below shows the entire candidate-config fine: sh file bootflash:candidate_config.txt

Rollback from file (candidate-config) fails. Rollback from checkpoint succeeds:

SW-02# rollback running-config checkpoint CHECKPOINT-1
Collecting Running-Config
Generating Rollback patch for switch profile
Rollback Patch is Empty
Note: Applying config parallelly may fail Rollback verification
Collecting Running-Config
#Generating Rollback Patch
Executing Rollback Patch
Generating Running-config for verification
Generating Patch for verification

Rollback completed successfully.

Note: My candidate-config is built base on (carbon-copy) "CHECKPOINT-1" that's been created on the device itself.

[ktbyers]

I am going to close this issue here. Please re-open in the NAPALM project and reference/link to this issue from there.

I say this as this issue looks like a more generic napalm issue (instead of pertaining to napalm-ansible).

[Ranjith5080]

hey @pjw9 , how are you able to get the config replace working. I am kind of stuck with same error. I am using napalm for my automation project to rollback changes on nxos(bios: 9.3) and get this error.

'ERROR: Rollback patch computation failed due to the following reason(s)\nThe checkpoint file was not created using checkpoint CLI\n'

[ktbyers]

@Ranjith5080 Are you using a checkpoint as the starting point. NX-OS + NAPALM requires a checkpoint file for configuration replace.