Config Parsing to Find & Replace

[newscriptkid]

Hi Gregg,

This is the other project I am working on after which i will just be doing my own practice and refactoring all the codes i have done so far. Any help with this will be appreciated.

Summary: I have a 6509 with about 320 sub-interfaces and about 160 cdp nei switches. The vlan number on the 6509 and the associated cdp nei will need to be changed. I have to do this on 4 or 5 6509's and couple of 100 cdp nei switches hanging off the each 6509.

Aim1: Replace the 4 digit vlans with a 3 digit vlan id on 6509 from a range of vlan for e.g. 300-700. So the 1st 4 digit vlan id will get 300 vlan id and the next 4 digit will get 301 and so on...

Aim2: This is tricky - the vlan replaced on the 6509 will need to be updated on the cdp nei switch also, for e.g. if on the 6509 vlan 3000 was replaced with vlan 301 then on the cdp nei switch for vlan 3000 will also need to be changed to 300.

I'm not sure how to best approach the method of finding the 4 digit vlan configuration, but i thought of either doing show run | inc or the below method:

Below is the final output i should get for each sub-interface once the script runs successfully, the script then will use this to find and replace 4 digit vlans.

show run | sec GigabitEthernet1/19

interface GigabitEthernet1/19.1333
 description *** xxxxxx ***
 encapsulation dot1Q 1333
 ip vrf forwarding corp
 ip address 10.10.10.1 255.255.255.252
 ip flow ingress
 ip flow egress

interface GigabitEthernet1/19.2333
 description *** xxxxxxx ***
 encapsulation dot1Q 2333
 ip address 10.10.10.5 255.255.255.252
 ip flow ingress
 ip flow egress

interface GigabitEthernet1/19.3000
 description *** xxxxxx ***
 encapsulation dot1Q 3000
 ip vrf forwarding BGFL
 ip address 192.168.10.1 255.255.255.252
 ip flow ingress
 ip flow egress

 passive-interface GigabitEthernet1/19.2333
 passive-interface GigabitEthernet1/19.3000

ip route vrf xxx 10.x.x.x 255.255.252.0 GigabitEthernet1/19.3000 192.168.10.2
ip route vrf xxx 10.x.x.x 255.255.255.128 GigabitEthernet1/19.1333 10.10.10.2
ip route vrf xxx 10.x.x.x 255.255.255.128 GigabitEthernet1/19.1333 10.10.10.2

What needs to be done: Step-1 - finds all the sub-interfaces with 4 digit vlans (eg. vlan 3000) and any other configs parameters using these vlans (such as the passive interface and ip route commands above). Step-2 - Replace the 4 digit vlan with a 3 digit vlan from the range specified (eg. 300-700) in every place the vlan number is present, for e.g. Step-3 - present the new config in notepad to user and give them option to let script paste it or not.

For step-1, i have managed to get the below together:

#!/usr/bin/python
import sys
import os
import time
import requests
from sys import exit
import fileinput
import optparse
import json
import netmiko

with open('S824-step1.txt', 'w') as fd:  # open file to write to
    old_stdout = sys.stdout  # send output
    sys.stdout = fd  # output goes to file variable 'fd' which is actually 'ip_int_bri_old.txt'
    with open('S824 intbrief.txt', 'r') as readfile:  # open the file with the list of interfaces as readonly
        lines = (line.split(' ') for line in readfile)  # read the lines in the file and split the lines from the space
        intf = (type[0] for type in lines)  # specify to read from 1st column
        for x in intf:
            x = x.replace('Gi', 'GigabitEthernet')  # find and replace
            print x.split('.').pop(0).strip()  # split from '.' and remove the last word and strip any white lines

with open('S824-duplicate.txt', 'w') as dup:
    lines_seen = set()  # holds lines already seen
    for line in open('S824-step1.txt', 'r'):  # read this file
        if line not in lines_seen:  # not a duplicate
            dup.write(line)
            lines_seen.add(line)

with open('S824final_output.txt', 'w') as final:
    output = open('S824-duplicate.txt', 'r')
    for y in output:
        interface = y.split(' ')[0]
        final.write('show run | sec ' + interface + '\n')

The script runs successfully but the output is not correct.

After running the above script I have the following problem:

The output for each interface should be "GigabitEthernetx/x" but its printing it as "GigabitEthernetgabitEthernetx/x" it also prints vlans and loopback interfaces which i dont need.

Below are all the outputs pre/post running script.

small portion of the original "show ip int brie" output:

6509#sh ip int brie
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1/1     unassigned      YES unset  up                    up
GigabitEthernet1/2     unassigned      YES NVRAM  administratively down down
GigabitEthernet1/3     unassigned      YES NVRAM  administratively down down
GigabitEthernet1/4     unassigned      YES NVRAM  administratively down down
GigabitEthernet1/5     unassigned      YES NVRAM  administratively down down
GigabitEthernet1/6     unassigned      YES NVRAM  up                    up
GigabitEthernet1/6.419 10.1.1.1      YES NVRAM  up                    up
GigabitEthernet1/6.523 10.2.1.1     YES NVRAM  up                    up
Gi1/6.1795             10.3.1.1   YES NVRAM  up                    up
Gi1/6.2456             10.4.1.1    YES NVRAM  up                    up
Gi1/6.3531             10.5.1.1  YES NVRAM  up                    up
GigabitEthernet1/7     unassigned      YES manual up                    up
Gi1/7.1297             unassigned      YES NVRAM  deleted               down
Gi1/7.1718             unassigned      YES NVRAM  deleted               down
Gi1/7.2110             10.6.1.1    YES NVRAM  up                    up
Gi1/7.3560             10.7.1.1  YES manual up                    up
GigabitEthernet1/8     unassigned      YES NVRAM  up                    up
GigabitEthernet1/8.553 10.8.1.1    YES NVRAM  up                    up
Gi1/8.1292             10.9.1.1  YES NVRAM  up                    up
Gi1/8.2106             10.10.1.1    YES NVRAM  up                    up
GigabitEthernet1/9     unassigned      YES NVRAM  up                    up
Gi1/9.2962             10.11.1.1    YES NVRAM  up                    up
Gi1/9.4059             10.12.1.1   YES NVRAM  up                    up
GigabitEthernet1/10    unassigned      YES NVRAM  up                    up
Gi1/10.2953            10.13.1.1    YES NVRAM  up                    up
Gi1/10.4019            10.14.1.1    YES NVRAM  up                    up
GigabitEthernet1/11    unassigned      YES NVRAM  administratively down down
Group-Async9           unassigned      YES NVRAM  down                  down
Loopback0              10.15.1.1      YES NVRAM  up                    up
Loopback1              10.16.9.1       YES NVRAM  up                    up
Port-channel4          unassigned      YES manual down                  down
Port-channel4.70       unassigned      YES unset  down                  down
Port-channel7          10.17.1.1  YES NVRAM  up                    up
Port-channel20         10.18.1.1     YES NVRAM  up                    up
Port-channel30         10.19.1.1     YES NVRAM  up                    up
Port-channel32         10.20.1.1    YES NVRAM  up                    up
Port-channel34         10.21.1.1    YES NVRAM  up                    up
Port-channel37         unassigned      YES unset  up                    up
Tunnel0                10.22.1.1      YES unset  up                    up
Vlan1                  unassigned      YES NVRAM  administratively down down
Vlan50                 10.23.1.177     YES manual up                    up
Vlan72

This is some of the initial output file "S824-step1":

6509#sh
Interface
GigabitEthernetgabitEthernet1/1
GigabitEthernetgabitEthernet1/2
GigabitEthernetgabitEthernet1/3
GigabitEthernetgabitEthernet1/4
GigabitEthernetgabitEthernet1/5
GigabitEthernetgabitEthernet1/6
GigabitEthernetgabitEthernet1/6
GigabitEthernetgabitEthernet1/6
GigabitEthernet1/6
GigabitEthernet1/6
GigabitEthernet1/6
GigabitEthernetgabitEthernet1/7
GigabitEthernet1/7
GigabitEthernet1/7
GigabitEthernet1/7
GigabitEthernet1/7
GigabitEthernetgabitEthernet1/8
GigabitEthernetgabitEthernet1/8
GigabitEthernet1/8
GigabitEthernet1/8
GigabitEthernetgabitEthernet1/9
GigabitEthernet1/9
GigabitEthernet1/9
GigabitEthernetgabitEthernet1/10
GigabitEthernet1/10
GigabitEthernet1/10
Group-Async9
Loopback0
Loopback1
Port-channel4
Port-channel4
Port-channel7
Port-channel20
Port-channel30
Port-channel32
Port-channel34
Port-channel37
Tunnel0
Vlan1
Vlan50
Vlan72
Vlan311
Vlan312

Below is the output after running the part of the script to remove duplicate entries:

6509#sh
Interface
GigabitEthernetgabitEthernet1/1
GigabitEthernetgabitEthernet1/2
GigabitEthernetgabitEthernet1/3
GigabitEthernetgabitEthernet1/4
GigabitEthernetgabitEthernet1/5
GigabitEthernetgabitEthernet1/6
GigabitEthernet1/6
GigabitEthernetgabitEthernet1/7
GigabitEthernet1/7
GigabitEthernetgabitEthernet1/8
GigabitEthernet1/8
GigabitEthernetgabitEthernet1/9
GigabitEthernet1/9
GigabitEthernetgabitEthernet1/10
GigabitEthernet1/10
Group-Async9
Loopback0
Loopback1
Port-channel4
Port-channel7
Port-channel20
Port-channel30
Port-channel32
Port-channel34
Port-channel37
Tunnel0
Vlan1
Vlan50
Vlan72

This is the final output - these are the commands i will be running on the switch.

show run | sec 

show run | sec 6509#sh

show run | sec Interface

show run | sec GigabitEthernetgabitEthernet1/1

show run | sec GigabitEthernetgabitEthernet1/2

show run | sec GigabitEthernetgabitEthernet1/3

show run | sec GigabitEthernetgabitEthernet1/4

show run | sec GigabitEthernetgabitEthernet1/5

show run | sec GigabitEthernetgabitEthernet1/6

show run | sec GigabitEthernet1/6

show run | sec GigabitEthernetgabitEthernet1/7

show run | sec GigabitEthernet1/7

show run | sec GigabitEthernetgabitEthernet1/8

show run | sec GigabitEthernet1/8

show run | sec GigabitEthernetgabitEthernet1/9

show run | sec GigabitEthernet1/9

show run | sec GigabitEthernetgabitEthernet1/10

show run | sec GigabitEthernet1/10

show run | sec Group-Async9

show run | sec Loopback0

show run | sec Loopback1

show run | sec Port-channel4

show run | sec Port-channel7

show run | sec Port-channel20

show run | sec Port-channel30

show run | sec Port-channel32

show run | sec Port-channel34

show run | sec Port-channel37

show run | sec Tunnel0

show run | sec Vlan1

show run | sec Vlan50

show run | sec Vlan72

Once i get the correct output can you please guide me on how i can search for and replace the 4 digit vlans, I know how to use the '.replace' command but for that I have to specify the words, i dont know how to let it loop the file and find itself.

Any help appreciated.

Thanks again

[grelleum]

Here are my initial, high level, thoughts on how to go about this.

I suggest that you do a three step approach:

1. collect all the configurations
2. generate the new configs based on old configs
3. push the new configs.

Step 1 and 3 should be fairly trivial to accomplish, and probably best done with scp copying.

Regarding step 3, you could simplify things if you are able to reboot the devices, such that you only need to generate the final config, copy that to startup-config and reload. That might not be an option so if you need to do this without reloading then the new config would have to first generate the configuration to remove the old sub-interfaces, and then to create the new sub-interfaces.

You could also split step 2 into two parts. First, create a simple script that collects all of the vlans that need to be changed and creates a mapping to the new vlan number. This simple script would read in the main 6509 config and output a file in json, or any format, that creates a mapping that goes from old vlan to new vlan:

1333, 300
2333, 301
3000, 302

Once this mapping is completed, you don't have to go back and change that code, you can focus on the next task.

Then you would create a script, that reads each config file plus the vlan mapping file. This script would do the bulk of the work in creating the new config.

The thing is this - what I'm doing above is breaking a big task into several smaller tasks, that are each, much simpler.

Future tasks can each take advantage of the config grab / config push scripts.

With the main script that reads the config and the mapping, start small and build it up. Start by only letting it read one config file, and have it just identify the interfaces that need changing, and make sure that works before continuing. Then have it generate the config that removes the existing sub-interface. this way you build the script slowly and keep adding the parts that are needed.

By reading the config from a file and saving the config to a file, you can do all the development on your laptop and never have to touch the devices until you have it working properly.

[newscriptkid]

Thanks Gregg for the breakdown, fortunately I will be applying the config on new 6509 supervisors in a test chassis and when the upgrade date arrives i have to remove the old supervisor and insert the new one so I wont need to hopefully remove any old config as what i put on the new supervisors in the test chassis will be the final config.

I will crack on with your advice above,

Thanks again

[grelleum]

Here's some code that reads the config and creates a list of the existing vlans, then creates a "map" (a dictionary) of the old vlan to new vlan.

>>> config = """
... show run | sec GigabitEthernet1/19
... 
... interface GigabitEthernet1/19.1333
...  description *** xxxxxx ***
...  encapsulation dot1Q 1333
...  ip vrf forwarding corp
...  ip address 10.10.10.1 255.255.255.252
...  ip flow ingress
...  ip flow egress
... 
... 
... interface GigabitEthernet1/19.2333
...  description *** xxxxxxx ***
...  encapsulation dot1Q 2333
...  ip address 10.10.10.5 255.255.255.252
...  ip flow ingress
...  ip flow egress
... 
... 
... interface GigabitEthernet1/19.3000
...  description *** xxxxxx ***
...  encapsulation dot1Q 3000
...  ip vrf forwarding BGFL
...  ip address 192.168.10.1 255.255.255.252
...  ip flow ingress
...  ip flow egress
... 
...  passive-interface GigabitEthernet1/19.2333
...  passive-interface GigabitEthernet1/19.3000
... 
... ip route vrf xxx 10.x.x.x 255.255.252.0 GigabitEthernet1/19.3000 192.168.10.2
... ip route vrf xxx 10.x.x.x 255.255.255.128 GigabitEthernet1/19.1333 10.10.10.2
... ip route vrf xxx 10.x.x.x 255.255.255.128 GigabitEthernet1/19.1333 10.10.10.2
... """
>>> 
>>> existing_vlans = []
>>> for line in config.splitlines():
...     if line.startswith('interface') and '.' in line:
...         vlan = line.split('.')[-1]  # split line on dot and grab only last item
...         if len(vlan) = 4:
...             existing_vlans.append(vlan)
... 
>>> 
>>> existing_vlans
['1333', '2333', '3000']
>>> 
>>> vlan_map = dict(zip(existing_vlans, range(300, 700)))
>>> 
>>> vlan_map
{'1333': 300, '2333': 301, '3000': 302}
>>> 
>>> config = """
... show run | sec GigabitEthernet1/19
... 
... interface GigabitEthernet1/19.1333
...  description *** xxxxxx ***
...  encapsulation dot1Q 1333
...  ip vrf forwarding corp
...  ip address 10.10.10.1 255.255.255.252
...  ip flow ingress
...  ip flow egress
... 
... 
... interface GigabitEthernet1/19.2333
...  description *** xxxxxxx ***
...  encapsulation dot1Q 2333
...  ip address 10.10.10.5 255.255.255.252
...  ip flow ingress
...  ip flow egress
... 
... 
... interface GigabitEthernet1/19.3000
...  description *** xxxxxx ***
...  encapsulation dot1Q 3000
...  ip vrf forwarding BGFL
...  ip address 192.168.10.1 255.255.255.252
...  ip flow ingress
...  ip flow egress
... 
...  passive-interface GigabitEthernet1/19.2333
...  passive-interface GigabitEthernet1/19.3000
... 
... ip route vrf xxx 10.x.x.x 255.255.252.0 GigabitEthernet1/19.3000 192.168.10.2
... ip route vrf xxx 10.x.x.x 255.255.255.128 GigabitEthernet1/19.1333 10.10.10.2
... ip route vrf xxx 10.x.x.x 255.255.255.128 GigabitEthernet1/19.1333 10.10.10.2
... """
>>> 
>>> existing_vlans = []
>>> for line in config.splitlines():
...     if line.startswith('interface') and '.' in line:
...         vlan = line.split('.')[-1]  # split line on dot and grab only last item
...         existing_vlans.append(vlan)
... 
>>> 
>>> existing_vlans
['1333', '2333', '3000']
>>> 
>>> vlan_map = dict(zip(existing_vlans, range(300, 700)))
>>> 
>>> vlan_map
{'1333': 300, '2333': 301, '3000': 302}
>>> 

A couple of notes: this was done in the Python interactive shell (known as the "REPL"). I put the config into a "docstring" - so the variable "config" is a long, multi-line string. The way I loop over the string is by calling the .splitlines() method, which produces a list of individual lines.

I first check that the line startswith interface and that it also has a dot, which tells me it is a subinterface. All other lines will be ignored.
I could have instead matched the lines with "encapsulation dot1Q" as follows:

    if ' encapsulation dot1Q' in line:
        vlan = line.split()[-1]

Notice in this version I do not split on the dot - there is none, rather I split the line on the whitespace. Each vlan of length 4 gets added to the list: existing_vlans

I use the range(300, 700) function to create a series of numbers starting from 300 and going up to 699. The zip() function takes two "sequences" and returns pairs of values, based on those sequences. So given "1333, 2333, 3000" and "300, 301, 302, 303, 304, 305, 306 ..... 699", zip returns:

  (1333, 300),
  (2333, 301),
  (3000, 302)

wrapping that in the dictionary constructor dict, we get a dictionay whose keywords are the original vlan and the corresponding value is the replacement vlan number.

>>> list(zip(existing_vlans, range(300, 700)))
[('1333', 300), ('2333', 301), ('3000', 302)]

>>> dict(zip(existing_vlans, range(300, 700)))
{'1333': 300, '2333': 301, '3000': 302}

The vlans in the existing_vlans list are all of type string (they have quotes around them). The range function produces type integer, so when using these integers, you might need to convert them to string.

>>> list(zip(existing_vlans, range(300, 700)))
[('1333', 300), ('2333', 301), ('3000', 302)]

[grelleum]

Once you create a vlan_map, it is important that you do not lose your mappings, so you would want to save it to a file. Since the map is stored in a dictionary, the easiest way to save it and later retrieve it would be in json format.

>>> import json
>>> with open('vlan_map.json', 'wt') as f:
...     json.dump(vlan_map, f)
... 
>>> 

Then, when you process your config files you can read the vlan map in first:

>>> import json
>>> with open('vlan_map.json', 'rt') as f:
...     vlan_map = json.load(f)
... 
>>> vlan_map
{'1333': 300, '2333': 301, '3000': 302}

[grelleum]

Here is something that can read in the vlan_map file as well as a single config file and generate a new config file. Once validated and expanded to cover anything else that is required, the main code could be put into a loop to mass create new config files.

import json
import sys

config_filename = sys.argv[1]
new_config_filename = config_filename + '.NEW'

def find_vlan(line):
    if line.startswith('interface ') or  line.startswith(' passive-interface '):
        return line.split('.')[-1]
    if line.startswith(' encapsulation dot1Q '):
        return line.split()[-1]
    if line.startswith('ip route vrf '):
        interface = line.split()[6]
        return interface.split('.')[-1]

def hanlde_replacement(line, vlan):
    if vlan in vlan_map:
        replacement = str(vlan_map[vlan])
        return line.replace(vlan, replacement)

with open('vlan_map.json', 'rt') as f:
    vlan_map = json.load(f)

with open(config_filename, 'rt') as f:
    config = f.readlines()

with open(new_config_filename, 'wt') as f:
    for line in config:
        line = line.rstrip()  # remove '\n'
        vlan = find_vlan(line)
        new_line = hanlde_replacement(line, vlan)
        if new_line:
            f.write(new_line + '\n')
        else:
            f.write(line + '\n')

Somethings to note about this code - any function that does not explicitly return a value will always return the object None. find_vlan and handle_replacement both return values only IF conditions are met, otherwise they will return None. If handle_replacement returns None, then the if new_line: conditional will be false.

[newscriptkid]

WOW!!! I honestly feel like im speechless, I can't believe you've taken so much time out to get this complicated code in a bite size easy to understand format.

This code is awesome! I was being pushed for speeding up the upgrade and now with this i can wipe out 10-15 days off the project planner!

I dont know how to thank you enough,

Can you please provide an address where i can send some chocolates to.

I honestly owe you big time!

Thanks ever sooo much, Shah

[newscriptkid]

Hi Gregg,

Just wanted to let you know that unfortunately the plans have changed and no longer replacing the Sub-interface 4 digit id's with 3 digit id's as due to the limitation on the 6509 plans are now to convert the sub-int to svi's. So thank you again and your code i will be using for many other tasks.

[grelleum]

Sure - good luck with your Python adventures. Thank you for the kind offer of Chocolates, but that is not necessary and stay in touch. You can connect with me on LinkedIn if you like.