Chapter 10. Network Data Analytics

[anitsh]

Learning Objectives

By the end of this chapter, you should be able to:

        Discover available network automation tools which can help automate repetitive tasks.
        Review the architecture of network automation tools and how they work.

[anitsh]

Introduction to Network Automation

Next, we will explore the network automation tools that can help you build automated workflows to execute tasks in a network. Network automation works on both next-generation SDN and legacy networks. Even if you have an aging network equipment, you are still able to use automation tools to make faster and reliable changes.

Automation tools that we are exploring in this chapter are:

        Ansible
        Puppet
        Chef.

All of these tools have been used by system engineers and application professionals to automate the deployment and provisioning of servers and applications. Ensuring applications are installed properly on the servers, dependencies are met, components meet a predefined version and correct patches are applied.

Automation tools have a modular architecture, which allows them to control and program almost any device or application. In recent years, networking plugins have been developed for automation tools, to allow them to control a network device or a Virtual Network Function as well. The plugins are device-specific (for example, to control a Cisco IOS router, you will need to load the Cisco IOS plugin on your automation tool, i.e. the ios_command module on Ansible). This will help you create a simple run book on Ansible to instruct your underlying switches to create 500 VLANs.

Automation tools are also useful to query the status of networking devices, especially if such queries are not supported via other protocols, such as SNNP or NETCONF (for example, to frequently retrieve the NAT table of a router or its BGP neighbors). Such queries are basically not available via SNMP in most of devices. However, by using automation tools, you will be able to get the desired output in a structured format.

[anitsh]

Knowledge Bridge

In this chapter, we will talk about automation tools and how they can help networking teams automate the repetitive tasks and build intelligence tools.

Many of you have been using CLI for many years. You may even be familiar with multiple CLI environments, such as Cisco IOS, Juniper JunOS, Arista EOS, Brocade, Huawei, HP, and others. Mastering the CLI commands will help you accomplish tasks faster, as well as have a clear understanding of how to troubleshoot, debug, and know where to look for problems.

Using automation tools will not replace your terminal and CLI, but it will help you ease your repetitive tasks. For example, if you need to frequently provision and decommission VLANs in your network with multiple switches, you will be able to simplify your work by using an automation tool to create a single button program to execute all the tasks on multiple devices at the same time. Most automation tools support the legacy equipment, and you will be able to use the tools not only to automate, but also to integrate with other platforms, such as ITSM. Automation tools are not limited to controlling routers and switches. They can control other networking devices, such as wireless controllers, firewalls, load balancers, etc.

[anitsh]

Ansible

Ansible is an open source agentless automation tool. Based on Python, Ansible is a very handy tool to manage both your legacy networking devices and your next-generation networking devices.

Ansible - Quick Summary Name	Ansible
By	Red Hat
Where it runs	On a workstation or a server
What it does	You can build automation playbooks to execute repeatable tasks on multiple devices
Features	Supports SSH/telnet access to networking devices. Ansible has multiple ready-made plugins for networking products, such as Cisco IOS, Arista, F5, Juniper, Cumulus, etc.
What it can do out-of-the-box	You can simply create Ansible scripts to tell Ansible to execute specific tasks on your equipment.

The key fundamentals of Ansible are: Inventory A list of devices that can be grouped together based on their type or function (for example, an inventory of switches, inventory of routers, etc.). Inventory is just a list of IP addresses. Task A task is a single executable action that Ansible can perform on a single device or on all devices in your inventory. Playbook A playbook is a YAML file, combining tasks and some control and action logic that manage the execution flow. For example, you may build a playbook to save the configuration of a switch, back up the running configuration and the firmware to an FTP server on a daily basis.

Ansible Galaxy is a community of Ansible users sharing their Ansible modules, roles and playbooks. Here you can find numerous ready-made modules for most networking devices, from generic Cisco IOS commands to Cisco Nexus, Junos OS, and next generation network products like Cumulus.

You can define an inventory file by creating a file and list down your equipment:

/etc/ansible/hosts [switches] 192.168.212.131 ansible_user=user ansible_ssh_pass=password 192.168.212.132 ansible_user=user ansible_ssh_pass=password 192.168.212.133 ansible_user=user ansible_ssh_pass=password 192.168.212.134 ansible_user=user ansible_ssh_pass=password 192.168.212.135 ansible_user=user ansible_ssh_pass=password 192.168.212.136 ansible_user=user ansible_ssh_pass=password

We can verify reachability to the equipment using the Ansible ping command.

$ ansible -m ping all

Using Ansible, we can run a command on all devices in our inventory:

$ ansible -m command -a "hostname" all

[anitsh]

Puppet

Puppet is an open source software configuration management tool. Puppet was initially created to ease the configuration management and package deployment on Linux servers. For example, to install MySQL, PHP, Python and Apache on a server, you don’t need to make any manual installation. You just need to create (or just reuse a template) and tell Puppet to install the required packages on a server. Puppet manages the dependencies, installs your required packages, and further monitors the server to ensure components are not changed all the time.

Puppet is slightly different than Ansible, as it uses a Master-Agent architecture. Puppet requires a Puppet agent software to be installed on all devices that will be controlled by Puppet. To install a Puppet agent on a device and load the certificates and key, you need to have full administration access to the device's operating system. This is not possible on devices that do not allow full access to their operating system, such as legacy CLI-based networking devices. A legacy networking device only presents a jailed CLI.

Puppet recently started adding the agentless features, but it can only manage a specific range of legacy Cisco devices. This may change in future by adding agentless support for other networking devices and other vendors.

Puppet - Quick Summary Name	Puppet
By	PuppetLabs
Where it runs	On a server. Requires agents to be installed on all managed devices (except Cisco IOS devices)
What it does	Manages the configuration of network devices and other servers
What it can do out-of-the-box	You can use Puppet to manage your Cisco IOS-based catalyst switches without the need of agents (as of June 2018). Puppet also supports other products, from Juniper, Cumulus Networks, OpenSwitch, Cisco ACI, etc.

Puppet programs are stored with a .pp extension and they are called manifests.

An example of a Cisco IOS configuration device:

ntp_server { '10.0.0.10': ensure => 'present', key => 94, prefer => true, minpoll => 4, maxpoll => 14, source_interface => 'Vlan 500', }

The above configuration will tell Puppet to add the following configuration to the Cisco IOS: ntp server 10.0.0.10 key 94 source Vlan500.

Also, the keyword ensure => 'present' tells Puppet to keep monitoring the configuration, and, if this command is removed from the configuration, Puppet will add it again. This method ensures that your network devices don't lose a configuration if it is changed by mistake.

On the other hand, if you are applying a manual change in your network devices, you should be very careful, as Puppet may have been configured to ensure the network device configuration is not changed. In such situations, once a manual change is applied to a device (for example, changing the NTP server IP address), Puppet will detect the change and revert back the configuration (in our case, to the old NTP server configuration).

Puppet Forge is similar to Ansible Galaxy. It is a library that contains numerous ready made Puppet templates and modules shared by the Puppet community.

[anitsh]

Chef

Chef is another automation tool based on a Server-Agent architecture - similar to Puppet, it requires an agent software to be installed on the nodes that it manages. Chef provides three open source projects:

    Chef - for infrastructure
    InSpec - for security compliance
    Habitat - for application deployment automation.

Chef also provides a commercial tool called Chef Automate, which provides analytics and a web GUI that allows you to manage a fully-automated infrastructure with Chef.

Chef - Quick Summary Name	Chef
By	Chef
Where it runs	On a separate server. Requires Chef agents to be installed on all managed nodes
What it does	Automates tasks for managing the infrastructure network, servers, application servers, as well as building and deploying applications
What it can do out-of-the-box	You can manage and automate the configuration of networking devices that come with a Chef agent or allow you to install a Chef agent. This includes all networking tools running on a host-based such as OVS, as well as other networking appliances such as Cisco Nexus switches.

The Chef server communicates with Chef agents installed on managed nodes. Chef uses a workstation where the user will be able to test all their Chef programs before uploading them to the Chef server. Knife is a CLI tool on the Chef workstation that interacts with the Chef server to upload Chef programs from the workstation to the server. [Chef Architecture]

Chef uses recipes and cookbooks:

        A recipe is a collection of defined resources that describe a system configuration. It describes how Chef should manage applications such as Apache, PHP, MySQL, etc., and how they should be configured. In networking, a recipe will describe how to configure a router or a switch (for example, how to set up VLANs, IP interfaces, etc.).
        A cookbook is a combination of recipes, defining a scenario that tells the Chef server to execute multiple recipes. For example, you can build a cookbook to configure virtual switches on 4 servers, create virtual interfaces and VLANs between them, assign IP addresses and establish BGP neighborship between them.

Chef Supermarket is a community site, where developers share their cookbooks and modules. This is similar to Ansible Galaxy and Puppet Forge.

Chef comes with readymade modules for configuration of some networking devices. The following example is a recipe that tells Chef to configure Ethernet 1/2 interface as an access port, in VLAN 100:

cisco_interface 'Ethernet1/2' do action :create access_vlan 100 shutdown false switchport_mode 'access' switchport_vtp true end

[anitsh]

Python

Python is a very popular modern programming language. Python is used for network automation, especially when you need to write complex, customized rules that are needed to perform specific sets of tasks. Python has many networking libraries that you can use to manage your network. Using Python, you can build programs that can connect to a network device, execute commands, grab the outputs, and show you the results.

Python supports multiple protocols such as SSH, SNMP, Telnet and APIs to communicate with a networking device. If you have an aging device that only supports older versions of SSH, you will be able to use Python’s SSH library (Paramiko) to manage that device.

When dealing with legacy networking devices that only support CLI (via SSH or Telnet), the main issue is to parse the outputs. In a CLI environment, the device always returns a stream of characters which is formatted for human reading. For example, when you issue a “show run interface Gigabit 1” command, the output is a set of characters that define the interface speed, its mode (Layer 2 without VLANs, Layer 2 with 802.1q, or Layer 3), VLANs, etc.:

interface gigabit 1 switchport switchport mode access switchport access vlan 100 speed 100 duplex full

If you are planning to find all the interfaces in your network that are configured with a “speed 100” command, you need to find a way to parse the output. Output parsing is a programming method that allows you to map the information to variables. In our example, if we parse the output, we should be able to have multiple variables filled with information from the raw output.

$port_speed[$i] = “speed 100” $port_duplex[$i] = “duplex full”

Parsing the output will help us easily find the facts and knowledge from text outputs. To parse the output, you need to either build your own parser, or use other parsing libraries. For example, Ansible Cisco ios_command module includes a built-in parser which can parse the output and generate a JSON-formatted output which can be used in program logic.

[anitsh]

Netmiko is a multi-vendor Python library that you can use to program different types of networking devices, as well as read and load outputs (not parsed). Netmiko uses the Paramiko library to establish an SSH connection to the networking devices and execute commands.

The Netmiko library:

        Successfully establishes an SSH connection to the device
        Simplifies the execution of show commands and the retrieval of output data
        Simplifies the execution of configuration commands, including possibly commit actions
        Performs the above across a broad set of networking vendors and platforms.

Using Netmiko to Connect to a Networking Device

The following example shows how to define a networking device and use Netmiko to connect to the device and execute some configurations.

from netmiko import ConnectHandler

cisco_881 = { 'device_type': 'cisco_ios', 'ip': '10.10.10.10', 'username': 'test', 'password': 'password', 'port' : 8022, # optional, defaults to 22 'secret': 'secret', # optional, defaults to '' 'verbose': False, # optional, defaults to False }

Establish an SSH connection to the device by passing in the device dictionary.

net_connect = ConnectHandler(**cisco_881)

Execute show commands.

output = net_connect.send_command('show ip int brief') print(output) Interface IP-Address OK? Method Status Protocol FastEthernet0 unassigned YES unset down down FastEthernet1 unassigned YES unset down down FastEthernet2 unassigned YES unset down down FastEthernet3 unassigned YES unset down down FastEthernet4 10.10.10.10 YES manual up up Vlan1

[anitsh]

NETCONF

The Network Configuration Protocol, or NETCONF, is a standard configuration protocol that uses a secure communication from the configuration host to the network device. It uses XML to send the configuration to a network device or when the network device sends the configuration results back to the manager node.

To use NETCONF, you need to ensure the networking device supports the NETCONF protocol (most networking devices support NETCONF). NETCONF uses SSH as the underlying transport protocol. The default port for the NETCONF agent is TCP 830.

NETCONF is similar to the Simple Network Management Protocol (SNMP), but has more capabilities. SNMP is at tool that has been used to monitor networking devices, including the interface traffic. In addition, it is used to apply some minimal configurations. The main problem with SNMP is the lack of discovery process, where you need to know exact Object Identifiers (OIDs) that you would like to read or write, and have the relevant MIB databases to load in your network monitoring system. NETCONF includes a discovery function that can find what devices are supported.

NETCONF features:

        Separation between state information and configuration data
        Stores multiple configuration data
        Configuration charge transactions
        Configuration testing and validation
        Data filtering while reading data from device.

NETCONF Command	Description
	Retrieve running configuration and device state information
	Retrieve all or part of a specified configuration datastore
	Edit a configuration datastore by creating, deleting, merging, or replacing content
	Copy an entire configuration datastore to another configuration datastore
	Delete a configuration datastore
	Lock an entire configuration datastore of a device
	Release a configuration datastore lock previously obtained with the command
	Request graceful termination of a NETCONF session
	Force the termination of a NETCONF session

[anitsh]

Video: Network automation with Ansible and Puppet.

[anitsh]

Learning Objectives (Review)

You should now be able to:

        Discover available network automation tools which can help automate repetitive tasks.
        Review the architecture of network automation tools and how they work.

[anitsh]

Summary

In this chapter, we reviewed some of the automation tools that can be used for network automation. Network automation can be used in a legacy or modern SDN network. Creating YANG models for your network will help you prepare for future changes, upgrades and complete migration from Vendor A to Vendor B equipment.

All automation tools that we discussed in this chapter are free and open source (apart from commercial add-ons). You can try and use any of the tools to find out how you can automate and integrate your network with other applications.