#! META
name: chkp-gaia-routes-vsx
description: Report configured static and direct routes, and compare configured static routes with Linux routes.
type: monitoring
monitoring_interval: 10 minutes
requires:
vendor: "checkpoint"
os.name: "gaia"
vsx: "true"
role-firewall: "true"
asg:
neq: "true"
#! COMMENTS
static-routing-table:
why: |
It is important that the routing is configured the same for all cluster members of the same cluster. Otherwise there can be downtime in the event of a failover.
how: |
By parsing the gaia configuration database, /config/active, the static routes are retrieved. It can also be retrieved via Clish, but that creates a lot of log entries in /var/log/messages.
without-indeni: |
An administrator could login and manually run the command.
can-with-snmp: true
can-with-syslog: false
vendor-provided-management: |
Listing static routes is only available from the command line interface or via SNMP. In VSX it is also visible in SmartDashboard.
connected-networks-table:
why: |
It is important that the connected interfaces is configured the same, for all cluster members of the same cluster. Otherwise there can be downtime in the event of a failure.
how: |
By parsing the gaia configuration database, /config/active, the routes for directly connected interfaces are retrieved. It can also be retrieved via Clish, but that creates a lot of log entries in /var/log/messages.
without-indeni: |
An administrator could login and manually run the command.
can-with-snmp: true
can-with-syslog: false
vendor-provided-management: |
Listing routes for directly connected interfaces is only available from the command line interface, or SNMP.
routes-missing-kernel:
why: |
If a static route is configured via Clish or WebUI, sometimes the system does not write the route into the Linux kernel routing table. To make sure all routes have been written, we compare the actual kernel routes with those configured in Check Point.
how: |
Retrieve Linux kernel routes using the Linux "netstat" command, and then the Check Point configured routes from Gaia's /config/active file. Then compare two route sets to make sure they are the same.
without-indeni: |
An administrator could login and manually list routes from both commands, and then compare it. However, often there are a many routes configured; combine this with the difference in output format (for example subnet), and it can be a very cumbersome task.
can-with-snmp: false
can-with-syslog: false
vendor-provided-management: |
Listing routes from kernel is only available from the command line interface. Listing configured routes is also available from the WebUI.
#! REMOTE::SSH
${nice-path} -n 15 vsx stat -l | ${nice-path} -n 15 awk '/^VSID:/{ vsid=$NF } /^Type:/{ type = ""; for (i=2; i<=NF;i++) {type = type $i} } /^Name:/{ if (type == "VSXGateway" || type == "VirtualSystem"){ print vsid ":" $NF } }' | while read idAndName; do echo "_VSID:Name $idAndName"; IFS=':' read -ra arr <<< "$${idAndName}"; vsenv "$${arr[0]}" && ${nice-path} -n 15 netstat -rn; done; ${nice-path} -n 15 grep "route" /config/active
#! PARSER::AWK
#*** SSH Commandline Comments ***
# Here's a step-by-step explanation of the above command:
# - greps the output of 'vsx stat -l' for the 'VSID:' and 'Name:' lines. Filters on 'Type:'.
# - pipes the vsid:name, as $idAndName, into a while loop, which:
# - echos each VS ID and Name
# - parses the ID back out of $idAndName:
# - splits the line on ':' into the 'arr' variable --> IFS=':' read -ra arr <<< "${idOrName}"
# - gets the VSID # out of 'arr' as arr[0] (second index in array)
# - sets the vsenv to the VSID and runs netstat in that context
# - greps /config/active for 'route'
#
# NOTE: even though this commandline grep's /config/active, it shouldn't be a problem for this script,
# since the last many lines of the input are already ignored by this script (i.e., not pertinent or used
# in any way).
function dumpVsRouteData() {
if (last_instance == "default") { # default is VSID 0
last_instance = "0"
}
vs_tags["vs.id"] = last_instance
vs_tags["vs.name"] = vsid_to_name[last_instance]
if (arraylen(static_routes)) {
writeComplexMetricObjectArrayWithLiveConfig("static-routing-table", vs_tags, static_routes, "Static Routing Table")
}
if (arraylen(direct_routes)) {
writeComplexMetricObjectArrayWithLiveConfig("connected-networks-table", vs_tags, direct_routes, "Directly Connected Networks")
}
# delete arrays between "vs groups" of data
delete static_routes
delete direct_routes
last_instance = current_instance # set 'last' to the new (next) instance group
}
BEGIN {
last_instance = ""
ospf_enabled = 0
net_mask_to_CIDR["0.0.0.0"] = 0
net_mask_to_CIDR["128.0.0.0"] = 1
net_mask_to_CIDR["192.0.0.0"] = 2
net_mask_to_CIDR["224.0.0.0"] = 3
net_mask_to_CIDR["240.0.0.0"] = 4
net_mask_to_CIDR["248.0.0.0"] = 5
net_mask_to_CIDR["252.0.0.0"] = 6
net_mask_to_CIDR["254.0.0.0"] = 7
net_mask_to_CIDR["255.0.0.0"] = 8
net_mask_to_CIDR["255.128.0.0"] = 9
net_mask_to_CIDR["255.192.0.0"] = 10
net_mask_to_CIDR["255.224.0.0"] = 11
net_mask_to_CIDR["255.240.0.0"] = 12
net_mask_to_CIDR["255.248.0.0"] = 13
net_mask_to_CIDR["255.252.0.0"] = 14
net_mask_to_CIDR["255.254.0.0"] = 15
net_mask_to_CIDR["255.255.0.0"] = 16
net_mask_to_CIDR["255.255.128.0"] = 17
net_mask_to_CIDR["255.255.192.0"] = 18
net_mask_to_CIDR["255.255.224.0"] = 19
net_mask_to_CIDR["255.255.240.0"] = 20
net_mask_to_CIDR["255.255.248.0"] = 21
net_mask_to_CIDR["255.255.252.0"] = 22
net_mask_to_CIDR["255.255.254.0"] = 23
net_mask_to_CIDR["255.255.255.0"] = 24
net_mask_to_CIDR["255.255.255.128"] = 25
net_mask_to_CIDR["255.255.255.192"] = 26
net_mask_to_CIDR["255.255.255.224"] = 27
net_mask_to_CIDR["255.255.255.240"] = 28
net_mask_to_CIDR["255.255.255.248"] = 29
net_mask_to_CIDR["255.255.255.252"] = 30
net_mask_to_CIDR["255.255.255.254"] = 31
net_mask_to_CIDR["255.255.255.255"] = 32
}
#_VSID:Name 0:lab-CP-VSXVSLS1
/^_VSID:Name/ {
arr_len = split($2, id_and_name, ":")
if (arr_len != 2) {
# TODO: what to do?
}
stat_vs_id = id_and_name[1]
vs_name = id_and_name[2]
vsid_to_name[stat_vs_id] = vs_name
ordered_vsid[++ordered_vsid_index] = stat_vs_id
next
}
# netstat output: store information about routes configured in the OS.
#10.11.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/ {
destination = $1
subnet_mask = $3
cidr = net_mask_to_CIDR[subnet_mask]
flags = $4
gateway = $2
if (destination == "0.0.0.0") {
cidr = "0"
}
netstat_routes_for_compare[stat_vs_id ":" destination "/" cidr ":" gateway] = ""
next
}
#routed:instance:7:static:network:10.20.10.0:masklen:27:gateway:address:10.10.10.6 t
#routed:instance:default:static:network:10.10.100.0:masklen:24:gateway:address:10.11.2.50 t
/routed:instance:/ {
# This entire section is just to track the groups of VS data from /config/active
split($1, split_line, ":")
current_instance = split_line[3]
# This whole "last_instance" "current_instance" dance lets us process each group of instance data (from /config/active)
# as a group, group by group.
if (last_instance == "") { # only first time through
last_instance = current_instance
}
# Every time we hit a 'new' current_instance (except the first one), dump all the data from the previous instance
if (current_instance != last_instance) {
dumpVsRouteData()
}
}
# Store information about a route which has been configured on the device.
#routed:instance:default:static:network:10.10.100.0:masklen:24:gateway:address:10.11.2.50 t
#routed:instance:7:static:network:10.20.10.0:masklen:27:gateway:address:10.10.10.6 t
#routed:instance:7:static:network:10.10.10.0:masklen:24:gateway:lname:eth0.10 t
#routed:instance:2:static:network:192.168.194.0:masklen:24:gateway t
/routed:instance:.*:static:network:.+:masklen:[0-9]+:gateway:/ {
split($1, split_line, ":")
# if the second-to-last field is an IP address, it's a static route
if (split_line[11] ~ /[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/) {
# This is a static route
gateway = split_line[11] # gateway:address --> 10.10.10.6
destination_network = split_line[6] # destination network --> 10.20.10.0
cidr = split_line[8] # masklen --> 27
config_vs_id = split_line[3]
if (config_vs_id == "default")
config_vs_id = 0
# use this format to compare (below) with data collected from netstat
static_routes_for_compare[config_vs_id ":" destination_network "/" cidr ":" gateway] = ""
static_count++
static_routes[static_count, "network"] = destination_network
static_routes[static_count, "mask"] = cidr
static_routes[static_count, "next-hop"] = gateway
} else {
# This is a directly connected route
destination_network = split_line[6]
cidr = split_line[8]
direct_count++
direct_routes[direct_count, "network"] = destination_network
direct_routes[direct_count, "mask"] = cidr
}
next
}
# Store information about a default gateway configured in Clish.
#routed:instance:default:static:default:gateway:address:192.168.194.1 t
/routed:instance:default:static:default:gateway:address/ {
split($1, split_line, ":")
if (split_line[8] ~ /[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/) {
gateway = split_line[8]
static_routes_for_compare["0:0.0.0.0/0:" gateway] = ""
static_count++
static_routes[static_count, "network"] = "0.0.0.0"
static_routes[static_count, "mask"] = "0"
static_routes[static_count, "next-hop"] = gateway
}
next
}
# If the VSX is using OSPF, we will still report configured routes, but we won't look for missing routes.
#routed:instance:default:ospf2:interface:eth0:area 0.0.0.1
#routed:instance:default:ospf2:interface:eth1:area backbone
/^routed:instance:default:ospf2:interface:.*:area/ {
ospf_enabled = 1
next
}
END {
# Dump the last data
dumpVsRouteData()
# If host uses OSPF, we're done: no route comparison with netstat
if (ospf_enabled)
exit
missing_sorted[1] = ""
missing_route_index = 0
# For each static route configured on the device, make sure it's also in the netstat output
for (static_route in static_routes_for_compare) {
if (! (static_route in netstat_routes_for_compare)) { # A missing route
# Sort missing routes as we go so that we can write them out, per VSID, as 'missing-route' metrics below
missing_sorted[++missing_route_index] = static_route
for (i = missing_route_index; i > 0; i--) {
if (static_route < missing_sorted[i]) {
hold = missing_sorted[i]
missing_sorted[i] = static_route
missing_sorted[i + 1] = hold
}
}
}
}
# Dump the missing route metrics. There can be 0 -> * missing routes for each VSID.
# get the first missing route VSID if there is one
curr_missing_id = -1
curr_missing_index = 1
if (missing_sorted[curr_missing_index] != "") { # we have at least one missing route
arr_len = split(missing_sorted[1], missing_split, ":")
# TODO: validate l
curr_missing_id = missing_split[1]
}
# for each VSID
for (i in ordered_vsid) {
split("", this_vs_missing) # This is how to initialize an empty array in AWK -- see use below
vs_id = ordered_vsid[i]
metric_tags["vs.name"] = "VSID: " vs_id ", VS Name: " vsid_to_name[vs_id]
if (vs_id == curr_missing_id) {
missing_per_vs_index = 0
while (curr_missing_id == vs_id) { # collect all missing routes for this VSID
# add current missing to list
this_vs_missing[++missing_per_vs_index, "missing-route"] = "network: " missing_split[2] " next-hop: " missing_split[3]
# get next missing VSID
if (++curr_missing_index in missing_sorted) {
arr_len = split(missing_sorted[curr_missing_index], missing_split, ":")
# TODO: validate l
curr_missing_id = missing_split[1]
} else {
curr_missing_id = -1 # we walked off the end of the missing routes list -- no more of them
}
}
}
# Note: if there are no missing routes for this VSID, we write an empty array object to insure correct alert resolution.
# See the initialization of this_vs_missing above.
writeComplexMetricObjectArray("routes-missing-kernel", metric_tags, this_vs_missing)
delete this_vs_missing
}
}
haughki
commented
6 years ago 