Make Route Reflector peering with a Edge Router - Solved

[gunboe]

Expected Behavior

Make Route Reflector peering with a Edge Router

Current Behavior

Actualy, RR has no mechanism to configure peerings with non-Calico BGP speakers (e.g. edge routers).

Possible Solution

1. Create a new file named: bird.cfg.template with the contents:

   
   # Generated by confd
   router id {{getenv "IP"}};

Watch interface up/down events.

protocol device { scan time 2; # Scan interfaces every 2 seconds }

Template for all BGP clients

template bgp bgp_template { debug off; description "Connection to BGP peer"; multihop; import all; # Import all routes, since we don't know what the upstream

topology is and therefore have to trust the ToR/RR.

export all; # Export all. source address {{getenv "IP"}}; # The local address we use for the TCP connection graceful restart; # See comment in kernel section about graceful restart. }

{{$our_rr_key := printf "/rr_v4/%s" (getenv "IP")}} {{if ls $our_rr_key}}{{$our_rr_data := json (getv $our_rr_key)}}

---------------- EDGE Router -----------------

{{if eq "" (getenv "IP_EDGE") (getenv "AS_EDGE") (getenv "NET_CONTAINER")}}

No Edge Router set!

{{else}}

Peering with Edge Router

protocol bgp edge_router from bgp_template { local as {{getv "/global/as_num"}}; neighbor {{getenv "IP_EDGE"}} as {{getenv "AS_EDGE"}}; next hop keep; export filter { if net ~ {{getenv "NET_CONTAINER"}} then accept; }; } {{end}}

------------- RR-to-RR full mesh -------------

{{if ls "/rr_v4"}} {{range gets "/rr_v4/*"}}{{$data := json .Value}}{{$rr_ip := $data.ip}} {{$nums := split $rrip "."}}{{$id := join $nums ""}}

For RR {{$rr_ip}}

{{if eq $rr_ip (getenv "IP") }}# Skipping ourselves {{else if ne "" $rrip}}protocol bgp Mesh{{$id}} from bgp_template { local as {{getv "/global/as_num"}}; neighbor {{$rr_ip}} as {{getv "/global/as_num"}}; }{{end}}{{end}} {{end}}

------------- RR as a global peer -------------

{{if ls "/global/peer_v4"}} {{range gets "/global/peer_v4/*"}}{{$data := json .Value}} {{if eq $data.ip (getenv "IP")}}

This RR is a global peer with all calico nodes.

{{range $cnode := lsdir "/host"}} {{$cnode_as_key := printf "/host/%s/as_num" $cnode}} {{$cnode_ip_key := printf "/host/%s/ip_addr_v4" $cnode}}{{$cnode_ip := getv $cnode_ip_key}} {{$nums := split $cnodeip "."}}{{$id := join $nums ""}}

Peering with Calico node {{$cnode}}

protocol bgp Global_{{$id}} from bgp_template { local as {{$data.as_num}}; neighbor {{$cnode_ip}} as {{if exists $cnode_as_key}}{{getv $cnode_as_key}}{{else}}{{getv "/global/as_num"}}{{end}}; rr client; {{if $our_rr_data.cluster_id}}rr cluster id {{$our_rr_data.cluster_id}};{{end}} } {{end}} {{end}} {{end}} {{end}}

------------- RR as a node-specific peer -------------

{{range $cnode := lsdir "/host"}} {{$node_peers_key := printf "/host/%s/peer_v4" $cnode}} {{if ls $node_peers_key}} {{range $peer := gets (printf "%s/*" $node_peers_key)}}{{$data := json $peer.Value}} {{if eq $data.ip (getenv "IP")}} {{$cnode_as_key := printf "/host/%s/as_num" $cnode}} {{$cnode_ip_key := printf "/host/%s/ip_addr_v4" $cnode}}{{$cnode_ip := getv $cnode_ip_key}} {{$nums := split $cnodeip "."}}{{$id := join $nums ""}}

RR configured as a specific peer for calico node {{$peer.Key}}

protocol bgp Node_{{$id}} from bgp_template { local as {{$data.as_num}}; neighbor {{$cnode_ip}} as {{if exists $cnode_as_key}}{{getv $cnode_as_key}}{{else}}{{getv "/global/as_num"}}{{end}}; rr client; {{if $our_rr_data.cluster_id}}rr cluster id {{$our_rr_data.cluster_id}};{{end}} } {{end}} {{end}} {{end}} {{end}} {{else}}

--------------- Standalone RR -----------------

------------- RR as a global peer -------------

{{if ls "/global/peer_v4"}} {{range gets "/global/peer_v4/*"}}{{$data := json .Value}} {{if eq $data.ip (getenv "IP")}}

This RR is a global peer with all calico nodes.

{{range $cnode := lsdir "/host"}} {{$cnode_as_key := printf "/host/%s/as_num" $cnode}} {{$cnode_ip_key := printf "/host/%s/ip_addr_v4" $cnode}}{{$cnode_ip := getv $cnode_ip_key}} {{$nums := split $cnodeip "."}}{{$id := join $nums ""}}

Peering with Calico node {{$cnode}}

protocol bgp Global_{{$id}} from bgp_template { local as {{$data.as_num}}; neighbor {{$cnode_ip}} as {{if exists $cnode_as_key}}{{getv $cnode_as_key}}{{else}}{{getv "/global/as_num"}}{{end}}; rr client; } {{end}} {{end}} {{end}} {{end}}

------------- RR as a node-specific peer -------------

{{range $cnode := lsdir "/host"}} {{$node_peers_key := printf "/host/%s/peer_v4" $cnode}} {{if ls $node_peers_key}} {{range $peer := gets (printf "%s/*" $node_peers_key)}}{{$data := json $peer.Value}} {{if eq $data.ip (getenv "IP")}} {{$cnode_as_key := printf "/host/%s/as_num" $cnode}} {{$cnode_ip_key := printf "/host/%s/ip_addr_v4" $cnode}}{{$cnode_ip := getv $cnode_ip_key}} {{$nums := split $cnodeip "."}}{{$id := join $nums ""}}

RR configured as a specific peer for calico node {{$peer.Key}}

protocol bgp Node_{{$id}} from bgp_template { local as {{$data.as_num}}; neighbor {{$cnode_ip}} as {{if exists $cnode_as_key}}{{getv $cnode_as_key}}{{else}}{{getv "/global/as_num"}}{{end}}; rr client; } {{end}} {{end}} {{end}} {{end}}

{{end}}

2. Build a new image with this Dockerfile:

FROM calico/routereflector:v0.4.2 COPY bird.cfg.template /templates/bird.cfg.template

> Take care with the apropriate version

docker build -t routereflector:v0.4.2-new .

3. Run the RR container and define apropriate enviroments:

docker run --rm --privileged --net=host -d --name=rreflector \ -e IP_EDGE="10.79.200.1" \ -e AS_EDGE="65533" \ -e NET_CONTAINER="10.79.128.0/18" \ -e IP="10.79.200.13" \ -e ETCD_ENDPOINTS="https://srv001:2379,https://srv002:2379,https://srv003:2379" \ -v /etc/ssl/private/:/ssl \ -e ETCD_CA_CERT_FILE=/ssl/ca.pem \ -e ETCD_CERT_FILE=/ssl/rr001.pem \ -e ETCD_KEY_FILE=/ssl/rr001-key.pem \ routereflector:v0.4.2-new

## Your Environment
* Calico version: 2.6
* Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.7.11
* Operating System and version: CoreOS 

Sory all, but I dont know how to make a PR and submit the code above. Enjoy!!!

[gunboe]

Yeah! The magic is on template here:

# ---------------- EDGE Router -----------------
{{if eq "" (getenv "IP_EDGE") (getenv "AS_EDGE") (getenv "NET_CONTAINER")}}
# No Edge Router set!
{{else}}
# Peering with Edge Router 
protocol bgp edge_router from bgp_template {
  local as {{getv "/global/as_num"}};
  neighbor {{getenv "IP_EDGE"}} as {{getenv "AS_EDGE"}};
  next hop keep;
  export filter { if net ~ {{getenv "NET_CONTAINER"}} then accept; };
}
{{end}}

[ozdanborne]

Nice job @gunboe, thanks for sharing. this is really useful for others looking to do the same thing

[caseydavenport]

FYI Calico v3.3 will support running calico/node in a route reflection mode, in which it will also support peering with edge routers / TOR / etc, configurable using BGPPeer resources - https://docs.projectcalico.org/v3.2/reference/calicoctl/resources/bgppeer

[nicolaka]

@caseydavenport just clarifying your note, I'm running calico/routereflector:0.6.4 today on a node in Docker EE cluster. Everything is working fine. I'm trying to prove that we can integrate/peer with edge router. When I add edge router as a node peer based on https://docs.projectcalico.org/v3.2/reference/calicoctl/resources/bgppeer, bgp connection never come up. I'd like to also point out that calico/node is not running on the RR node, it's only the routreflector pod that is running. I'm assuming that since we're not running calico/node, the node peer is not taking effect?

[caseydavenport]

@nicolaka the feature I was referring to will be released in Calico v3.3, which isn't available yet.

[nicolaka]

thanks @caseydavenport that's what i thought too.

[caseydavenport]

it's only the routreflector pod that is running. I'm assuming that since we're not running calico/node, the node peer is not taking effect?

Yep, this is right - in v3.3, you'd stop running the calico/routereflector container and just run calico/node, which will be able to perform as a RR and also peer with external BGP speakers.