search

loxilb-io / kube-loxilb

Implementation of kubernetes service load-balancer spec for loxilb
Apache License 2.0
85 stars 16 forks source link

ip loadbalancer bgp can not access from browser #53

Closed zainal-abidin-assegaf closed 12 months ago

zainal-abidin-assegaf commented 12 months ago

image

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-loxilb
  namespace: kube-system
  labels:
    app: kube-loxilb-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-loxilb-app
  template:
    metadata:
      labels:
        app: kube-loxilb-app
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      priorityClassName: system-node-critical
      serviceAccountName: kube-loxilb
      terminationGracePeriodSeconds: 0
      containers:
      - name: kube-loxilb
        image: 172.24.25.210:5000/kube-loxilb:latest
        imagePullPolicy: Always
        command:
        - /bin/kube-loxilb
        args:
        - --loxiURL=http://172.24.25.207:11111,http://172.24.25.208:11111,http://172.24.25.209:11111
        - --externalCIDR=172.24.25.160/27
        #- --externalSecondaryCIDRs=124.124.124.1/24,125.125.125.1/24
        - --monitor=true
        - --setUniqueIP=true
        #- --setRoles=0.0.0.0
        - --setBGP=64505
        - --extBGPPeers=172.24.25.207:64502,172.24.25.208:64503,172.24.25.209:64504
        - --setLBMode=2
        #- --config=/opt/loxilb/agent/kube-loxilb.conf
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]

svc

apiVersion: v1
kind: Service
metadata:
  annotations:
   # If there is a need to do liveness check from loxilb
   loxilb.io/liveness: "yes"
   # Specify LB mode - one of default, onearm or fullnat 
   loxilb.io/lbmode: "onearm"
   # Specify loxilb IPAM mode - one of ipv4, ipv6 or ipv6to4 
   loxilb.io/ipam: "ipv4"
   # Specify number of secondary networks for multi-homing
   # Only valid for SCTP currently
   # loxilb.io/num-secondary-networks: "2
  labels:
    app.kubernetes.io/name: production-rabbitmqcluster
  name: production-rabbitmqcluster-lb-static
  namespace: default
spec:
  ports:
  - appProtocol: amqp
    name: amqp
    port: 5672
    protocol: TCP
    targetPort: 5672
  - appProtocol: http
    name: management
    port: 15672
    protocol: TCP
    targetPort: 15672
  - appProtocol: prometheus.io/metrics
    name: prometheus
    port: 15692
    protocol: TCP
    targetPort: 15692
  selector:
    app.kubernetes.io/name: production-rabbitmqcluster
  sessionAffinity: None
  type: LoadBalancer
  loadBalancerClass: loxilb.io/loxilb
  externalTrafficPolicy: Local
  externalIPs:
    - 172.24.25.180

curl 172.24.25.180:15672

core@dev-manager-01 ~/rabbitmq-operator $ curl 172.24.25.180:15672
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>RabbitMQ Management</title>
    <script src="js/ejs-1.0.min.js" type="text/javascript"></script>
    <script src="js/jquery-3.5.1.min.js"></script>
    <script src="js/jquery.flot-0.8.1.min.js" type="text/javascript"></script>
    <script src="js/jquery.flot-0.8.1.time.min.js" type="text/javascript"></script>
    <script src="js/sammy-0.7.6.min.js" type="text/javascript"></script>
    <script src="js/json2-2016.10.28.js" type="text/javascript"></script>
    <script src="js/base64.js" type="text/javascript"></script>
    <script src="js/global.js" type="text/javascript"></script>
    <script src="js/main.js" type="text/javascript"></script>
    <script src="js/prefs.js" type="text/javascript"></script>
    <script src="js/formatters.js" type="text/javascript"></script>
    <script src="js/charts.js" type="text/javascript"></script>
    <script src="js/oidc-oauth/helper.js"></script>
    <script src="js/oidc-oauth/oidc-client-ts.js" type="text/javascript"></script>
    <script src="js/oidc-oauth/bootstrap.js"></script>

    <link href="css/main.css" rel="stylesheet" type="text/css"/>
    <link href="favicon.ico" rel="shortcut icon" type="image/x-icon"/>

    <script type="application/javascript">
      var oauth = oauth_initialize_if_required();

      if (oauth.enabled) {
        if (!oauth.sp_initiated) {
            oauth.logged_in = has_auth_credentials();
            oauth.access_token = get_auth_credentials(); // DEPRECATED
        } else {
          oauth_is_logged_in().then( status => {
            if (status.loggedIn && !has_auth_credentials()) {
              oauth.logged_in = false;
              oauth_initiateLogout();
            } else {
              if (!status.loggedIn) {
                replace_content('outer', format('login_oauth', {}));
                clear_auth();
              } else {
                oauth.logged_in = true;
                oauth.access_token = status.user.access_token;  // DEPRECATED
                oauth.expiryDate = new Date(status.user.expires_at * 1000);  // it is epoch in seconds
                let current = new Date();
                _management_logger.debug('token expires in ', (oauth.expiryDate-current)/1000,
                  'secs at : ', oauth.expiryDate );
                oauth.user_name = status.user.profile['user_name'];
                if (!oauth.user_name || oauth.user_name == '') {
                  oauth.user_name = status.user.profile['sub'];
                }
                oauth.scopes = status.user.scope;
              }
            }
          });
        }
      }

    </script>

<!--[if lte IE 8]>
    <script src="js/excanvas.min.js" type="text/javascript"></script>
    <link href="css/evil.css" rel="stylesheet" type="text/css"/>
<![endif]-->
  </head>

  <body>
    <div id="outer"></div>
    <div id="debug"></div>
    <div id="scratch"></div>
  </body>
</html>

loxilb1 logs image loxilb2 logs image loxilb3 logs image

kube-loxilb logs:

2023-09-19T22:45:28.970284211+07:00 I0919 15:45:28.970081       1 logLevel.go:40] Set log level to 0
2023-09-19T22:45:28.970321371+07:00 I0919 15:45:28.970183       1 agent.go:52] Starting kube-loxilb:
2023-09-19T22:45:28.970326190+07:00 I0919 15:45:28.970187       1 agent.go:53]   Version: 
2023-09-19T22:45:28.970330397+07:00 I0919 15:45:28.970190       1 agent.go:54]   Build: 2023_09_17-main-a33ca0e
2023-09-19T22:45:28.970334445+07:00 I0919 15:45:28.970199       1 client.go:36] No kubeconfig file was specified. Falling back to in-cluster config
2023-09-19T22:45:28.971069073+07:00 I0919 15:45:28.970924       1 agent.go:72] URLs: [http://172.24.25.207:11111 http://172.24.25.208:11111 http://172.24.25.209:11111]
2023-09-19T22:45:28.971080023+07:00 I0919 15:45:28.970966       1 agent.go:73] LB Class: loxilb.io/loxilb
2023-09-19T22:45:28.971083921+07:00 I0919 15:45:28.970970       1 agent.go:74] CIDR: 172.24.25.160/27
2023-09-19T22:45:28.971087908+07:00 I0919 15:45:28.970973       1 agent.go:75] SetBGP: 64505
2023-09-19T22:45:28.971091555+07:00 I0919 15:45:28.970977       1 agent.go:76] ListenBGPPort: 0
2023-09-19T22:45:28.971095032+07:00 I0919 15:45:28.970980       1 agent.go:77] SetLBMode: 2
2023-09-19T22:45:28.971098578+07:00 I0919 15:45:28.971029       1 agent.go:78] ExclIPAM: true
2023-09-19T22:45:28.971102075+07:00 I0919 15:45:28.971033       1 agent.go:79] Monitor: true
2023-09-19T22:45:28.971105571+07:00 I0919 15:45:28.971037       1 agent.go:80] SecondaryCIDRs: []
2023-09-19T22:45:28.971109388+07:00 I0919 15:45:28.971040       1 agent.go:81] ExtBGPPeers: [172.24.25.207:64502 172.24.25.208:64503 172.24.25.209:64504]
2023-09-19T22:45:28.971112885+07:00 I0919 15:45:28.971045       1 agent.go:82] SetRoles: 
2023-09-19T22:45:28.972438455+07:00 I0919 15:45:28.971477       1 client.go:64] NewLoxiClient Created: http://172.24.25.207:11111
2023-09-19T22:45:28.972457571+07:00 I0919 15:45:28.971494       1 client.go:64] NewLoxiClient Created: http://172.24.25.208:11111
2023-09-19T22:45:28.972461899+07:00 I0919 15:45:28.971498       1 client.go:64] NewLoxiClient Created: http://172.24.25.209:11111
2023-09-19T22:45:28.972465817+07:00 I0919 15:45:28.971786       1 loadbalancer.go:227] Starting LoxilbLoadBalancerManager
2023-09-19T22:45:28.972470185+07:00 I0919 15:45:28.971802       1 shared_informer.go:273] Waiting for caches to sync for LoxilbLoadBalancerManager
2023-09-19T22:45:28.973191083+07:00 I0919 15:45:28.973086       1 client.go:80] LoxiHealthCheckChan: loxilb(http://172.24.25.209:11111) is alive
2023-09-19T22:45:28.973204358+07:00 I0919 15:45:28.973123       1 client.go:80] LoxiHealthCheckChan: loxilb(http://172.24.25.208:11111) is alive
2023-09-19T22:45:28.973208225+07:00 I0919 15:45:28.973145       1 client.go:80] LoxiHealthCheckChan: loxilb(http://172.24.25.207:11111) is alive
2023-09-19T22:45:29.173435813+07:00 I0919 15:45:29.173309       1 shared_informer.go:280] Caches are synced for LoxilbLoadBalancerManager
2023-09-19T22:45:29.174589287+07:00 I0919 15:45:29.174441       1 loadbalancer.go:1384] set-bgp-global failed(loxilb BGP mode is disabled)
2023-09-19T22:45:29.175528854+07:00 I0919 15:45:29.175354       1 loadbalancer.go:1384] set-bgp-global failed(loxilb BGP mode is disabled)
2023-09-19T22:45:29.176492677+07:00 I0919 15:45:29.176354       1 loadbalancer.go:1384] set-bgp-global failed(loxilb BGP mode is disabled)
2023-09-19T22:45:29.180541426+07:00 I0919 15:45:29.180383       1 loadbalancer.go:573] Endpoint IP Pairs [172.24.25.201 172.24.25.203 172.24.25.202]
2023-09-19T22:45:29.180567725+07:00 I0919 15:45:29.180490       1 loadbalancer.go:574] Secondary IP Pairs []
2023-09-19T22:45:29.182896363+07:00 I0919 15:45:29.182701       1 loadbalancer.go:668] load-balancer ([{true true  {{172.24.25.180 5672 tcp 0 1 true true 1800 0 true  0  } [] [{172.24.25.201 30862 4 } {172.24.25.203 30862 3 } {172.24.25.202 30862 3 }]}}]) added
2023-09-19T22:45:29.186054781+07:00 I0919 15:45:29.185794       1 loadbalancer.go:668] load-balancer ([{true true  {{172.24.25.180 15672 tcp 0 1 true true 1800 0 true  0  } [] [{172.24.25.201 30943 4 } {172.24.25.203 30943 3 } {172.24.25.202 30943 3 }]}}]) added
2023-09-19T22:45:29.188174196+07:00 I0919 15:45:29.188041       1 loadbalancer.go:668] load-balancer ([{true true  {{172.24.25.180 15692 tcp 0 1 true true 1800 0 true  0  } [] [{172.24.25.201 32415 4 } {172.24.25.203 32415 3 } {172.24.25.202 32415 3 }]}}]) added
zainal-abidin-assegaf commented 12 months ago

docker frr used in all loxilb nodes, frr loxilb1 nodes image

frr version 7.5.1
frr defaults traditional
hostname dev-worker-001.brids.co.id
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface ens160
 ip address 172.24.25.207/24
!
router bgp 64502
 bgp router-id 172.24.25.207
 timers bgp 3 15
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor loxilb peer-group
 neighbor loxilb remote-as 64505
 neighbor 172.24.25.208 remote-as 64503
 neighbor 172.24.25.209 remote-as 64504
 !
 address-family ipv4 unicast
  neighbor 172.24.25.208 activate
  neighbor 172.24.25.208 route-map 172.24.25.202-in in
  neighbor 172.24.25.209 activate
  neighbor 172.24.25.209 route-map 172.24.25.203-in in
 exit-address-family
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
line vty
!
end

image frr loxilb2 nodes

frr version 7.5.1
frr defaults traditional
hostname dev-worker-002.brids.co.id
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface ens160
 ip address 172.24.25.208/24
!
router bgp 64503
 bgp router-id 172.24.25.208
 timers bgp 3 15
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor loxilb peer-group
 neighbor loxilb remote-as 64505
 neighbor 172.24.25.207 remote-as 64502
 neighbor 172.24.25.209 remote-as 64504
 !
 address-family ipv4 unicast
  neighbor 172.24.25.207 activate
  neighbor 172.24.25.207 route-map 172.24.25.201-in in
  neighbor 172.24.25.209 activate
  neighbor 172.24.25.209 route-map 172.24.25.203-in in
 exit-address-family
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
line vty
!
end

image frr loxilb3 nodes

frr version 7.5.1
frr defaults traditional
hostname dev-worker-003.brids.co.id
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface ens160
 ip address 172.24.25.209/24
!
router bgp 64504
 bgp router-id 172.24.25.209
 timers bgp 3 15
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor loxilb peer-group
 neighbor loxilb remote-as 64505
 neighbor 172.24.25.207 remote-as 64502
 neighbor 172.24.25.208 remote-as 64503
 !
 address-family ipv4 unicast
  neighbor 172.24.25.207 activate
  neighbor 172.24.25.207 route-map 172.24.25.201-in in
  neighbor 172.24.25.208 activate
  neighbor 172.24.25.208 route-map 172.24.25.202-in in
 exit-address-family
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
line vty
!
end
TrekkieCoder commented 12 months ago

Summarizing overall discussion. The overall topology of OP is as follows:

kube-loxilb-activeactive

  1. No need to run FRR or anything on loxilb node as loxilb includes its own bgp
  2. Just need to enable BGP in the kube-loxilb yaml 
    args:
    - --loxiURL=http://172.24.25.207:11111,http://172.24.25.208:11111,http://172.24.25.209:11111
    - --externalCIDR=20.20.20.0/24
    #- --externalSecondaryCIDRs=124.124.124.1/24,125.125.125.1/24
    - --monitor=true
    - --setUniqueIP=true
    #- --setRoles=0.0.0.0
    - --setBGP=64505
    - --extBGPPeers=172.24.25.210:64506
    - --setLBMode=2
  3. BGP setup would rather be necessary in client side (router/host). Need to choose "bgp" software that can support ecmp.
  4. Usually using loxilb node as a traffic source itself has some problem because local traffic is not subject to LB rules in case of loxilb implementation.
  5. There is a trade off between active-standby vs active-active depending on performance/availability/complexity considerations. Loxilb supports both modes but finally the user has to decide which mode to use as per its requirements.
nik-netlox commented 12 months ago

One more point: You have to run LoxiLB with additional "-b" option, e.g: sudo docker run -u root --cap-add SYS_ADMIN --restart unless-stopped \ --privileged -dit -v /dev/log:/dev/log \ --net=host \ --name loxilb ghcr.io/loxilb-io/loxilb:latest -b