k8gb-io / k8gb

A cloud native Kubernetes Global Balancer
https://www.k8gb.io
Apache License 2.0
891 stars 94 forks source link

flaky: yet another non-deterministic tests failure #974

Closed jkremser closed 5 months ago

jkremser commented 2 years ago

sometimes our unit tests fails on this:

--- FAIL: TestCanGetExternalTargetsFromK8gbInAnotherLocation (0.01s)
    gslb_controller_reconciliation_test.go:525: 
            Error Trace:    gslb_controller_reconciliation_test.go:525
                                        fakedns.go:78
                                        gslb_controller_reconciliation_test.go:503
            Error:          Not equal: 
                            expected: []*endpoint.Endpoint{(*endpoint.Endpoint)(0xc0002bc1c0), (*endpoint.Endpoint)(0xc0002bc230)}
                            actual  : []*endpoint.Endpoint{(*endpoint.Endpoint)(0xc00010d1f0), (*endpoint.Endpoint)(0xc00010d260)}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -17,8 +17,8 @@
                               Targets: (endpoint.Targets) (len=6) {
                            +   (string) (len=8) "10.1.0.1",
                            +   (string) (len=8) "10.1.0.2",
                            +   (string) (len=8) "10.1.0.3",
                                (string) (len=8) "10.0.0.1",
                                (string) (len=8) "10.0.0.2",
                            -   (string) (len=8) "10.0.0.3",
                            -   (string) (len=8) "10.1.0.1",
                            -   (string) (len=8) "10.1.0.2",
                            -   (string) (len=8) "10.1.0.3"
                            +   (string) (len=8) "10.0.0.3"
                               },
            Test:           TestCanGetExternalTargetsFromK8gbInAnotherLocation
            Messages:       got:
                             [
                                {
                                    "dnsName": "localtargets-roundrobin.cloud.example.com",
                                    "targets": [
                                        "10.0.0.1",
                                        "10.0.0.2",
                                        "10.0.0.3"
                                    ],
                                    "recordType": "A",
                                    "recordTTL": 30
                                },
                                {
                                    "dnsName": "roundrobin.cloud.example.com",
                                    "targets": [
                            actual  : map[string][]string{"roundrobin.cloud.example.com":[]string{"10.0.0.1", "10.0.0.2", "10.0.0.3", "10.1.0.1", "10.1.0.2", "10.1.0.3"}}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -2,8 +2,8 @@
                              (string) (len=28) "roundrobin.cloud.example.com": ([]string) (len=6) {
                            +  (string) (len=8) "10.0.0.1",
                            +  (string) (len=8) "10.0.0.2",
                            +  (string) (len=8) "10.0.0.3",
                               (string) (len=8) "10.1.0.1",
                               (string) (len=8) "10.1.0.2",
                            -  (string) (len=8) "10.1.0.3",
                            -  (string) (len=8) "10.0.0.1",
                            -  (string) (len=8) "10.0.0.2",
                            -  (string) (len=8) "10.0.0.3"
                            +  (string) (len=8) "10.1.0.3"
                              }
            Test:           TestCanGetExternalTargetsFromK8gbInAnotherLocation
            Messages:       got:
                             map[roundrobin.cloud.example.com:[10.1.0.1 10.1.0.2 10.1.0.3 10.0.0.1 10.0.0.2 10.0.0.3]] Gslb Records status,

                             want:
                             map[roundrobin.cloud.example.com:[10.0.0.1 10.0.0.2 10.0.0.3 10.1.0.1 10.1.0.2 10.1.0.3]]
FakeDNS listening on port 7753 

https://github.com/k8gb-io/k8gb/actions/runs/3273083165/jobs/5384868955#step:7:819

jkremser commented 2 years ago

hmm:

            Messages:       got:
                             map[roundrobin.cloud.example.com:[10.1.0.1 10.1.0.2 10.1.0.3 10.0.0.1 10.0.0.2 10.0.0.3]] Gslb Records status,

                             want:
                             map[roundrobin.cloud.example.com:[10.0.0.1 10.0.0.2 10.0.0.3 10.1.0.1 10.1.0.2 10.1.0.3]]

looks like ordering issue to me

jkremser commented 2 years ago

another instance of the same thing (ordering of ips): https://github.com/k8gb-io/k8gb/actions/runs/3276047689/jobs/5391825487

        Messages:       got:
                             [
                                {
                                    "dnsName": "localtargets-roundrobin.cloud.example.com",
                                    "targets": [
                                        "10.0.0.1",
                                        "10.0.0.2",
                                        "10.0.0.3"
                                    ],
                                    "recordType": "A",
                                    "recordTTL": 30
                                },
                                {
                                    "dnsName": "roundrobin.cloud.example.com",
                                    "targets": [
                                        "10.1.0.1",
                                        "10.1.0.2",
                                        "10.1.0.3",
                                        "10.0.0.1",
                                        "10.0.0.2",
                                        "10.0.0.3"
                                    ],
                                    "recordType": "A",
                                    "recordTTL": 30,
                                    "labels": {
                                        "strategy": "roundRobin"
                                    }
                                }
                            ] DNSEndpoint,

                             want:
                             [
                                {
                                    "dnsName": "localtargets-roundrobin.cloud.example.com",
                                    "targets": [
                                        "10.0.0.1",
                                        "10.0.0.2",
                                        "10.0.0.3"
                                    ],
                                    "recordType": "A",
                                    "recordTTL": 30
                                },
                                {
                                    "dnsName": "roundrobin.cloud.example.com",
                                    "targets": [
                                        "10.0.0.1",
                                        "10.0.0.2",
                                        "10.0.0.3",
                                        "10.1.0.1",
                                        "10.1.0.2",
                                        "10.1.0.3"
                                    ],
                                    "recordType": "A",
                                    "recordTTL": 30,
                                    "labels": {
                                        "strategy": "roundRobin"
                                    }
                                }
                            ]
ytsarev commented 11 months ago

Can be reproduced locally by running

go test ./... -count=10