search

dynatrace-oss / progressiveDelivery-masterclass

Apache License 2.0
4 stars 11 forks source link

flagd panic causes crashloop #79

Closed agardnerIT closed 1 year ago

agardnerIT commented 1 year ago 
        /_____/\ /_/\     /_______/\ /______/\  /_____/\
        \::::_\/_\:\ \    \::: _  \ \\::::__\/__\:::_ \ \
         \:\/___/\\:\ \    \::(_)  \ \\:\ /____/\\:\ \ \ \
          \:::._\/ \:\ \____\:: __  \ \\:\\_  _\/ \:\ \ \ \
           \:\ \    \:\/___/\\:.\ \  \ \\:\_\ \ \  \:\/.:| |
            \_\/     \_____\/ \__\/\__\/ \_____\/   \____/_/

2023-08-04T03:55:21.374Z    info    cmd/start.go:117    flagd version: v0.5.2 (0fbd30f4b3ba25d3e8adf457d2708cd0990b68d8), built at: 2023-04-13  {"component": "start"}
2023-08-04T03:55:21.374Z    warn    telemetry/builder.go:59 skipping trace provider setup as collector target is not set. Traces will use NoopTracerProvider provider
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1668f90]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:114
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc0004c5c80, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:95 +0x6f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0004c48b8, 0x15}, {0xc0001268a0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:49 +0x125
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0xc0005deac0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:328 +0x3de
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc0005deef0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:203 +0x65
k8s.io/client-go/discovery.ServerGroupsAndResources({0x1dabc40, 0xc0003721e0})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:413 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:376 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0005def08)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:651 +0x71
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:375 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x1dabc40?, 0xc0003721e0?})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:94 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:130
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000129d40?, {0x0, 0x0, 0x1b01e4c?})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:110 +0x182
sigs.k8s.io/controller-runtime/pkg/client.newClient(0xc000129d40?, {0xc0004ec000?, {0x0?, 0x0?}, {0x20?, 0x0?}})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/client.go:109 +0x1d1
sigs.k8s.io/controller-runtime/pkg/client.New(...)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/client.go:77
github.com/open-feature/flagd/core/pkg/sync/kubernetes.GetClients()
    /workspace/core/pkg/sync/kubernetes/kubernetes_sync.go:63 +0x45
github.com/open-feature/flagd/core/pkg/runtime.NewK8s({0xc0004ab620, 0x24}, 0xc0005df8b0)
    /workspace/core/pkg/runtime/from_config.go:204 +0x45
github.com/open-feature/flagd/core/pkg/runtime.syncProvidersFromConfig(0xc0005df8b0, {0xc0000d1260?, 0x1, 0x0?})
    /workspace/core/pkg/runtime/from_config.go:152 +0xb33
github.com/open-feature/flagd/core/pkg/runtime.FromConfig(0xc000355170, {0x1d83bf8, 0x6}, {{0x0, 0x0}, 0x1f4e, {0x0, 0x0}, {0x0, 0x0}, ...})
    /workspace/core/pkg/runtime/from_config.go:120 +0x6e5
github.com/open-feature/flagd/flagd/cmd.glob..func1(0x2ac3f80?, {0x1ae70fc?, 0x2?, 0x2?})
    /workspace/flagd/cmd/start.go:154 +0x925
github.com/spf13/cobra.(*Command).execute(0x2ac3f80, {0xc000572860, 0x2, 0x2})
    /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x2ac3ca0)
    /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
    /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/open-feature/flagd/flagd/cmd.Execute({0x1d83bf8?, 0xc000012018?}, {0x1d95800?, 0x1d87760?}, {0x1d84150?, 0x0?})
    /workspace/flagd/cmd/root.go:37 +0xaa
main.main()
    /workspace/flagd/main.go:30 +0xd7
agardnerIT commented 1 year ago

Updating helm chart to v0.2.35 causes panic in open-feature-operator-controller-manager:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1302f10]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:114
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc0000e3d28, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:95 +0x6f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0000e20d8, 0x15}, {0xc0000f0fa0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/aggregated_discovery.go:49 +0x125
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0xc000398878?)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:328 +0x3de
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc000398ca8?)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:203 +0x65
k8s.io/client-go/discovery.ServerGroupsAndResources({0x1aefcf8, 0xc000279d70})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:413 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:376 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc000440cc0)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:651 +0x71
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.3/discovery/discovery_client.go:375 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x1aefcf8?, 0xc000279d70?})
    /go/pkg/mod/k8s.io/client-go@v0.26.3/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:94 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:130
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc0000f8120?, {0x0, 0x0, 0x47f4d7c94faa9501?})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/client/apiutil/dynamicrestmapper.go:110 +0x182
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc0004ff260?)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc0002ea000, {0xc000399660, 0x1, 0x0?})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc0004ff260, 0x0, 0x0, {{0x1aec038, 0xc00041f540}, 0x0}, 0x1, {0x0, 0x0}, ...})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/manager.go:351 +0xf9
main.main()
    /workspace/main.go:173 +0x1275
agardnerIT commented 1 year ago

Furthermore, this causes the demo app to crash. It seems that using gRPC there is a hard dependancy between the demo app and flagd:

Demo app logs:

Connecting to flagD at localhost:8030
Example app listening on port 3000
/usr/src/app/node_modules/@openfeature/flagd-provider/index.cjs:1533
            reject(new Error(errorMessage));
                   ^

Error: FlagdProvider: max stream connect attempts (5 reached)
    at GRPCService.handleError (/usr/src/app/node_modules/@openfeature/flagd-provider/index.cjs:1533:20)
    at /usr/src/app/node_modules/@openfeature/flagd-provider/index.cjs:1476:22
    at /usr/src/app/node_modules/@protobuf-ts/runtime-rpc/build/commonjs/rpc-output-stream.js:86:36
    at Array.forEach (<anonymous>)
    at RpcOutputStreamController.notifyError (/usr/src/app/node_modules/@protobuf-ts/runtime-rpc/build/commonjs/rpc-output-stream.js:86:23)
    at ClientReadableStreamImpl.<anonymous> (/usr/src/app/node_modules/@protobuf-ts/grpc-transport/build/commonjs/grpc-transport.js:90:27)
    at ClientReadableStreamImpl.emit (node:events:513:28)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:351:28)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at /usr/src/app/node_modules/@grpc/grpc-js/build/src/resolving-call.js:94:78
agardnerIT commented 1 year ago

Raised an issue in the OFO repo: https://github.com/open-feature/open-feature-operator/issues/515

heckelmann commented 1 year ago

Could you please test it with Kind and K8s v. 1.26? Just to double check before releasing a new flagd and ofo version?

Kavindu-Dodan commented 1 year ago

We have a new release of the OFO v0.2.36 with updated K8s dependencies ^1 (k8s.io/client-go v0.26.4)

@agardnerIT could you please verify the mentioned issue with this new OFO version?

agardnerIT commented 1 year ago

I've tested this and bumping OFO to v0.2.36 solves this issue. Thanks!