Using multiple featureflagsources with flagd-proxy provider results in crashloop of the sidecar

[Oro]

Note that using the same example but with the kubernetes provider works fine. The proxy works fine with exactly one openfeature.dev/featureflagsource, but crashes with 2. Reproduced with 0.5.4 and 0.6.0 version of the helm chart at https://open-feature.github.io/open-feature-operator/

apiVersion: core.openfeature.dev/v1beta1
kind: FeatureFlag
metadata:
  name: orotest1
spec:
  flagSpec:
    flags:
      bar:
        defaultVariant: "false"
        state: ENABLED
        variants:
          "false": false
          "true": true
---
apiVersion: core.openfeature.dev/v1beta1
kind: FeatureFlag
metadata:
  name: orotest2
spec:
  flagSpec:
    flags:
      foo:
        defaultVariant: "false"
        state: ENABLED
        variants:
          "false": false
          "true": true
---
apiVersion: core.openfeature.dev/v1beta1
kind: FeatureFlagSource
metadata:
  name: orotest1
spec:
  sources:
  - provider: flagd-proxy
    source: default/orotest1
---
apiVersion: core.openfeature.dev/v1beta1
kind: FeatureFlagSource
metadata:
  name: orotest2
spec:
  sources:
  - provider: flagd-proxy
    source: default/orotest2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-busybox-curl-app
  template:
    metadata:
      labels:
        app: my-busybox-curl-app
      annotations:
        # here are the annotations for OpenFeature Operator
        openfeature.dev/enabled: "true"
        openfeature.dev/featureflagsource: "default/orotest1, default/orotest2"
    spec:
      containers:
        - name: busybox
          image: yauritux/busybox-curl:latest
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
          args:
            - sleep
            - "10000"

flagd 
flagd          ______   __       ________   _______    ______      
flagd         /_____/\ /_/\     /_______/\ /______/\  /_____/\     
flagd         \::::_\/_\:\ \    \::: _  \ \\::::__\/__\:::_ \ \    
flagd          \:\/___/\\:\ \    \::(_)  \ \\:\ /____/\\:\ \ \ \   
flagd           \:::._\/ \:\ \____\:: __  \ \\:\\_  _\/ \:\ \ \ \  
flagd            \:\ \    \:\/___/\\:.\ \  \ \\:\_\ \ \  \:\/.:| | 
flagd             \_\/     \_____\/ \__\/\__\/ \_____\/   \____/_/                                                         
flagd 
flagd {"level":"info","ts":"2024-06-04T14:28:31.719Z","caller":"cmd/start.go:100","msg":"flagd version: v0.9.0 (534b5bf654384689964c0bab5f543457d29dab8f), built at: 2024-02-20","component":"start"}
flagd {"level":"info","ts":"2024-06-04T14:28:31.720Z","caller":"flag-evaluation/connect_service.go:243","msg":"metrics and probes listening at 8014","component":"service"}
flagd {"level":"info","ts":"2024-06-04T14:28:31.721Z","caller":"flag-evaluation/connect_service.go:223","msg":"Flag Evaluation listening at :8013","component":"service"}
flagd {"level":"error","ts":"2024-06-04T14:28:31.734Z","caller":"grpc/grpc_sync.go:82","msg":"error fetching all flags: rpc error: code = Unknown desc = error fetching all flags from sync store: unrecognized URI: ","component":"sync","sync":"grpc","stacktrace":"github.com/open-feature/flagd/core/pkg/sync/grpc.(*Sync).ReSync\n\t/src/core/pkg/sync/grpc/grpc_sync.go:82\ngithub.com/open-feature/flagd/core/pkg/runtime.(*Runtime).Start.func1.1.1\n\t/src/core/pkg/runtime/runtime.go:57\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75"}
flagd {"level":"warn","ts":"2024-06-04T14:28:31.734Z","caller":"grpc/grpc_sync.go:111","msg":"error with stream listener: error receiving payload from stream: rpc error: code = Canceled desc = context canceled","component":"sync","sync":"grpc"}
flagd {"level":"warn","ts":"2024-06-04T14:28:31.734Z","caller":"grpc/grpc_sync.go:111","msg":"error with stream listener: error receiving payload from stream: rpc error: code = Canceled desc = context canceled","component":"sync","sync":"grpc"}
flagd {"level":"error","ts":"2024-06-04T14:28:31.734Z","caller":"grpc/grpc_sync.go:82","msg":"error fetching all flags: rpc error: code = Canceled desc = context canceled","component":"sync","sync":"grpc","stacktrace":"github.com/open-feature/flagd/core/pkg/sync/grpc.(*Sync).ReSync\n\t/src/core/pkg/sync/grpc/grpc_sync.go:82\ngithub.com/open-feature/flagd/core/pkg/runtime.(*Runtime).Start.func1.1.1\n\t/src/core/pkg/runtime/runtime.go:57\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75"}
flagd {"level":"info","ts":"2024-06-04T14:28:31.734Z","caller":"runtime/runtime.go:89","msg":"Shutting down server...","component":"runtime"}
flagd {"level":"info","ts":"2024-06-04T14:28:31.734Z","caller":"runtime/runtime.go:91","msg":"Server successfully shutdown.","component":"runtime"}
flagd {"level":"fatal","ts":"2024-06-04T14:28:31.734Z","caller":"cmd/start.go:138","msg":"errgroup closed with error: error resyncing sources: error fetching all flags: rpc error: code = Unknown desc = error fetching all flags from sync store: unrecognized URI: ","component":"start","stacktrace":"github.com/open-feature/flagd/flagd/cmd.init.func1\n\t/src/flagd/cmd/start.go:138\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\ngithub.com/open-feature/flagd/flagd/cmd.Execute\n\t/src/flagd/cmd/root.go:37\nmain.main\n\t/src/flagd/main.go:30\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
Stream closed EOF for default/busybox-curl-6b94997689-6xtrj (flagd)

Proxy logs:

2024-06-04T14:28:30.558Z    error    subscriptions/manager.go:193    unable to build sync from URI for target : unrecognized URI:
github.com/open-feature/flagd/core/pkg/subscriptions.(*Coordinator).watchResource
    /src/core/pkg/subscriptions/manager.go:193

[Kavindu-Dodan]

I can recreate this and seems to be a bug in proxy/flagd

[Kavindu-Dodan]

This is a bug in flagd's flag merging and resync strategy. I will work on a fix and track here to update with OFO availability.

[Kavindu-Dodan]

I have opened a fix for this in flagd - https://github.com/open-feature/flagd/pull/1322 . If you want to try out OpenFeature Operator, please use sidecar image replacement with a fixed flagd image (pushed by myself)

helm upgrade --install openfeature openfeature/open-feature-operator --set sidecarConfiguration.image.tag=1,sidecarConfiguration.image.repository=kavindudodanduwa/flagd

[toddbaert]

I will include this in the next operator release, which should be today once we release the fix in flagd.

[Kavindu-Dodan]

@Oro we released OFO version 0.6.1 with a fix. Please update your helm charts to get the latest and write back if you find any other issue :)