[DISCUSSION] Should flagd support S3-compatible storage providers

[agardnerIT]

Requirements

Can / should flagd be extended to support s3-compatible storage services.

These would include:

• S3 (obviously)
• DigitalOcean spaces
• MinIO
• Cloudflare R2
• any other existing or future S3 compatible provider

While flagd already supports the Authorisation header, the difficulty seems to be that S3 auth is built from a dynamically generated (HMAC) header. Presumably calculating this would mean enhancements to flagd at the core.

Signature Timeouts

Amazon's documentation states that:

• Signatures last for 7 days (meaning flagd would need a restart (to regen the signature) at least every 7 days)

• A presigned URL can be valid for a maximum of seven days because the signing key you use in signature calculation is valid for up to seven days.

References

• https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
• https://docs.digitalocean.com/reference/api/spaces-api/

[beeme1mr]

Thanks for starting the discussion @agardnerIT. Let's leave this open for a bit to gauge interest. For those who are interested, please add a comment describing your use case and why an S3-compatible sync provider would address it.

[agardnerIT]

I'm going to close this as there is no immediate interest / need. We can re-open if someone else is interested in future.

My usecase was to store / retrieve flag values from cheap object storage like S3 and R2 etc.

[austindrenski]

Hey folks, just getting started with OpenFeature and flagd after reading about it in the CNCF announcement blog post over the holidays, and I've pretty quickly found myself yearning for built-in support for S3-compatible storage.

I was very excited to learn of this project, and I suspect there will be many more newcomers in the coming months looking to trial flagd in non-critical production workloads (as many have with otelcol over the last few years).

Far from a blocker to getting started, but in-the-box support would certainly help lower the barrier to entry for curious onlookers such as myself.

[beeme1mr]

Hey @austindrenski, thanks for letting us know you would like an S3-compatible sync. It would be great if we could figure out a clean way to sync with S3 via HTTP polling. That way, we could use the existing HTTP sync and avoid adding additional complexity and dependency. You could probably get it working without any code changes if you made the S3 bucket public, but that's likely not a great long-term solution.

I'll look into this a bit. It will likely come down to supporting the authentication mechanism required by AWS.

FYI @toddbaert

[beeme1mr]

It looks like Open Policy Agent has done something similar already.

https://github.com/open-policy-agent/opa/pull/5489

[beeme1mr]

I'm going to reopen this discussion.

[austindrenski]

You could probably get it working without any code changes if you made the S3 bucket public, but that's likely not a great long-term solution.

That's actually what I tried after posting here earlier and, after some tinkering, I got things working without making the bucket entirely public (i.e. anonymous access within my VPC, no public access outside of it) by using Block all public access + a bucket policy along the lines of:

// sharing for future readers, but be sure to consult your friendly
// neighborhood SecOps team before yeeting this anywhere near prod!
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "s3:GetObject",
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "vpce-***"
        }
      },
      "Effect": "Allow",
      "Principal": "*",
      "Resource": "arn:aws:s3:::open-feature/flagd/*"
    }
  ]
}

It would be great if we could figure out a clean way to sync with S3 via HTTP polling. That way, we could use the existing HTTP sync and avoid adding additional complexity and dependency. [...]

+1, skimmed through the sync providers earlier and came to the same conclusion of an s3 sync provider wrapping the http sync provider, but implementing the auth part without a direct dependency on (at least) github.com/aws/aws-sdk-go-v2/config is where I stumbled out of my depth.

I'll look into this a bit. It will likely come down to supporting the authentication mechanism required by AWS.

Really appreciate you looking into this, on whatever timetable you're able to.

I think finding a good answer for how to handle auth for AWS et al. will be important if flagd plans to support other storage backends in the future, which isn't to say it needs to, but given the success of otelcol and otelcol-contrib, I'm guessing there may be a similar push for broad, plug-and-play support from flagd (or some future flagd-contrib distro).

For example, while we can use S3-based flag definitions with a little bit of work today, it would be a neat option in the long-term to integrate with DynamoDB (or some other general object/document store) for read/write scenarios. (This is presumably far out of scope for now, but the building blocks here are pretty exciting.)

[toddbaert]

You could probably get it working without any code changes if you made the S3 bucket public, but that's likely not a great long-term solution.

That's actually what I tried after posting here earlier and, after some tinkering, I got things working without making the bucket entirely public (i.e. anonymous access within my VPC, no public access outside of it) by using Block all public access + a bucket policy along the lines of:

// sharing for future readers, but be sure to consult your friendly
// neighborhood SecOps team before yeeting this anywhere near prod!
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "s3:GetObject",
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "vpce-***"
        }
      },
      "Effect": "Allow",
      "Principal": "*",
      "Resource": "arn:aws:s3:::open-feature/flagd/*"
    }
  ]
}

It would be great if we could figure out a clean way to sync with S3 via HTTP polling. That way, we could use the existing HTTP sync and avoid adding additional complexity and dependency. [...]

+1, skimmed through the sync providers earlier and came to the same conclusion of an s3 sync provider wrapping the http sync provider, but implementing the auth part without a direct dependency on (at least) github.com/aws/aws-sdk-go-v2/config is where I stumbled out of my depth.

I'll look into this a bit. It will likely come down to supporting the authentication mechanism required by AWS.

Really appreciate you looking into this, on whatever timetable you're able to.

I think finding a good answer for how to handle auth for AWS et al. will be important if flagd plans to support other storage backends in the future, which isn't to say it needs to, but given the success of otelcol and otelcol-contrib, I'm guessing there may be a similar push for broad, plug-and-play support from flagd (or some future flagd-contrib distro).

For example, while we can use S3-based flag definitions with a little bit of work today, it would be a neat option in the long-term to integrate with DynamoDB (or some other general object/document store) for read/write scenarios. (This is presumably far out of scope for now, but the building blocks here are pretty exciting.)

Glad to hear you had a degree of success!

Agreed on all counts. @beeme1mr and I had a brief chat about this, and basically we came to the exact same conclusions, and share the same questions and goals.

[beeme1mr]

I've created an issue to add s3 support. https://github.com/open-feature/flagd/issues/1376