Support custom S3 endpoint URIs

[mikeseese]

To be able to use non-AWS S3 buckets (Cloudflare R2, DigitalOcean Spaces, etc), you need to specify a custom endpoint URI. Unfortunately, the AWS SDK doesn't provide an environment or config file variable to override this value (https://github.com/aws/aws-cli/issues/4454); it only supports a CLI parameter or using the direct SDK variable). So for longtail to support these custom endpoints, the SDK references would need to be changed

Using the docs here, here, here, and here, I believe that https://github.com/DanEngelbrecht/golongtail/blob/883905db7a273e0bd58bc3c5cdb5cf98eda3d8e5/longtailstorelib/s3Store.go#L60 can be modified to look something like (please excuse my lack of golang experience):

client := s3.NewFromConfig(cfg, func(o *s3.Options) {
    if ConfiguredBaseEndpointUri != nil {
        o.EndpointResolver := s3.EndpointResolverFromURL(ConfiguredBaseEndpointUri)
    }
})

or

if ConfiguredBaseEndpointUri != nil {
    cfg.EndpointResolver := s3.EndpointResolverFromURL(ConfiguredBaseEndpointUri)
}
client := s3.NewFromConfig(cfg)

where ConfiguredBaseEndpointUri could be some new CLI flag (i.e. --s3-base-endpoint).

It's also very possible that it's as simple as the below (I've seen another example say aws.Config has Endpoint, but the official docs say otherwise)

if ConfiguredBaseEndpointUri != nil {
    cfg.Endpoint := ConfiguredBaseEndpointUri
}
client := s3.NewFromConfig(cfg)

Once the endpoint can be configured, users can configure the region and credentials using the ~/.aws/config and ~/.aws/credentials files mentioned in longtail's readme or use AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables (which get read in during the LoadDefaultConfig call, so there's no reason to make any changes to support those variables.

I would attempt to implement this myself and make a PR, but my golang experience is measured in weeks and I would likely just be going down a rabbithole for simple issues

[DanEngelbrecht]

Thanks for the info, I'll take a peek once time permits, hopefully within a week or two. I don't have any of the non-AWS S3 buckets so testing might be an issue, would you be willing to test some stuff out once I have something @seesemichaelj ?

[mikeseese]

Absolutely!

[DanEngelbrecht]

@seesemichaelj Please try out https://github.com/DanEngelbrecht/golongtail/releases/tag/v0.3.4-pre14 and see how the new option feels and works out.,

[mikeseese]

@DanEngelbrecht I can confirm that v0.3.4-pre14 works with DigitalOcean S3 Spaces using the --s3-endpoint-resolver-uri https://sfo3.digitaloceanspaces.com as the endpoint configuration (and specifying the region, key, and secret variables via AWS config files and environment variables). Awesome, thank you!