Support non-AWS S3 endpoints

[awoods]

It would be great to be able to use ocfl-java to connect with other, non-AWS services that expose an S3 API.

[pwinckles]

@awoods It does. We use it locally against a local S3 implementation. What problem are you seeing exactly?

[awoods]

This issue was speculative. Can you point to the configuration item we should be setting?

[pwinckles]

It's not a configuration in ocfl-java. It's how you configure the S3 client. You'll want to configure it exactly like you would when you access your S3 normally. I can't say what that is exactly for your implementation. For example, for ours, we must use path-style access, which is not default, but that may or may not be the case for you.

Here's an example of how it's configured in Fedora: https://github.com/fcrepo/fcrepo/blob/main/fcrepo-persistence-ocfl/src/main/java/org/fcrepo/persistence/ocfl/impl/OcflPersistenceConfig.java#L128

[pwinckles]

@awoods I'll keep this open until I hear that you've successfully been able to connect to your S3 storage.

[awoods]

Thanks, @pwinckles . I will let you know once we have made it home safely.

[daveneiman]

@pwinckles - Getting pretty far with the above references. In using AWS CLI tools we've been told that, since we're using a self-signed cert. over HTTPS, we need to add --no-verify-ssl. Could you advise on how to configure this with the S3Client code? The error we're getting on building the s3 client is Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[pwinckles]

@daveneiman That's going to be a configuration setting on the HTTP client that the SDK uses. The SDK supports a few different options.

That said, there aren't great examples of how one might go about disabling certificate verification when using the SDK, likely because you should never do this when using AWS.

Here are a couple of configuration you could try. I have not tested them myself, so I cannot guarantee that either work.

I think the following should work if you aren't configuring a specific HTTP client:

builder.httpClient(new DefaultSdkHttpClientBuilder()
             .buildWithDefaults(AttributeMap.builder()
                     .put(SdkHttpConfigurationOption.TRUST_ALL_CERTIFICATES, true)
                     .build()));

This config should work if you're using Apache:

builder.httpClientBuilder(ApacheHttpClient.builder().tlsTrustManagersProvider(() -> {
    return new TrustManager[] {
            new X509TrustManager() {
                public void checkClientTrusted(X509Certificate[] chain, String authType)
                        throws CertificateException {
                    // noop
                }
                public void checkServerTrusted(X509Certificate[] chain, String authType)
                        throws CertificateException {
                    // noop
                }
                public X509Certificate[] getAcceptedIssuers() {
                    return null;
                }
            }
    };
}));

[daveneiman]

@pwinckles The first option worked. Thank you! Agreed that disabling cert verification isn't wise with AWS. However, this option is being used because of a self-signed cert to what I think is an on-premises ECS setup. For going to AWS s3 this option won't be used.

[awoods]

Given what @daveneiman mentioned, and the fact that indeed we are able to use ocfl-java to interact with a non-AWS service, I think this issue can be closed.