Allow custom endpoint to be used

[jamieps]

We use Skylable to provide an S3-compatible local storage cluster, as we don't want to store our data in AWS.

In order to use it instead of AWS, we need to be able to set a custom endpoint to use (rather than having it determined automatically).

[fellenberger]

Noticed that the code makes reference to the following private AWS method..

amazonS3 = createAmazonS3Client(this);

Was hoping I could just find a way to force communication to our local storage cluster (using hosts file as an example), but it seems like this wouldn't work? We are just about to license Rundeck Pro and would also like to be able to hook in to our own local storage cluster to support our clustered Rundeck deployment strategy.

Hoping this enhancement gets some attention!

Thanks.

[gschueler]

@fellenberger you should be able to do this. Set the framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.endpoint=https://your-host.com/path config value.

the createAmazonS3Client is not a private method, it simply creates the AmazonS3Client object: https://github.com/rundeck-plugins/rundeck-s3-log-plugin/blob/master/src/main/java/org/rundeck/plugins/S3LogFileStoragePlugin.java#L166

[fellenberger]

The plugin logic looks to be forming the FQDN of the endpoint by prepending [amazon-s3.bucket] to the [amazon-s3.endpoint] value. I understand that this is probably default behavior for the AWS solution, but not for Optums internal Object Storage. Access to our internal Object Storage is facilitated by the following construct:

https://{endpoint}/{bucket}/path/{object-name}

The error I am observing in catalina.out:

2016-05-23 13:47:42,631 ERROR LogFileStorageService - Storage request [ID#22:*] error: Unable to execute HTTP request: dev-ois-js.server.domain.com

NOTE: I also wanted to confirm that this plugin is able to support HTTPS communication to our object store per the endpoint specified below?

My framework config:

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.endpoint=https://server.domain.com framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket=dev-ois-js

Thanks.

[gschueler]

The AWS client library must be doing that by default. It looks like there is a way to set it to use "path style" endpoint requests, we could add that as another config option.

As a workaround, I wonder if you can try specifying the IP address instead of the Hostname? I suppose that wouldn't work for HTTPS access, but I see in the code that if it detects an IP address it will force the path style instead of DNS style for bucket access.

[fellenberger]

Yep, wont work for https access and we don't have http exposed to even try.

Tried specifying IP in the config and see the following error in catalina.out:

SEVERE: Unable to execute HTTP request: hostname in certificate didn't match: != com.amazonaws.AmazonClientException: Unable to execute HTTP request: hostname in certificate didn't match: !=

[gschueler]

ok i will add a flag to let you enable that feature, then you can try it

[elvin159]

Just wanted to say this is an invaluable feature! thank you

[willianantunes]

An example of using the variable pathStyle being used in a Rundeck version 5.7:

https://github.com/willianantunes/tutorials/commit/87d386e85b5bf4a5ab50c9d50e33e51d08e6dea5