Shards intermittently failing when snapshotting to S3

[BrandesEric]

Hey guys, we've been using the AWS ES plugin for some time with no issues. In the past few months we've started seeing random shard failures when doing snapshots. The problem appears to be getting worse. Here is the error we see:

{"node_id":"JIXdRZ6pRBqbuSKzz4TlXA","index":"errors","reason":"IndexShardSnapshotFailedException[[errors][8] Failed to perform snapshot (index files)]; nested: IOException[Unable to upload object Prod/indices/errors/8/__7i]; nested: AmazonClientException[Unable to execute HTTP request: our_bucket.s3.amazonaws.com]; nested: UnknownHostException[our_bucket.s3.amazonaws.com]; ","shard_id":8,"status":"INTERNAL_SERVER_ERROR"}

We're snapshotting ~70 shards and maybe 4-10 experience this issue during a snapshot. Any ideas what's going on?

[BrandesEric]

I've been investigating further, and no matter what I do, it always tries to use bucket.s3.amazon.com instead of s3.amazon.com/bucket. Is there any way for me to specify the latter url structure? GIven this issue seems DNS related, I'm grasping at straws but want to try the non-vanity domain name.

[dadoonet]

I guess you could use "cloud.aws.s3.endpoint"?

Would this work for you?

[BrandesEric]

I tried that - and it was still prefixing the bucket name, as a subdomain, rather than appending it as a url segment.

On Sat, Mar 7, 2015 at 1:36 PM, David Pilato notifications@github.com wrote:

I guess you could use "cloud.aws.s3.endpoint"?

Would this work for you?

— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/189#issuecomment-77705875 .

[dadoonet]

I see. In that case bucket name should may be not mandatory So you can keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as s3 and base_path with your bucket name?

[BrandesEric]

Now we're talking :) I'm trying one additional DNS tweak but if that doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato notifications@github.com wrote:

I see. In that case bucket name should may be not mandatory So you can set keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as s3 and base_path with your bucket name?

— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/189#issuecomment-77706562 .

[BrandesEric]

{

"error": "RemoteTransportException[[Warpath][inet[/10.4.0.5:9300]][cluster:admin/repository/put]];

nested: RepositoryException[[Prod_2015-03-08T00:36:17.4724345+00:00] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.IllegalArgumentException: Bucket name should be between 3 and 63 characters long\n at org.elasticsearch.repositories.s3.S3Repository.()\n at org.elasticsearch.repositories.s3.S3Repository\n at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: IllegalArgumentException[Bucket name should be between 3 and 63 characters long]; ",

"status": 500

}

On Sat, Mar 7, 2015 at 2:06 PM, Eric Brandes eric.brandes@gmail.com wrote:

Now we're talking :) I'm trying one additional DNS tweak but if that doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato notifications@github.com wrote:

I see. In that case bucket name should may be not mandatory So you can set keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as s3 and base_path with your bucket name?

— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/189#issuecomment-77706562 .

[BrandesEric]

Oops, send got clicked before I was done :)

I'm going to try a different region, see if I can get something longer than 3 chars.

On Sat, Mar 7, 2015 at 6:38 PM, Eric Brandes eric.brandes@gmail.com wrote:

{

"error": "RemoteTransportException[[Warpath][inet[/10.4.0.5:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[Prod_2015-03-08T00:36:17.4724345+00:00] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.IllegalArgumentException: Bucket name should be between 3 and 63 characters long\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: IllegalArgumentException[Bucket name should be between 3 and 63 characters long]; ",

"status": 500

}

On Sat, Mar 7, 2015 at 2:06 PM, Eric Brandes eric.brandes@gmail.com wrote:

Now we're talking :) I'm trying one additional DNS tweak but if that doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato notifications@github.com wrote:

I see. In that case bucket name should may be not mandatory So you can set keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as s3 and base_path with your bucket name?

— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/189#issuecomment-77706562 .

[BrandesEric]

I give up, nothing I can set seems to change the url to the configuration I want. Ah well, have to figure something else out.

On Sat, Mar 7, 2015 at 6:41 PM, Eric Brandes eric.brandes@gmail.com wrote:

Oops, send got clicked before I was done :)

I'm going to try a different region, see if I can get something longer than 3 chars.

On Sat, Mar 7, 2015 at 6:38 PM, Eric Brandes eric.brandes@gmail.com wrote:

{

"error": "RemoteTransportException[[Warpath][inet[/10.4.0.5:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[Prod_2015-03-08T00:36:17.4724345+00:00] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.IllegalArgumentException: Bucket name should be between 3 and 63 characters long\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: IllegalArgumentException[Bucket name should be between 3 and 63 characters long]; ",

"status": 500

}

On Sat, Mar 7, 2015 at 2:06 PM, Eric Brandes eric.brandes@gmail.com wrote:

Now we're talking :) I'm trying one additional DNS tweak but if that doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato notifications@github.com wrote:

I see. In that case bucket name should may be not mandatory So you can set keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as s3 and base_path with your bucket name?

— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/189#issuecomment-77706562 .