Closed mboersma closed 7 years ago
@kmala, @bacongobbler and @paulczar are potential reviewers of this pull request based on my analysis of git blame
information. Thanks @mboersma!
This does seem to work for bucket create, but the next frontier to fix is this:
DETAIL: Uploading "pg_xlog/000000010000000000000002" to "s3://mb-dbbucket-boss/wal_005/000000010000000000000002.lzo".
STRUCTURED: time=2016-12-02T20:31:27.255256-00 pid=411 action=push-wal key=s3://mb-dbbucket-boss/wal_005/000000010000000000000002.lzo prefix= seg=000000010000000000000002 state=begin
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/gevent/greenlet.py", line 534, in run
result = self._run(*self.args, **self.kwargs)
File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/upload.py", line 53, in __call__
self.gpg_key_id)
File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
k = blobstore.uri_put_file(creds, url, tf)
File "/usr/local/lib/python3.5/dist-packages/wal_e/blobstore/s3/s3_util.py", line 57, in uri_put_file
k.set_contents_from_file(fp, encrypt_key=True)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 750, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
query_args=query_args
File "/usr/local/lib/python3.5/dist-packages/boto/s3/connection.py", line 672, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 940, in _mexe
request.body, request.headers)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 884, in sender
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>mb-dbbucket-boss</Bucket><Endpoint>s3.amazonaws.com</Endpoint><RequestId>9CB2D73A198ED2E1</RequestId><HostId>OTGShS4THp86IvqEKaLftfbpF7Q3Z3w17qBnOmDtcCDONTij95miu5C9Eoi5BZc8wCBs1DZjovc=</HostId></Error>
<Greenlet at 0x7f02cf988d58: <wal_e.worker.upload.WalUploader object at 0x7f02cf8efcc0>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with S3ResponseError
wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling
DETAIL: Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/wal_e/cmd.py", line 617, in main
concurrency=args.pool_size)
File "/usr/local/lib/python3.5/dist-packages/wal_e/operator/backup.py", line 283, in wal_archive
group.join()
File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
raise val
File "/usr/local/lib/python3.5/dist-packages/gevent/greenlet.py", line 534, in run
result = self._run(*self.args, **self.kwargs)
File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/upload.py", line 53, in __call__
self.gpg_key_id)
File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
k = blobstore.uri_put_file(creds, url, tf)
File "/usr/local/lib/python3.5/dist-packages/wal_e/blobstore/s3/s3_util.py", line 57, in uri_put_file
k.set_contents_from_file(fp, encrypt_key=True)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 750, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
query_args=query_args
File "/usr/local/lib/python3.5/dist-packages/boto/s3/connection.py", line 672, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 940, in _mexe
request.body, request.headers)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 884, in sender
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>mb-dbbucket-boss</Bucket><Endpoint>s3.amazonaws.com</Endpoint><RequestId>9CB2D73A198ED2E1</RequestId><HostId>OTGShS4THp86IvqEKaLftfbpF7Q3Z3w17qBnOmDtcCDONTij95miu5C9Eoi5BZc8wCBs1DZjovc=</HostId></Error>
this should close #173
my bad. Will remove my commit and create a new PR for my changes.
Workflow v2.8.0 and earlier could mistakenly create the database S3 bucket in
us-east-1
regardless of what region was specified. That behavior is fixed now, but the fix could cause access problems when upgrading.This retries in "us-east-1" on a bucket create error to be backward-bug-compatible with existing Workflow installations.