search

deis / postgres

A PostgreSQL database used by Deis Workflow.
https://deis.com
MIT License
36 stars 22 forks source link

fix(create_bucket): try default s3 region on create error #175

Closed mboersma closed 7 years ago

mboersma commented 7 years ago

Workflow v2.8.0 and earlier could mistakenly create the database S3 bucket in us-east-1 regardless of what region was specified. That behavior is fixed now, but the fix could cause access problems when upgrading.

This retries in "us-east-1" on a bucket create error to be backward-bug-compatible with existing Workflow installations.

deis-bot commented 7 years ago

@kmala, @bacongobbler and @paulczar are potential reviewers of this pull request based on my analysis of git blame information. Thanks @mboersma!

mboersma commented 7 years ago

This does seem to work for bucket create, but the next frontier to fix is this:

       DETAIL: Uploading "pg_xlog/000000010000000000000002" to "s3://mb-dbbucket-boss/wal_005/000000010000000000000002.lzo".
        STRUCTURED: time=2016-12-02T20:31:27.255256-00 pid=411 action=push-wal key=s3://mb-dbbucket-boss/wal_005/000000010000000000000002.lzo prefix= seg=000000010000000000000002 state=begin
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gevent/greenlet.py", line 534, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/upload.py", line 53, in __call__
    self.gpg_key_id)
  File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
    k = blobstore.uri_put_file(creds, url, tf)
  File "/usr/local/lib/python3.5/dist-packages/wal_e/blobstore/s3/s3_util.py", line 57, in uri_put_file
    k.set_contents_from_file(fp, encrypt_key=True)
  File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 750, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/usr/local/lib/python3.5/dist-packages/boto/s3/connection.py", line 672, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 940, in _mexe
    request.body, request.headers)
  File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 884, in sender
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>mb-dbbucket-boss</Bucket><Endpoint>s3.amazonaws.com</Endpoint><RequestId>9CB2D73A198ED2E1</RequestId><HostId>OTGShS4THp86IvqEKaLftfbpF7Q3Z3w17qBnOmDtcCDONTij95miu5C9Eoi5BZc8wCBs1DZjovc=</HostId></Error>
<Greenlet at 0x7f02cf988d58: <wal_e.worker.upload.WalUploader object at 0x7f02cf8efcc0>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with S3ResponseError

wal_e.main   CRITICAL MSG: An unprocessed exception has avoided all error handling
        DETAIL: Traceback (most recent call last):
          File "/usr/local/lib/python3.5/dist-packages/wal_e/cmd.py", line 617, in main
            concurrency=args.pool_size)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/operator/backup.py", line 283, in wal_archive
            group.join()
          File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join
            raise val
          File "/usr/local/lib/python3.5/dist-packages/gevent/greenlet.py", line 534, in run
            result = self._run(*self.args, **self.kwargs)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/upload.py", line 53, in __call__
            self.gpg_key_id)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put
            k = blobstore.uri_put_file(creds, url, tf)
          File "/usr/local/lib/python3.5/dist-packages/wal_e/blobstore/s3/s3_util.py", line 57, in uri_put_file
            k.set_contents_from_file(fp, encrypt_key=True)
          File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
            chunked_transfer=chunked_transfer, size=size)
          File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 750, in send_file
            chunked_transfer=chunked_transfer, size=size)
          File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
            query_args=query_args
          File "/usr/local/lib/python3.5/dist-packages/boto/s3/connection.py", line 672, in make_request
            retry_handler=retry_handler
          File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 1071, in make_request
            retry_handler=retry_handler)
          File "/usr/local/lib/python3.5/dist-packages/boto/connection.py", line 940, in _mexe
            request.body, request.headers)
          File "/usr/local/lib/python3.5/dist-packages/boto/s3/key.py", line 884, in sender
            response.status, response.reason, body)
        boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently
        <?xml version="1.0" encoding="UTF-8"?>
        <Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>mb-dbbucket-boss</Bucket><Endpoint>s3.amazonaws.com</Endpoint><RequestId>9CB2D73A198ED2E1</RequestId><HostId>OTGShS4THp86IvqEKaLftfbpF7Q3Z3w17qBnOmDtcCDONTij95miu5C9Eoi5BZc8wCBs1DZjovc=</HostId></Error>
bacongobbler commented 7 years ago

this should close #173

bacongobbler commented 7 years ago

my bad. Will remove my commit and create a new PR for my changes.