Use upstream wal-e instead of deis fork for azure/gcs

[kmala]

Since wal-e upstream has the fixes to work for both azure and gcs, change the postgres codebase to use the upstream code instead of deis fork. Since https://github.com/wal-e/wal-e/issues/197 is still not fixed, we should continue to use the fork of wal-e but with just boto-3 changes on top.

[bacongobbler]

only one change in boto2 is required for minio support so we don't have to support the boto3 fork: https://github.com/boto/boto/pull/3513.

[fdr]

Wait, you people have a boto3 patch that you know works? I would like it unless you think it has big caveats.

[bacongobbler]

@fdr I broke a ton of backwards compatibility and it has a bunch of hacky workarounds, but you can check out https://github.com/deis/wal-e/commits/boto3-upgrade which is rebased on top of 0.8.1. We've been running it for a few months in prod but it breaks a lot of existing feature flags for wal-e and I never got around to cleaning it up for a PR. Sorry!

[bacongobbler]

https://github.com/deis/wal-e/commit/e1702a36d95b0eb5a545adda266c5347840e57d6 along with the rest of the follow-up fixups makes up the meat of the work done to bring wal-e to boto3.

[fdr]

Do you know which ones? it is too late to merge it for WAL-E 1.0, but maybe 1.1, unless the behavioral changes are big enough...

[bacongobbler]

Following list of commits should do it:

https://github.com/deis/wal-e/commit/e1702a36d95b0eb5a545adda266c5347840e57d6 https://github.com/deis/wal-e/commit/2616fe8ec93c6609241a89f170f0a36d3240026a https://github.com/deis/wal-e/commit/bde415b5f86d11dbc58bc77c9fd4eafde5ddafa7 https://github.com/deis/wal-e/commit/ed8f504938dca37dc1949ffc4db0f957a431a21d

couple of caveats:

• need some way to configure the signature version, or the boto config in general. For our use case we literally hardcoded s3v4 as the signature version. Bad code, but worked.
• AWS_SECURITY_TOKEN support is dropped due to lack of time/understanding
• AWS instance profile support is dropped for the same reason above
• calling format support was dropped in boto3, so WALE_S3_ENDPOINT needs some way to enable/disable calling format handlers in boto3 (we just used the plain path style, no vhost support)
• dropped RequestTimeTooSkewed error catching because there is no equivalent in boto3 as far as I could tell

Happy to help in any way possible!

[fdr]

On Wed, Aug 17, 2016 at 2:49 PM Matthew Fisher notifications@github.com wrote:

• need some way to configure the signature version, or the boto config in general. For our use case we literally hardcoded s3v4 https://github.com/deis/wal-e/commit/faaf39492db4b3bd55b3a2de07255a63f2360c26 as the signature version. Bad code, but worked.

Not bad, but for the sake of legacy use I'd want to support older versions as a non-default if it's not terrible.

• AWS_SECURITY_TOKEN support is dropped due to lack of time/understanding

I could likely add this. I use it.

• AWS instance profile support is dropped for the same reason above

I think that's okay. It was a pain because I was quite particular about not letting the library (boto) abstractions leak to the user. That idea long predates the era of first-party SDKs such as boto3. Largely for this reason, I've changed my mind.

• calling format support was dropped, so needs some way to enable/disable calling format handlers in boto3 https://github.com/deis/wal-e/commit/e1702a36d95b0eb5a545adda266c5347840e57d6#diff-c9eb6600f87fec6e454773dfa1caa6e1R116 (we just used the plain path style, no vhost support)

Interesting. I thought virtualhosted was preferred and can have some performance advantages, but I remember having difficulty researching the issue back in the past.

• dropped RequestTimeTooSkewed error catching https://github.com/deis/wal-e/commit/e1702a36d95b0eb5a545adda266c5347840e57d6#diff-8f06e7a59b8f9bcd254918cc3128c3c3L125 because there is no equivalent in boto3 as fasr as I could tell

I think it reduces to a regular "client error". Not sure what the http status is.

[monaka]

Sorry for my long absence. I tried some recent official deis/postgres images.

Azure support in v2.2.6 may be broken. git-8448c7c is better but crashed on my host. I'm inspecting the root cause. FYI.

. stopped waiting
server is still starting up
Performing an initial backup...
wal_e.operator.backup INFO     MSG: promoted prefetched wal segment
        STRUCTURED: time=2016-11-06T05:30:42.932846-00 pid=587 action=wal-fetch key=wabs://database/wal_005/0000000900000005000000E1.lzo prefix= seg=0000000900000005000000E1
LOG:  restored log file "0000000900000005000000E1" from archive
wal_e.main   INFO     MSG: starting WAL-E
        DETAIL: The subcommand is "backup-push".
        STRUCTURED: time=2016-11-06T05:30:43.636232-00 pid=599
FATAL:  the database system is starting up
psql: FATAL:  the database system is starting up
wal_e.main   ERROR    MSG: could not run one or more external programs WAL-E depends upon
        DETAIL: Could not get a connection to the database: note that superuser access
        is required
        Could not run the following programs, are they installed? psql
        STRUCTURED: time=2016-11-06T05:30:43.813226-00 pid=599