search

deis / builder

Git server and application builder for Deis Workflow
https://deis.com
MIT License
40 stars 41 forks source link

aws s3 go sdk only works for minio and aws S3 #222

Closed smothiki closed 8 years ago

smothiki commented 8 years ago

the cli code to access S3 works only for AWS S3 and minio . Tested with goamz/goamz S3 client which works for ceph , aws, gcs but not minio . Tested with AdRoll/goamz which is slight modification of above mentioned repo which works for Aws S3, GCS, minio but not for ceph.

For beta release atleast GCS should work . So we have to implement AdRoll goamz

smothiki commented 8 years ago

Also change MC client which we are using Slug builder and slug Runner accordingly

arschles commented 8 years ago

@smothiki regarding your comment, the mc client binary seems to work with GCS in compatability mode. Here's the total output:

ENG000656:Desktop aaronschlesinger$ mc version
Version: 2016-02-19T04:11:55Z
Release-tag: RELEASE.2016-02-19T04-11-55Z
Commit-id: 54761b22fdcc4c85c15b82904bb5670c80a5b800
ENG000656:Desktop aaronschlesinger$ mc config host add gcs https://storage.googleapis.com GOOGLBM4I6636WYDPTJK $GCS_SECRET S3v2
Added ‘gcs’ successfully.
ENG000656:Desktop aaronschlesinger$ mc ls gcs
[2016-02-16 09:39:33 PST]     0B aaron-outside-storage-test/
[2016-02-19 11:14:04 PST]     0B aaron-utils/
[2016-02-11 10:52:02 PST]     0B kubernetes-staging-d205461b3a/
[2016-02-11 11:32:58 PST]     0B michelle-charts/
ENG000656:Desktop aaronschlesinger$ mc ls gcs/aaron-outside-storage-test/
[2016-03-02 11:10:23 PST]   408B deploy.sh
ENG000656:Desktop aaronschlesinger$ mc cp repo_lock.go gcs/aaron-outside-storage-test/repo_lock.go
repo_lock.go:                       416 B / 416 B  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  100.00 % 1.56 KB/s 0ENG000656:Desktop aaronschlesinger$ mc ls gcs/aaron-outside-storage-test/
[2016-03-02 11:10:23 PST]   408B deploy.sh
[2016-03-02 11:15:12 PST]   416B repo_lock.go
ENG000656:Desktop aaronschlesinger$ mc cp gcs/aaron-outside-storage-test/repo_lock.go ./repo_lock_downloaded.go
...tside-storage-test/repo_lock.go: 416 B / 416 B  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  100.00 % 1.11 KB/s 0

Still in-progress on the AWS Go SDK changes...

smothiki commented 8 years ago

It's S3v2 signature which might not work with Minio. We have to add conditions to slugbuilder to move signature from s3v2 to s3v4

arschles commented 8 years ago

yep, I'll add tickets to slugbuilder & runner for checking the condition

arschles commented 8 years ago

Ref https://github.com/deis/slugrunner/issues/17 Ref https://github.com/deis/slugbuilder/issues/31