search

deis / builder

Git server and application builder for Deis Workflow
https://deis.com
MIT License
40 stars 41 forks source link

fix(pkg/gitreceive): replace the AWS SDK with minio-go #223

Closed arschles closed 8 years ago

arschles commented 8 years ago

To support Google Cloud Storage compatibility mode, the builder must support AWS version 2 signatures. Singe the github.com/aws/aws-sdk-go package package doesn't support v2 signing for S3, this PR replaces that package with github.com/minio/minio-go, which does.

This PR also emphasizes testing related to object storage, and while necessary, that emphasis led to more changes here than strictly necessary to achieve the functionality we needed.

See below for a list of additional changes necessary to aid testing:

Fixes #222

TODO: After this is done, create an issue to:

arschles commented 8 years ago

My most recent manual test (done with the example-go repo) correctly launches a slugbuilder, and the slugbuilder correctly downloads the tarball and builds it. When it tries to upload the slug, it hangs however. This is the log line on which it hangs:

+ mc --quiet -C /app/mc_config cp /tmp/slug.tgz http://10.171.248.9:9000/git/home/gotest:git-49f11b24/push/

Full build transcript:

ENG000656:example-go aaronschlesinger$ git push deis master
The authenticity of host '[deis.104.197.119.42.xip.io]:2222 ([104.197.119.42]:2222)' can't be established.
ECDSA key fingerprint is SHA256:qxr5/B8QObpvaBfIzfBobGu3qyc+ZEgAAItNXaBQQmY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[deis.104.197.119.42.xip.io]:2222,[104.197.119.42]:2222' (ECDSA) to the list of known hosts.
Counting objects: 138, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (133/133), done.
Writing objects: 100% (138/138), 22.61 KiB | 0 bytes/s, done.
Total 138 (delta 85), reused 0 (delta 0)
remote: 2016/03/03 01:02:34 Running in debug mode
Running git hook
read [0000000000000000000000000000000000000000,49f11b242bb53542b688cce37094226a62c5697d,refs/heads/master]
Workflow request /v2/hooks/push (body elided)
Workflow request POST /v2/hooks/config
{"receive_user":"arschles","receive_repo":"gotest"}
got the following config back for app gotest: {Owner:arschles App:gotest Values:map[] Memory:map[] CPU:map[] Tags:map[] UUID:e455375f-a12c-441f-a934-9c691b51b3ec Created:2016-03-03 00:41:54 +0000 UTC Updated:2016-03-03 00:41:54 +0000 UTC}
running [git archive --format=tar.gz --output=gotest.tar.gz 49f11b24] in directory /home/git/gotest.git
running [tar -xzf gotest.tar.gz -C /home/git/gotest.git/build/tmp256322223/] in directory /home/git/gotest.git
Uploading tar to 10.171.248.9:9000/git/home/gotest:git-49f11b24/tar
Starting build... but first, coffee!
Starting pod slugbuild-gotest-49f11b24-25c1c52c
Pod spec: {
  "metadata": {
    "name": "slugbuild-gotest-49f11b24-25c1c52c",
    "namespace": "deis",
    "creationTimestamp": null,
    "labels": {
      "heritage": "slugbuild-gotest-49f11b24-25c1c52c"
    }
  },
  "spec": {
    "volumes": [
      {
        "name": "minio-user",
        "secret": {
          "secretName": "minio-user"
        }
      }
    ],
    "containers": [
      {
        "name": "deis-slugbuilder",
        "image": "quay.io/deisci/slugbuilder:v2-beta",
        "env": [
          {
            "name": "DEBUG",
            "value": "1"
          },
          {
            "name": "TAR_URL",
            "value": "http://10.171.248.9:9000/git/home/gotest:git-49f11b24/tar"
          },
          {
            "name": "put_url",
            "value": "http://10.171.248.9:9000/git/home/gotest:git-49f11b24/push"
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "minio-user",
            "readOnly": true,
            "mountPath": "/var/run/secrets/object/store"
          }
        ],
        "imagePullPolicy": "Always"
      }
    ],
    "restartPolicy": "Never",
    "serviceAccountName": ""
  },
  "status": {}
}

+ app_dir=/app
+ build_root=/tmp/build
+ cache_root=/tmp/cache
+ buildpack_root=/tmp/buildpacks
+ mkdir -p /app
+ mkdir -p /tmp/cache
+ mkdir -p /tmp/buildpacks
+ mkdir -p /tmp/build/.profile.d
+ MC_PREFIX='mc --quiet -C /app/mc_config'
+ [[ -z http://10.171.248.9:9000/git/home/gotest:git-49f11b24/tar ]]
+ [[ -e /var/run/secrets/object/store/access-key-id ]]
+ [[ -e /var/run/secrets/object/store/access-secret-key ]]
++ cat /var/run/secrets/object/store/access-key-id
+ keyID=8TZRY2JRWMPT6UMXR6I5
++ cat /var/run/secrets/object/store/access-secret-key
+ secretKey=gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
++ awk -F/ '{print $3}'
++ echo http://10.171.248.9:9000/git/home/gotest:git-49f11b24/tar
+ domain=10.171.248.9:9000
+ mc --quiet -C /app/mc_config config host add http://10.171.248.9:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
+ mc --quiet -C /app/mc_config cp http://10.171.248.9:9000/git/home/gotest:git-49f11b24/tar /tmp/slug.tgz
+ tar -xzf /tmp/slug.tgz -C /app/
+ unset TAR_URL
+ [[ '' == \- ]]
+ slug_file=/tmp/slug.tgz
+ [[ -n '' ]]
+ app_dir=/app
+ build_root=/tmp/build
+ cache_root=/tmp/cache
+ buildpack_root=/tmp/buildpacks
+ mkdir -p /app
+ mkdir -p /tmp/cache
+ mkdir -p /tmp/buildpacks
+ mkdir -p /tmp/build/.profile.d
+ '[' -d /tmp/app ']'
+ cp -r /app/. /tmp/build
+ export APP_DIR=/app
+ APP_DIR=/app
+ export HOME=/app
+ HOME=/app
++ openssl rand -base64 32
+ export REQUEST_ID=jL35CpcI3GqBvSYrFO2FBYS/beJK/5FHW+h9i2/yc0M=
+ REQUEST_ID=jL35CpcI3GqBvSYrFO2FBYS/beJK/5FHW+h9i2/yc0M=
+ export STACK=cedar-14
+ STACK=cedar-14
+ [[ -n '' ]]
+ buildpacks=($buildpack_root/*)
+ selected_buildpack=
+ [[ -n '' ]]
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-clojure.git/bin/detect /tmp/build
+ buildpack_name=no
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-go.git/bin/detect /tmp/build
+ buildpack_name=Go
+ selected_buildpack=/tmp/buildpacks/heroku-buildpack-go.git
+ break
+ [[ -n /tmp/buildpacks/heroku-buildpack-go.git ]]
+ echo_title 'Go app detected'
+ output_redirect
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
----->' Go app detected
-----> Go app detected
+ ensure_indent
+ read line
+ /tmp/buildpacks/heroku-buildpack-go.git/bin/compile /tmp/build /tmp/cache
+ [[ -----> Checking Godeps/Godeps.json file. == --* ]]
+ output_redirect
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
----->' Checking Godeps/Godeps.json file.
-----> Checking Godeps/Godeps.json file.
+ read line
 !     
 !     Deprecated version of go (go1.4.2)
 !     See https://devcenter.heroku.com/articles/go-support#go-versions for supported version information.
 !     
+ [[ -----> Installing go1.4.2... done == --* ]]
+ output_redirect
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
----->' Installing go1.4.2... done
-----> Installing go1.4.2... done
+ read line
+ [[ -----> Running: godep go install -tags heroku . == --* ]]
+ output_redirect
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
----->' Running: godep go install -tags heroku .
-----> Running: godep go install -tags heroku .
+ read line
+ /tmp/buildpacks/heroku-buildpack-go.git/bin/release /tmp/build /tmp/cache
+ echo_title 'Discovering process types'
+ output_redirect
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
----->' Discovering process types
-----> Discovering process types
+ [[ -f /tmp/build/Procfile ]]
++ ruby -e 'require '\''yaml'\'';puts YAML.load_file('\''/tmp/build/Procfile'\'').keys().join('\'', '\'')'
+ types=web
+ echo_normal 'Procfile declares types -> web'
+ output_redirect
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
      ' Procfile declares types '->' web
       Procfile declares types -> web
+ default_types=
+ [[ -s /tmp/build/.release ]]
++ ruby -e 'require '\''yaml'\'';puts (YAML.load_file('\''/tmp/build/.release'\'')['\''default_process_types'\''] || {}).keys().join('\'', '\'')'
+ default_types=
+ [[ -n '' ]]
+ chown -R slug:slug /tmp/build/Godeps /tmp/build/LICENSE /tmp/build/Procfile /tmp/build/README.md /tmp/build/bin /tmp/build/mc_config /tmp/build/web.go
+ [[ -f /tmp/build/.slugignore ]]
+ cat
+ tar -z --exclude=.git -C /tmp/build -cf /tmp/slug.tgz .
+ [[ /tmp/slug.tgz != \- ]]
++ cut -f1
++ du -Sh /tmp/slug.tgz
+ slug_size=1.7M
+ echo_title 'Compiled slug size is 1.7M'
+ output_redirect
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
----->' Compiled slug size is 1.7M
-----> Compiled slug size is 1.7M
+ [[ -n http://10.171.248.9:9000/git/home/gotest:git-49f11b24/push ]]
+ [[ -e /var/run/secrets/object/store/access-key-id ]]
+ [[ -e /var/run/secrets/object/store/access-secret-key ]]
++ cat /var/run/secrets/object/store/access-key-id
+ keyID=8TZRY2JRWMPT6UMXR6I5
++ cat /var/run/secrets/object/store/access-secret-key
+ secretKey=gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
++ awk -F/ '{print $3}'
++ echo http://10.171.248.9:9000/git/home/gotest:git-49f11b24/push
+ domain=10.171.248.9:9000
+ mc --quiet -C /app/mc_config config host add http://10.171.248.9:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
+ mc --quiet -C /app/mc_config cp /tmp/slug.tgz http://10.171.248.9:9000/git/home/gotest:git-49f11b24/push/
smothiki commented 8 years ago

Will do manual testing but so far code looks good.

smothiki commented 8 years ago

worked with GCS

arschles commented 8 years ago

Can you try a slug build with minio as well? My last test showed it hanging and I'd like to see if that's systemic

Sent from my iPhone

On Mar 2, 2016, at 18:12, Sivaram Mothiki notifications@github.com wrote:

worked with GCS

— Reply to this email directly or view it on GitHub.

smothiki commented 8 years ago

removed my LGTM as checking for slug files is timing out.

arschles commented 8 years ago

Status as of now: git pushes using minio as the backing object store now succeed, and the resultant app (running in a slugrunner) responds to requests properly.

This PR is ready for a full review.

smothiki commented 8 years ago

Just tested working

arschles commented 8 years ago

This is tested (again) as working post-rebase