search

deis / builder

Git server and application builder for Deis Workflow
https://deis.com
MIT License
40 stars 41 forks source link

Question: socket.timout with k8s v1.5.2 #477

Closed Akshaykapoor closed 7 years ago

Akshaykapoor commented 7 years ago

Guys I'm new to deis and wanting to test and setup for my workflows.

I'm running deis client version 2.10.0 with k8s installed on aws,

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.7", GitCommit:"92b4f971662de9d8770f8dcd2ee01ec226a6f6c0", GitTreeState:"clean", BuildDate:"2016-12-10T04:49:33Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl get po -n=deis
deis-builder-2413555432-s8p46            1/1       Running   0          1h
deis-controller-1606776397-0vhw8         1/1       Running   0          3h
deis-database-4098120950-dq540           1/1       Running   0          1d
deis-logger-9343270-q2wqc                1/1       Running   2          1d
deis-logger-fluentd-cchzh                1/1       Running   0          1d
deis-logger-fluentd-mrjmj                1/1       Running   0          1d
deis-logger-fluentd-ws2jm                1/1       Running   0          1d
deis-logger-redis-215589726-t2h2t        1/1       Running   0          1d
deis-minio-517538921-m039m               1/1       Running   0          1d
deis-monitor-grafana-323247549-m2w87     1/1       Running   0          1d
deis-monitor-influxdb-2577745094-wddsw   1/1       Running   0          1d
deis-monitor-telegraf-5jc9h              1/1       Running   1          1d
deis-monitor-telegraf-b8gcg              1/1       Running   1          1d
deis-monitor-telegraf-djh5m              1/1       Running   1          1d
deis-nsqd-3430976322-cqxrb               1/1       Running   0          1d
deis-registry-4288276514-02dzg           1/1       Running   0          2h
deis-registry-proxy-hg1nl                1/1       Running   0          1d
deis-registry-proxy-j891f                1/1       Running   0          1d
deis-registry-proxy-qjj6l                1/1       Running   0          1d
deis-router-2099825860-g3nhw             1/1       Running   0          1d
deis-workflow-manager-2369615478-dqfwc   1/1       Running   0          1d

I tried running the example-go app from the docs, and facing the issue below. Ofcourse git cone and other stuuf was done before that.

running deis create

Following are the logs in deis-router

[2017-02-02T14:38:57+00:00] - deis/deis-controller - X.X.X.X - - - 201 - "POST /v2/apps/ HTTP/1.1" - 510 - "-" - "Deis Client v2.10.0" - "~^deis\x5C.(?<domain>.+)$" - X.X.X.X:80 - deis.REDACTED.com - 0.200 - 0.200
2017/02/02 14:39:07 INFO: Router configuration has changed in k8s.
2017/02/02 14:39:07 INFO: Reloading nginx...
2017/02/02 14:39:07 INFO: nginx reloaded.

and deis-controller

INFO [webbed-handbill]: config webbed-handbill-2154fcc updated
INFO [webbed-handbill]: akshay created initial release
INFO [webbed-handbill]: appsettings webbed-handbill-87c9ab8 updated
INFO [webbed-handbill]: domain webbed-handbill added
100.96.2.31 "POST /v2/apps/ HTTP/1.1" 201 168 "Deis Client v2.10.0"

running git push deis master after the above results in the following. So far so good, until after which it craps out with socket.timeout. Please see the trace below for more information

I've been successful in trying out deis locally with minikube though a different version of k8s, 1.4.7 . Am i missing something ? Would be helpful if somebody can point me in the right direction.

Logs from deis-builder,

receiving git repo name: webbed-handbill.git, operation: git-receive-pack, fingerprint: 66:7f:f3:ab:39:b0:3f:8e:34:64:ad:f0:43:6c:17:7d, user: akshay
creating repo directory /home/git/webbed-handbill.git
Creating new directory at /home/git/webbed-handbill.git
writing pre-receive hook under /home/git/webbed-handbill.git
git-shell -c git-receive-pack 'webbed-handbill.git'
Working Dir: /home/git
Environment: REDACTED
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.

deis-router

INFO [webbed-handbill]: config webbed-handbill-2154fcc updated
INFO [webbed-handbill]: akshay created initial release
INFO [webbed-handbill]: appsettings webbed-handbill-87c9ab8 updated
INFO [webbed-handbill]: domain webbed-handbill added
100.96.2.31 "POST /v2/apps/ HTTP/1.1" 201 168 "Deis Client v2.10.0"
100.96.1.54 "GET /v2/hooks/key/REDACTED HTTP/1.1" 200 66 "deis-builder"
100.96.1.54 "POST /v2/hooks/config/ HTTP/1.1" 200 228 "deis-builder"

This is what i get in the end,

Counting objects: 39, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (38/38), done.
Writing objects: 100% (39/39), 5.17 KiB | 0 bytes/s, done.
Total 39 (delta 12), reused 0 (delta 0)
remote: 2017/02/02 14:42:48 Running in debug mode
Running git hook
read [0000000000000000000000000000000000000000,99a73c05890a2062f2819ef89a1c74eeec6b3408,refs/heads/master]
got the following config back for app webbed-handbill: {Owner:akshay App:webbed-handbill Values:map[] Memory:map[] CPU:map[] Healthcheck:map[] Tags:map[] Registry:map[] Created:2017-02-02T14:38:57Z Updated:2017-02-02T14:38:57Z UUID:2154fcca-a6c0-40e3-8723-8680e97fa47d}
running [git archive --format=tar.gz --output=webbed-handbill.tar.gz 99a73c05] in directory /home/git/webbed-handbill.git
running [tar -xzf webbed-handbill.tar.gz -C /home/git/webbed-handbill.git/build/tmp792185540/] in directory /home/git/webbed-handbill.git
Uploading tar to home/webbed-handbill:git-99a73c05/tar
Starting build... but first, coffee!
Starting pod dockerbuild-webbed-handbill-99a73c05-296ca18c
Pod spec: {
  "metadata": {
    "name": "dockerbuild-webbed-handbill-99a73c05-296ca18c",
    "namespace": "deis",
    "creationTimestamp": null,
    "labels": {
      "heritage": "dockerbuild-webbed-handbill-99a73c05-296ca18c"
    }
  },
  "spec": {
    "volumes": [
      {
        "name": "objectstorage-keyfile",
        "secret": {
          "secretName": "objectstorage-keyfile"
        }
      },
      {
        "name": "docker-socket",
        "hostPath": {
          "path": "/var/run/docker.sock"
        }
      }
    ],
    "containers": [
      {
        "name": "deis-dockerbuilder",
        "image": "quay.io/deis/dockerbuilder:v2.5.2",
        "env": [
          {
            "name": "DEIS_DEBUG",
            "value": "1"
          },
          {
            "name": "TAR_PATH",
            "value": "home/webbed-handbill:git-99a73c05/tar"
          },
          {
            "name": "SOURCE_VERSION",
            "value": "99a73c05"
          },
          {
            "name": "IMG_NAME",
            "value": "webbed-handbill:git-99a73c05"
          },
          {
            "name": "BUILDER_STORAGE",
            "value": "minio"
          },
          {
            "name": "DEIS_REGISTRY_SERVICE_HOST",
            "value": "localhost"
          },
          {
            "name": "DEIS_REGISTRY_SERVICE_PORT",
            "value": "5555"
          },
          {
            "name": "DEIS_REGISTRY_LOCATION",
            "value": "on-cluster"
          },
          {
            "name": "DEIS_REGISTRY_PROXY_PORT",
            "value": "5555"
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "objectstorage-keyfile",
            "readOnly": true,
            "mountPath": "/var/run/secrets/deis/objectstore/creds"
          },
          {
            "name": "docker-socket",
            "mountPath": "/var/run/docker.sock"
          }
        ],
        "imagePullPolicy": "IfNotPresent"
      }
    ],
    "restartPolicy": "Never",
    "serviceAccountName": ""
  },
  "status": {}
}

2017/02/02 14:42:50 Successfully copied home/webbed-handbill:git-99a73c05/tar to apptar
Step 1 : FROM debian:jessie
download tar file complete
extracting tar file complete
 ---> e5599115b6a6
Step 2 : RUN apt-get update && apt-get install -qy curl
 ---> Using cache
 ---> 8f998a72b6bd
Step 3 : RUN curl -s https://storage.googleapis.com/golang/go1.2.2.linux-amd64.tar.gz | tar -C /usr/local -xz
 ---> Using cache
 ---> fb5bd6afc36c
Step 4 : ENV GOPATH /go
 ---> Using cache
 ---> 125311877d94
Step 5 : ENV GOROOT /usr/local/go
 ---> Using cache
 ---> c4aee01a70fc
Step 6 : ENV PATH $PATH:/usr/local/go/bin:/go/bin
 ---> Using cache
 ---> e3e63a326559
Step 7 : ADD . /go/src/github.com/deis/helloworld
 ---> Using cache
 ---> b966a6753b8a
Step 8 : RUN cd /go/src/github.com/deis/helloworld && go install -v .
 ---> Using cache
 ---> c11dc2c37d06
Step 9 : EXPOSE 80
 ---> Using cache
 ---> e8d9ba39ffd3
Step 10 : ENTRYPOINT /go/bin/helloworld
 ---> Using cache
 ---> 2562a6abd20e
Successfully built 2562a6abd20e
Pushing to registry
{'status': 'The push refers to a repository [localhost:5555/webbed-handbill]'}
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/response.py", line 298, in _error_catcher
    yield
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/response.py", line 380, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 478, in readinto
    return self._readinto_chunked(b)
  File "/usr/lib/python3.5/http/client.py", line 573, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/lib/python3.5/http/client.py", line 541, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.5/http/client.py", line 501, in _read_next_chunk_size
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/deploy.py", line 109, in <module>
    log_output(stream, False)
  File "/deploy.py", line 15, in log_output
    for chunk in stream:
  File "/usr/local/lib/python3.5/dist-packages/docker/client.py", line 247, in _stream_helper
    data = reader.read(1)
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/response.py", line 397, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/response.py", line 303, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
size of streamed logs 3021
Waiting for the deis/dockerbuild-webbed-handbill-99a73c05-296ca18c pod to end. Checking every 100ms for 15m0s
Done
Checking for builder pod exit code
remote: 2017/02/02 14:43:51 Error running git receive hook [Build pod exited with code 1, stopping build.]
To ssh://deis-builder.REDACTED.com:2222/webbed-handbill.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@deis-builder.REDACTED.com:2222/webbed-handbill.git'
bacongobbler commented 7 years ago

This issue is a duplicate of https://github.com/deis/workflow/issues/678 which was fixed in v2.11. If you want to run with v2.10 just patch "localhost" with "127.0.0.1" in your charts. Thanks!

Akshaykapoor commented 7 years ago

@bacongobbler thanks for the prompt response, worked like a charm!