Closed jshimko closed 7 years ago
Getting error from helm registry install
2017/09/18 13:17:58 [INFO] payload req_id=O3CM5eahLb-5 raw={"channel":"stable","deployment_name":"jeremy-mongodb","name":"mongodb-replicaset","namespace_id":"02678191","registry":"application/samsung_cnct","server":"quay.io","version":"latest"}
2017/09/18 13:17:58 [INFO] completed req_id=O3CM5eahLb-5 status=202 bytes=424 time=524.06µs ctrl=ApplicationController action=create
2017/09/18 13:17:58 Template file is /tmp/prefix131654328
I0918 13:17:58.432214 1 api_datastore.go:220] successfully archived persistent API state udpate.
W0918 13:18:00.020120 1 exec.go:89] cmd: helm, args: [registry install application/samsung_cnct/mongodb-replicaset --namespace jeremy --name jeremy-mongodb --values /tmp/prefix131654328 --version 1.2.0-0] returned error: exit status 1
W0918 13:18:00.020172 1 exec.go:90] cmd: helm, stderr: usage: appr [-h]
{run-server,show,inspect,logout,plugins,helm,pull,list,delete-package,version,push,login,config,channel}
...
appr: error:
message: 'HTTPSConnectionPool(host=''application'', port=443): Max retries exceeded with url: /cnr/api/v1/packages/samsung_cnct/mongodb-replicaset/default/helm/pull?format=json (Caused by NewConnectionError(''<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4decbfff10>: Failed to establish a new connection: [Errno -2] Name does not resolve'',))'
Error: plugin "registry" exited with error
Running directly within the API server container itself:
# helm registry list quay.io/application/samsung_cnct/
Traceback (most recent call last):
File "/etc/helm/plugins/registry/appr", line 6, in <module>
cli()
File "/usr/lib/python2.7/site-packages/appr/commands/cli.py", line 56, in cli
args.func(args)
File "/usr/lib/python2.7/site-packages/appr/commands/command_base.py", line 73, in call
obj.exec_cmd(render=render)
File "/usr/lib/python2.7/site-packages/appr/commands/command_base.py", line 77, in exec_cmd
self._call()
File "/usr/lib/python2.7/site-packages/appr/commands/list_package.py", line 39, in _call
self.result = client.list_packages(params)
File "/usr/lib/python2.7/site-packages/appr/client.py", line 123, in list_packages
return resp.json()
File "/usr/lib/python2.7/site-packages/requests/models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Error: plugin "registry" exited with error
And...
$ curl -i -XGET quay.io/application/samsung_cnct/mongodb-replicaset
HTTP/1.1 301 Moved Permanently
Content-Type: text/html
Date: Mon, 18 Sep 2017 17:28:38 GMT
Location: https://quay.io/application/samsung_cnct/mongodb-replicaset
Server: nginx/1.13.3
Content-Length: 185
Connection: keep-alive
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.13.3</center>
</body>
</html>
The error here was in the formation of the registry URL. This was updated when the generic chart support was added, but the changes didn't get applied to the mongo chart rules.
registry install application/samsung_cnct/mongodb-replicaset --namespace jeremy --name jeremy-mongodb --values /tmp/prefix131654328 --version 1.2.0-0
Should be:
registry install quay.io/samsung_cnct/mongodb-replicaset --namespace jeremy --name jeremy-mongodb --values /tmp/prefix131654328 --version 1.2.0-0
Also updated API Definitions documentation example: https://github.com/samsung-cnct/krak8s/commit/ef5527b265e655bf7937d92f47f313f7c8a13982
Changes pushed to gitlab for build/tag/deploy.
As far as I can tell, this is not actually fixed. I used the exact curl command from your updated docs (with my own name and namespace values) and it still doesn't work. The resource is created in krak8s (and later marked as failed status), but nothing ever gets deployed in k8s. And the original problem deployment that I opened this issue for appears to have been orphaned on k8s (as in I deleted it with the API, but the pod is still in pending state on k8s).
Based on the git commit history of our state repo, it also looks like this wasn't actually tested before being declared fixed.
And I don't appear to be able to reopen my own issue here anymore, so I'm forced to comment on a closed issue or open a duplicate.
You should be able to "reopen and comment" and the testing wasn't done in your cluster. I'll look at the logs and see what the latest run shows.
I agree, I should be able to "reopen and comment". And I can in other repos. But as of a few hours ago, I can go to any closed issue that I've created in this repo and I can't reopen them.
For example, here's #8 (created by me)...
But I was able to 5 hours ago on #11...
Strange... have no idea what the inconsistency is there... but really weird.
$ kubectl get deploy,rs,svc,ep,pods -n jeremy
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/jeremy-mongodb-mongodb-replicaset None <none> 27017/TCP 2h
NAME ENDPOINTS AGE
ep/jeremy-mongodb-mongodb-replicaset <none> 2h
NAME READY STATUS RESTARTS AGE
po/jeremy-mongodb-mongodb-replicaset-0 0/1 Pending 0 2h
Checking Pod
$ kubectl describe po/jeremy-mongodb-mongodb-replicaset-0 -n jeremy
. . . SNIP . . .
Node-Selectors: <none>
Tolerations: customer=jeremy:Equal:NoSchedule
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
2h 56s 582 default-scheduler Warning FailedScheduling No nodes are available that match all of the following predicates:: MatchNodeSelector (18), NoVolumeZoneConflict (11), PodToleratesNodeTaints (7).
Checking node labels/taints.
$ kubectl get nodes -l nodepool=jeremyNodes
NAME STATUS AGE VERSION
ip-10-0-159-202.us-west-2.compute.internal Ready 7d v1.6.4
ip-10-0-27-37.us-west-2.compute.internal Ready 7d v1.6.4
ip-10-0-3-113.us-west-2.compute.internal Ready 7d v1.6.4
ip-10-0-77-158.us-west-2.compute.internal Ready 7d v1.6.4
Checking Taints
$ kubectl describe node ip-10-0-159-202.us-west-2.compute.internal
. . . SNIP . . .
Taints: node-role.kubernetes.io/jeremy=jeremy:NoSchedule
. . . SNIP . . .
Rebuild with kraken-lib latest (k2:latest) https://github.com/samsung-cnct/krak8s/commit/2b6779ce65b47833d985118d7aad398ecbdbc94c
Verbatim, from the steps outlined here: https://github.com/samsung-cnct/krak8s/blob/master/API%20Definitions.md#a-longer-example
Changing only
1) the host from localhost
to 10.37.145.20
2) replacing the OID's in the example with the actual generated OID.
3) adding -i
to curl to display return codes
$ curl -i http://10.37.145.20:8080/v1/projects
HTTP/1.1 200 OK
Content-Type: application/project+json; type=collection
Date: Fri, 22 Sep 2017 06:33:45 GMT
Content-Length: 182
[{"created_at":"2017-09-11T21:49:32.44708075Z","id":"77907b06","name":"jeremy","namespaces":[{"oid":"02678191","url":"/v1/projects/77907b06/namespaces/02678191"}],"type":"project"}]
$ curl -XPOST -H "Content-Type: application/json" -d '{"name":"neptune"}' http://10.37.145.20:8080/v1/pr
ojects
{"created_at":"2017-09-22T06:34:38.877033246Z","id":"b519b0db","name":"neptune","namespaces":null,"type":"project"}
$ curl -i http://10.37.145.20:8080/v1/projects
HTTP/1.1 200 OK
Content-Type: application/project+json; type=collection
Date: Fri, 22 Sep 2017 06:35:12 GMT
Content-Length: 298
[{"created_at":"2017-09-22T06:34:38.877033246Z","id":"b519b0db","name":"neptune","namespaces":null,"type":"project"},{"created_at":"2017-09-11T21:49:32.44708075Z","id":"77907b06","name":"jeremy","namespaces":[{"oid":"02678191","url":"/v1/projects/77907b06/namespaces/02678191"}],"type":"project"}]
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db
HTTP/1.1 200 OK
Content-Type: application/project+json
Date: Fri, 22 Sep 2017 06:36:42 GMT
Content-Length: 116
{"created_at":"2017-09-22T06:34:38.877033246Z","id":"b519b0db","name":"neptune","namespaces":null,"type":"project"}
$ curl -i -XPOST -H "Content-Type: application/json" -d '{"name":"neptune-test"}' http://10.37.145.20:80
80/v1/projects/b519b0db/namespaces
HTTP/1.1 201 Created
Content-Type: application/namespace+json
Date: Fri, 22 Sep 2017 06:37:50 GMT
Content-Length: 142
{"applications":null,"created_at":"2017-09-22T06:37:50.238869229Z","id":"edc7cbac","name":"neptune-test","resources":null,"type":"namespace"}
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/namespaces
HTTP/1.1 200 OK
Content-Type: application/namespace+json; type=collection
Date: Fri, 22 Sep 2017 06:38:38 GMT
Content-Length: 144
[{"applications":null,"created_at":"2017-09-22T06:37:50.238869229Z","id":"edc7cbac","name":"neptune-test","resources":null,"type":"namespace"}]
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/namespaces/edc7cbac
HTTP/1.1 200 OK
Content-Type: application/namespace+json
Date: Fri, 22 Sep 2017 06:39:30 GMT
Content-Length: 142
{"applications":null,"created_at":"2017-09-22T06:37:50.238869229Z","id":"edc7cbac","name":"neptune-test","resources":null,"type":"namespace"}
curl -i -XPOST -H "Content-Type: application/json" -d '{"namespace_id":"edc7cbac", "nodePoolSize": 3}'
http://10.37.145.20:8080/v1/projects/b519b0db/cluster
HTTP/1.1 202 Accepted
Content-Type: application/cluster+json
Date: Fri, 22 Sep 2017 06:41:01 GMT
Content-Length: 188
{"created_at":"2017-09-22T06:41:01.213473167Z","id":"f6912cd3","namespace_id":"edc7cbac","nodePoolSize":3,"state":"create_requested","type":"Resource","updated_at":"0001-01-01T00:00:00Z"}
curl -i http://10.37.145.20:8080/v1/projects/b519b0db/cluster/f6912cd3
HTTP/1.1 200 OK
Content-Type: application/cluster+json
Date: Fri, 22 Sep 2017 06:42:44 GMT
Content-Length: 188
{"created_at":"2017-09-22T06:41:01.213473167Z","id":"f6912cd3","namespace_id":"edc7cbac","nodePoolSize":3,"state":"active","type":"Resource","updated_at":"2017-09-22T06:42:24.217646231Z"}
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/namespaces/edc7cbac
HTTP/1.1 200 OK
Content-Type: application/namespace+json
Date: Fri, 22 Sep 2017 06:43:16 GMT
Content-Length: 203
{"applications":null,"created_at":"2017-09-22T06:37:50.238869229Z","id":"edc7cbac","name":"neptune-test","resources":{"oid":"f6912cd3","url":"/v1/projects/b519b0db/cluster/f6912cd3"},"type":"namespace"}
$ kubectl get nodes -l nodepool=neptuneNodes
NAME STATUS AGE VERSION
ip-10-0-121-235.us-west-2.compute.internal Ready 1m v1.6.4
ip-10-0-140-190.us-west-2.compute.internal Ready 1m v1.6.4
ip-10-0-7-4.us-west-2.compute.internal NotReady 14s v1.6.4
$ curl -i -XPOST -H "Content-Type: application/json" -d '{ "deployment_name": "neptune-mongodb", "server
": "quay.io", "registry": "samsung_cnct", "name": "mongodb-replicaset", "version": "latest", "channel": "stable", "namespace_id": "edc
7cbac"}' http://10.37.145.20:8080/v1/projects/b519b0db/applications
HTTP/1.1 202 Accepted
Content-Type: application/application+json
Date: Fri, 22 Sep 2017 06:51:59 GMT
Content-Length: 412
{"channel":"stable","config":"","created_at":"2017-09-22T06:51:59.139909415Z","deployment_name":"neptune-mongodb","id":"76081c15","json_values":"","name":"mongodb-replicaset","namespace_id":"edc7cbac","registry":"samsung_cnct","server":"quay.io","status":{"deployed_at":"0001-01-01T00:00:00Z","state":"UNKNOWN"},"type":"application","updated_at":"2017-09-22T06:51:59.13991163Z","username":"","version":"latest"}
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/applications/76081c15
HTTP/1.1 200 OK
Content-Type: application/application+json
Date: Fri, 22 Sep 2017 06:53:22 GMT
Content-Length: 424
{"channel":"stable","config":"","created_at":"2017-09-22T06:51:59.139909415Z","deployment_name":"neptune-mongodb","id":"76081c15","json_values":"","name":"mongodb-replicaset","namespace_id":"edc7cbac","registry":"samsung_cnct","server":"quay.io","status":{"deployed_at":"2017-09-22T06:52:02.567256017Z","state":"DEPLOYED"},"type":"application","updated_at":"2017-09-22T06:52:02.567256338Z","username":"","version":"latest"}
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/namespaces/edc7cbac
HTTP/1.1 200 OK
Content-Type: application/namespace+json
Date: Fri, 22 Sep 2017 06:53:52 GMT
Content-Length: 271
{"applications":[{"oid":"76081c15","url":"/v1/projects/b519b0db/applications/76081c15"}],"created_at":"2017-09-22T06:37:50.238869229Z","id":"edc7cbac","name":"neptune-test","resources":{"oid":"f6912cd3","url":"/v1/projects/b519b0db/cluster/f6912cd3"},"type":"namespace"}
$ helm list
NAME REVISION UPDATED STATUS CHART NAMESPACE
dex 1 Fri Jul 28 22:49:11 2017 DEPLOYED dex-0.1.0-0 kube-auth
foray-mongo 1 Fri Aug 11 05:45:15 2017 DEPLOYED mongodb-replicaset-1.2.0-0 foray
foray-mongo-beta 1 Wed Aug 23 12:36:22 2017 DEPLOYED mongodb-replicaset-1.2.0-0 foray
foray-reaction 1 Wed Aug 9 13:10:01 2017 DEPLOYED reactioncommerce-0.0.1 foray
foray-reaction-beta 1 Thu Aug 24 09:35:09 2017 DEPLOYED reactioncommerce-0.3.0-0 foray
gitlab 23 Wed Sep 13 10:19:08 2017 DEPLOYED gitlab-ce-v0.0.1-dummy gitlab
gitlab-runner 1 Mon Aug 7 10:48:32 2017 DEPLOYED gitlab-runner-0.1.9 gitlab
glingress 1 Sat Jul 29 07:47:59 2017 DEPLOYED nginx-ingress-0.2.1-0 gitlab
glpostgresql 1 Sat Jul 29 07:48:06 2017 DEPLOYED postgresql-0.7.1-0 gitlab
glredis 1 Sat Jul 29 07:48:03 2017 DEPLOYED redis-0.8.0-0 gitlab
heapster 2 Mon Sep 18 12:23:03 2017 DEPLOYED heapster-0.2.0-0 kube-system
k8s-dashboard 1 Fri Sep 1 09:56:00 2017 DEPLOYED kubernetes-dashboard-0.1.0 kube-system
krak8s 20 Tue Sep 19 00:44:42 2017 DEPLOYED krak8s-api-0.5.4-0 krak8s
kube-lego 1 Fri Jul 28 23:44:45 2017 DEPLOYED kube-lego-0.0.5-0 ingress
launchdock 2 Wed Sep 20 07:14:51 2017 DEPLOYED launchdock-0.3.0 launchdock
ld-mongo 1 Sat Jul 29 09:04:19 2017 DEPLOYED mongodb-replicaset-1.2.0-0 launchdock
neptune-mongodb 1 Thu Sep 21 23:52:01 2017 DEPLOYED mongodb-replicaset-1.2.0-0 neptune-test
nginx-ingress 1 Fri Jul 28 22:49:05 2017 DEPLOYED nginx-ingress-0.2.1-0 ingress
$ helm status neptune-mongodb
LAST DEPLOYED: Thu Sep 21 23:52:01 2017
NAMESPACE: neptune-test
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
neptune-mongodb-mongodb-replicaset None <none> 27017/TCP 2m
==> v1beta1/StatefulSet
NAME DESIRED CURRENT AGE
neptune-mongodb-mongodb-replicaset 3 3 2m
==> v1/ConfigMap
NAME DATA AGE
neptune-mongodb-mongodb-replicaset 1 2m
neptune-mongodb-mongodb-replicaset-tests 1 2m
$ kubectl get statefulset,svc,ep,pods -n neptune-test
NAME DESIRED CURRENT AGE
statefulsets/neptune-mongodb-mongodb-replicaset 3 3 3m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/neptune-mongodb-mongodb-replicaset None <none> 27017/TCP 3m
NAME ENDPOINTS AGE
ep/neptune-mongodb-mongodb-replicaset 10.128.33.2:27017,10.128.34.2:27017,10.128.35.2:27017 3m
NAME READY STATUS RESTARTS AGE
po/neptune-mongodb-mongodb-replicaset-0 1/1 Running 0 3m
po/neptune-mongodb-mongodb-replicaset-1 1/1 Running 0 2m
po/neptune-mongodb-mongodb-replicaset-2 1/1 Running 0 1m
In reverse order, deleting all the things that were created in the previous steps.
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/cluster/f6912cd3
HTTP/1.1 200 OK
Content-Type: application/cluster+json
Date: Fri, 22 Sep 2017 07:09:14 GMT
Content-Length: 188
{"created_at":"2017-09-22T06:41:01.213473167Z","id":"f6912cd3","namespace_id":"edc7cbac","nodePoolSize":3,"state":"active","type":"Resource","updated_at":"2017-09-22T06:42:24.217646231Z"}
$ curl -i -XDELETE http://10.37.145.20:8080/v1/projects/b519b0db/cluster/f6912cd3
HTTP/1.1 204 No Content
Date: Fri, 22 Sep 2017 07:09:24 GMT
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/cluster/f6912cd3
HTTP/1.1 404 Not Found
Date: Fri, 22 Sep 2017 07:09:36 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/namespaces/edc7cbac
HTTP/1.1 200 OK
Content-Type: application/namespace+json
Date: Fri, 22 Sep 2017 07:10:08 GMT
Content-Length: 142
{"applications":null,"created_at":"2017-09-22T06:37:50.238869229Z","id":"edc7cbac","name":"neptune-test","resources":null,"type":"namespace"}
$ kubectl get nodes -l nodepool=neptuneNodes
No resources found.
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db
HTTP/1.1 200 OK
Content-Type: application/project+json
Date: Fri, 22 Sep 2017 07:12:21 GMT
Content-Length: 182
{"created_at":"2017-09-22T06:34:38.877033246Z","id":"b519b0db","name":"neptune","namespaces":[{"oid":"edc7cbac","url":"/v1/projects/b519b0db/namespaces/edc7cbac"}],"type":"project"}
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/namespaces/edc7cbac
HTTP/1.1 200 OK
Content-Type: application/namespace+json
Date: Fri, 22 Sep 2017 07:12:30 GMT
Content-Length: 142
{"applications":null,"created_at":"2017-09-22T06:37:50.238869229Z","id":"edc7cbac","name":"neptune-test","resources":null,"type":"namespace"}
$ curl -i -XDELETE http://10.37.145.20:8080/v1/projects/b519b0db/namespaces/edc7cbac
HTTP/1.1 204 No Content
Date: Fri, 22 Sep 2017 07:13:18 GMT
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db/namespaces/edc7cbac
HTTP/1.1 404 Not Found
Date: Fri, 22 Sep 2017 07:13:32 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db
HTTP/1.1 200 OK
Content-Type: application/project+json
Date: Fri, 22 Sep 2017 07:13:45 GMT
Content-Length: 116
{"created_at":"2017-09-22T06:34:38.877033246Z","id":"b519b0db","name":"neptune","namespaces":null,"type":"project"}
$ curl -i -XDELETE http://10.37.145.20:8080/v1/projects/b519b0db
HTTP/1.1 204 No Content
Date: Fri, 22 Sep 2017 07:14:08 GMT
$ curl -i http://10.37.145.20:8080/v1/projects/b519b0db
HTTP/1.1 404 Not Found
Date: Fri, 22 Sep 2017 07:14:15 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
$ curl -i http://10.37.145.20:8080/v1/projects
HTTP/1.1 200 OK
Content-Type: application/project+json; type=collection
Date: Fri, 22 Sep 2017 07:14:40 GMT
Content-Length: 182
[{"created_at":"2017-09-11T21:49:32.44708075Z","id":"77907b06","name":"jeremy","namespaces":[{"oid":"02678191","url":"/v1/projects/77907b06/namespaces/02678191"}],"type":"project"}]
Need to pull additional fix from Kraken: https://github.com/samsung-cnct/kraken-lib/pull/845
Resolved.
I’ve been trying to get a Mongo deployment to happen via krak8s for hours and I’m still not seeing anything other than a single pending pod.
10-15 mins later…
However…
and…