Could not fetch loadbalancer in kubernetes architecture

[littlechicks]

Hello Zato Community!

I followed the tutorial https://zato.io/docs/admin/guide/install/py3/kubernetes.html for create an Zato cluster.

Note : The yaml file (zato-k8.yaml) for the deployment contains error on the url of docker image. Original yaml have the url registry.gitlab.com/zatosource/docker-registry/cloud:3.1 , we should remove the ':3.1'

In order to access the zato api from my laptop, I used the kube-proxy for binding request on my vm address.

I can fetch zato api with the url http://{MY_VM}:8001/api/v1/namespaces/zato/services/webadmin:web/proxy/zato/cluster/

And I can log on in to zato webadmin.

But when I navigate in the server I have serveral errors.

First, in Cluster menu I have this error:

And then, when I navigate to REST channels :

Finally, I got other error below:

I tried to troublshoot minikube but I did find any solution for my problem.

Any idea ?

[littlechicks]

On my minikube dashboard it's seems that the scheduler crash every time....

With the following error scheduler app: scheduleregistry.gitlab.com/zatosource/docker-registry/cloud more_vert Back-off restarting failed container

Then I checked the logs of the pods "scheduler"

linux@eowyn:~/envdev/zato-build/docker/kubernetes$ kubectl logs --namespace=zato scheduler-788fd8cc4c-q9wv2 2019/09/09 09:36:17 Ready: tcp://redis.zato.svc.cluster.local:6379. 2019/09/09 09:36:17 Ready: tcp://postgres.zato.svc.cluster.local:5432. Zato 3.1+rev.015a80f-py3.6.8 /opt/zato/env/qs-1/ /opt/zato/current/bin/zato create scheduler --odb_host postgres.zato.svc.cluster.local --odb_port 5432 --odb_user zato --odb_db_name zato --odb_password zatodev --kvdb_password '' --secret_key Iua4PhafkwPILtW38gUaFre_WD6oBNqh65qZKzHU4Y4 /opt/zato/env/qs-1/ postgresql redis.zato.svc.cluster.local 6379 zato SecretKeyError: (('Incorrect padding',),) (Hint: re-run with --verbose for full traceback)

The error is with the secretKey ?

For the deployment I used this environnement file, with key generated with zato.

ZATO_WEB_ADMIN_PASSWORD=zatodev
IDE_PUBLISHER_PASSWORD=zatodev
ZATO_ADMIN_INVOKE_PASSWORD=zatodev
CLUSTER_NAME=zato
SECRET_KEY=Iua4PhafkwPILtW38gUaFre_WD6oBNqh65qZKzHU4Y4
JWT_SECRET_KEY=YbSW_2VQt4zk8BicxDdqFoj4WuCEMSyGMR_eWGEThaE
ODB_PASSWORD=zatodev
VERBOSE=y

[littlechicks]

I don't know if my errors are because of the secret key. But I can't navigate threw page in zato webadmin. I've "connection refused" always...

`Method could not be invoked, e:`Traceback (most recent call last):
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/connection.py", line 196, in connect
    conn = self._new_conn()
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='server.zato.svc.cluster.local', port=80): Max retries exceeded with url: /zato/admin/invoke (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zato/3.1.0/code/zato-web-admin/src/zato/admin/web/views/scheduler.py", line 304, in index
    data, meta = parse_response_data(req.zato.client.invoke('zato.scheduler.job.get-list', request))
  File "/opt/zato/3.1.0/code/zato-web-admin/src/zato/admin/middleware.py", line 79, in invoke
    response = super(Client, self).invoke(*args, headers={'X-Zato-Forwarded-For': self.forwarded_for}, **kwargs)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 484, in invoke
    return self._invoke(async=False, *args, **kwargs)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 481, in _invoke
    ServiceInvokeResponse, async, headers, output_repeated)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 413, in invoke
    return self.inner_invoke(request, response_class, async, headers)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 395, in inner_invoke
    raw_response = self.session.post(self.service_address, request, headers=headers, verify=self.tls_verify)
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/requests/sessions.py", line 649, in send
    r = adapter.send(request, **kwargs)
  File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='server.zato.svc.cluster.local', port=80): Max retries exceeded with url: /zato/admin/invoke (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))`

[littlechicks]

I have corrected the "error padding" issue. This errors is caused by the "=" missing at the end of the secret key.

[littlechicks]

I think this is related to an "out of memory issue"

So finally I've resolved my issue by not using minikube with virtualbox hypervisor. I used the command minikube start --vm-driver=none