Internal server error 500 when deploying to Kubernetes

[zepp-div]

Hi! Continuation of issue [#1108]

We are trying to deploy a service to kubernetes. Trying bare-metal and minikube After deployment, we catch the server error 500 when working with the connections->sftp section. The problem is floating, it occurs every other time (50 to 50 success / error).

there is an error in the server log:

2021-03-02 18:12:45,269 - ERROR - 32:DummyThread-55 - zato.server.connection.http_soap.channel:0 - Caught an exception, cid:`817f02f8dacc3676f1ab7b5a`, status_code:`HTTPStatus.INTERNAL_SERVER_ERROR`, `Traceback (most recent call last):
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/pg8000/core.py", line 1908, in execute
    self._flush()
  File "/usr/lib/python3.6/socket.py", line 604, in write
    return self._sock.send(b)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/gevent/_socket3.py", line 515, in send
    return self._sock.send(data, flags)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2338, in _wrap_pool_connect
    return fn()
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
    result = pool._dialect.do_ping(fairy.connection)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 560, in do_ping
    cursor.execute(self._dialect_specific_select_one)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/pg8000/core.py", line 860, in execute
    self._c.execute(self, "begin transaction", None)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/pg8000/core.py", line 1911, in execute
    raise OperationalError(e.args[0])
pg8000.core.OperationalError: 32
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/connection/http_soap/channel.py", line 382, in dispatch
    payload, worker_store, self.simple_io_config, post_data, path_info, soap_action)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/connection/http_soap/channel.py", line 689, in handle
    params_priority=channel_item.params_pri)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 748, in update_handle
    raise e if isinstance(e, Exception) else Exception(e)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 701, in update_handle
    self._invoke(service, channel)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 587, in _invoke
    service.handle()
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/internal/service.py", line 366, in handle
    response = func(id_, payload, channel, data_format, transport, serialize=True)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 825, in invoke
    return self.invoke_by_impl_name(self.server.service_store.name_to_impl_name[name], *args, **kwargs)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 803, in invoke_by_impl_name
    out = self.update_handle(*invoke_args, **kwargs)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 748, in update_handle
    raise e if isinstance(e, Exception) else Exception(e)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 701, in update_handle
    self._invoke(service, channel)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 587, in _invoke
    service.handle()
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/internal/generic/connection.py", line 237, in handle
    search_result = self.get_data(session)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/internal/generic/connection.py", line 191, in get_data
    return self._search(connection_list, session, self.request.input.cluster_id, self.request.input.type_, False)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/internal/__init__.py", line 158, in _search
    result = sql_search(search_func, self.request.input, self._filter_by, session, cluster_id, *args, **kwargs)
  File "/opt/zato_dir/zato/code/zato-common/src/zato/common/util/sql.py", line 85, in search
    result = search_func(session, cluster_id, *args, **kwargs)
  File "/opt/zato_dir/zato/code/zato-common/src/zato/common/odb/query/__init__.py", line 122, in inner
    tool = _SearchWrapper(result, **kwargs)
  File "/opt/zato_dir/zato/code/zato-common/src/zato/common/odb/query/__init__.py", line 92, in __init__
    self.total = q.session.execute(total_q).scalar()
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1291, in execute
    return self._connection_for_bind(bind, close_with_result=True).execute(
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1147, in _connection_for_bind
    engine, execution_options
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
    conn = bind._contextual_connect()
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2304, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2342, in _wrap_pool_connect
    e, dialect, self
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _handle_dbapi_exception_noconnection
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2338, in _wrap_pool_connect
    return fn()
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
    result = pool._dialect.do_ping(fairy.connection)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 560, in do_ping
    cursor.execute(self._dialect_specific_select_one)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/pg8000/core.py", line 860, in execute
    self._c.execute(self, "begin transaction", None)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/pg8000/core.py", line 1911, in execute
    raise OperationalError(e.args[0])
sqlalchemy.exc.OperationalError: (pg8000.core.OperationalError) 32
(Background on this error at: http://sqlalche.me/e/13/e3q8)

Based on the commit e6a637f1d83f755023bce43fa3ef0aedbeae9d44

[DimShadoWWW]

Hello @zepp-div ,

It seems to be an error accessing the Database when saving the new connection. Please, can you check if it is working? How about the resources in minikube? I have had problems with the resources allocated to minikube and I have had to allocate more by parameters.

[zepp-div]

Hi, @DimShadoWWW The error occurs when accessing the interface for creating a new connection. We haven't gotten to saving the new connection yet = (

I noticed that the problem arises with a multi-node configuration of a Kubernetes cluster (regardless of the cluster - bare-metal or minikube)

Additional

Could not handle broker message: ('SCHEDULER_JOB_EXECUTED':'100004') `{"'action'":"'100004'","'cid'":"'274997cf626f4e008b88f88a'","'job_type'":"'interval_based'","'msg_type'":"'0001'","'name'":"'zato.outgoing.sql.auto-ping'","'payload'":"''","'service'":"'zato.outgoing.sql.auto-ping'"}`, e:`'Traceback (most recent call last):
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 729, in update_handle
    response = set_response_func(service, data_format=data_format, transport=transport, **kwargs)
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/__init__.py", line 551, in set_response_data
    response = response.getvalue(serialize=kw'`

Could not ping ODB, e:`'Traceback (most recent call last):
  File "/opt/zato_dir/zato/code/zato-server/src/zato/server/service/internal/outgoing/sql.py", line 256, in handle
    self.server.sql_pool_store[ZATO_ODB_POOL_NAME].pool.ping(self.server.fs_sql_config)
  File "/opt/zato_dir/zato/code/zato-common/src/zato/common/odb/api.py", line 354, in ping
    func = self.engine.connect().execute
  File "/opt/zato_dir/zato/cod'`

Error in cleanup: `'Traceback (most recent call last):
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/pg8000/core.py", line 1908, in execute
    self._flush()
  File "/usr/lib/python3.6/socket.py", line 604, in write
    return self._sock.send(b)
  File "/opt/zato_dir/zato/code/lib/python3.6/site-packages/gevent/_socket3.py", line 515, in send
    return self._sock.send(data, flags)
BrokenPipeError: [Err'`

[DimShadoWWW]

@zepp-div , it seems to be related to your overlay network. Which networking model are you using to connect the nodes?

[zepp-div]

@DimShadoWWW , we use calico

[DimShadoWWW]

Hello @zepp-div , I have tried to reproduce your problem, but I have not succeeded. Please, can you give me more information about your environment? Are you only using Calico as networking stack or are you using it on top of flannel networking?

thanks

[zepp-div]

@DimShadoWWW , calico is used as a standalone network plugin

[zepp-div]

What additional information would be good to provide? Maybe there is an actual example of a deployment in Kubernetes?

[dsuch]

Hello @zepp-div,

the idea with Docker quickstart, Kubernetes, Vagrant, Ansible etc. is that you can use them as a starting point for your own customisation. Just like you did with calico which is not part of what we ship.

What you are doing here is that you are asking for support with something that we did not produce and it is not really possible to do it over GitHub because it is no longer part of what we ourselves built.

To get this kind of assistance, you would need commercial support. Please feel free to contact me on dsuch at zato.io to discuss it.

Thank you.

[dsuch]

No activity - I am closing the ticket. @zepp-div Please feel free to reopen if the question does not include any customisations that were not produced by us or please contact me on email per the message above.