zatosource / zato

ESB, SOA, REST, APIs and Cloud Integrations in Python
https://zato.io
GNU Affero General Public License v3.0
1.12k stars 240 forks source link

Random timeouts on basic_http outgoing invokes #1202

Closed yhorndt closed 2 years ago

yhorndt commented 2 years ago

Hi, I recently started using zato and am currently in the process of building the following concept: I use the docker quickstart 3.2 on a macbook pro, osx 12.4. (so the calls come from my local machine) All the services are in one file, but do get recognized by zato as different services.

My problem is, that I get random timeout errors for calling one of the outgoing rest services. Sometimes it keeps timeoutting the first channel, sometimes the second. When I get a timeout, it usually keeps throwing timeouts for a while. I also verify the service is up, by using postman, with fast responses and no timeouts.

conn = self.out.plain_http['Afas'].conn
afas_response = conn.get(self.cid, {'connector': 'organisations?take=1000'}, headers=headers)
if afas_response.status_code == 200:
      self.logger.info('status = 200')
      return afas_response.json()

Outgoing Afas connection has a timeout of 300s. (returns data via postman in under a second). The entire process get's started by a api call to a rest channel in zato. /api/v1/afas

The error I get for POST calls is usually:

ERROR - 15303:Dummy-166 - zato.server.connection.http_soap.channel:0 - Caught an exception,  status_code:`HTTPStatus.INTERNAL_SERVER_ERROR`, `Traceback (most recent call last):
socket.timeout: timed out

followed by During handling of the above exception, another exception occurred:
which is another timeout
followed by During handling of the above exception, another exception occurred:
which is max retry error
followed by During handling of the above exception, another exception occurred:
which is max retry error
etc.

Full Stacktrace 1

For GET Call it is a little different but similar: Full Stacktrace 1

note: I have changed some sensitive info out of the domains. Also, the outgoing calls should return data in under 0.5s, so the 10s timeout trigger should be more than plenty.

Thanks, maybe you've seen this before.

dsuch commented 2 years ago

Hello,

exception "socket.timeout: timed out" is a common TCP-level one, reported by the container's kernel and it should be handled accordingly. In itself, it is not directly related to Zato.

Regards.