Jenkins - Max retries exceeded

lukaszgryglicki commented 4 years ago

Hi, I'm getting this error, what can I do?
The error message seems to point that ES timeouts and after a few retries, it fails. But ES is fine, checked its state - its healthy and other commands executed from different terminals are working OK
Tried with bulk settings: 1000, then 100 finally 10 - nothing helps.
Here are logs, any suggestions p2o.py --enrich --index jenkins-raw --index-enrich jenkins -e [...]--bulk-size 100 --scroll-size 100 --db-host [...] --db-sortinghat [...] --db-user [...]--db-password [...] jenkins https://build.opnfv.org/ci/ --no-archive

2020-05-21 07:07:24,473 Retrying (Retry(total=13, connect=21, read=0, redirect=5, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))': /jenkins-raw/_bulk?refresh=true
2020-05-21 07:07:24,795 Error feeding raw from jenkins (https://build.opnfv.org/ci/): HTTPSConnectionPool(host='[...]', port=[...]): Max retries exceeded with url: /jenkins-raw/_bulk?refresh=true (Caused by ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe')))
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 899, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 869, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 594, in write
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 667, in urlopen
    **response_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/dist-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='[...]', port=[...]): Max retries exceeded with url: /jenkins-raw/_bulk?refresh=true (Caused by ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/repos/grimoirelab-elk/grimoire_elk/elk.py", line 163, in feed_backend
    ocean_backend.feed(**params)
  File "/repos/grimoirelab-elk/grimoire_elk/raw/elastic.py", line 228, in feed
    self.feed_items(items)
  File "/repos/grimoirelab-elk/grimoire_elk/raw/elastic.py", line 254, in feed_items
    self._items_to_es(items_pack)
  File "/repos/grimoirelab-elk/grimoire_elk/raw/elastic.py", line 287, in _items_to_es
    inserted = self.elastic.bulk_upload(json_items, field_id)
  File "/repos/grimoirelab-elk/grimoire_elk/elastic.py", line 351, in bulk_upload
    new_items += self.safe_put_bulk(url, bulk_json)
  File "/repos/grimoirelab-elk/grimoire_elk/elastic.py", line 182, in safe_put_bulk
    res = self.requests.put(url + '?refresh=true', data=bulk_json, headers=headers)
  File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 593, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='[...], port=[...]): Max retries exceeded with url: /jenkins-raw/_bulk?refresh=true (Caused by ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe')))
2020-05-21 07:07:24,810 [jenkins] Done collection for https://build.opnfv.org/ci/
2020-05-21 07:07:24,810 Backend feed completed

valeriocos commented 4 years ago

Hi @lukaszgryglicki , sorry for the late reply!

The problem may be due to the presence of a large document that exceeds the max content of an HTTP request. You can try to modify your docker-compose to change the value of http.max_content_length. You can find an example of it at: https://github.com/crossminer/scava-deployment/blob/master/docker-compose.yml#L140

Hope this helps!

EDIT: When working on that project, I remember that we spend some time to find a good combination between the http.max_content_length and bulk size (to avoid the error and not affect too much the time to import the documents to ES). In that case, it was: http.max_content_length=500mb and bulk size = 500

lukaszgryglicki commented 4 years ago

Great, thanks, we're using managed ES, but maybe it is possible to set that value there.

valeriocos commented 4 years ago

Hi @lukaszgryglicki , I'm closing this issue, feel free to reopen it if needed.

chaoss / grimoirelab-elk

Jenkins - Max retries exceeded #879