search

elastic / connectors

Official Elastic connectors for third-party data sources
https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors.html
Other
86 stars 138 forks source link

Certificate verify error is thrown when syncing S3 #359

Closed ppf2 closed 1 year ago

ppf2 commented 1 year ago
image

Running a connector service locally (mac Monterey OS 12.6.1) using 8.6 branch of connectors-python (via make run, using bin/elastic-ingest does not make a difference). The connector service is connected to an Elasticsearch 8.6.0 instance on Elastic Cloud.

The connector service log shows:

[FMWK][14:58:47][DEBUG] Syncing 's3'
[FMWK][14:58:47][INFO] *** Connector pfVWy4UBbYkBe3GCEisS HEARTBEAT
[FMWK][14:58:47][INFO] Sync starts, Job id: e_pBzIUBbYkBe3GCk6ke
[FMWK][14:58:47][DEBUG] Pinging the Datasource `Amazon S3` backend
[FMWK][14:59:00][ERROR] Error while connecting to AWS.
Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1097, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1127, in _create_connection_transport
    await waiter
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 534, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 975, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/sources/s3.py", line 93, in ping
    self.bucket_list = await s3.list_buckets()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 140, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 160, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 89, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 197, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 156, in _do_get_response
    http_response = await self._send(request)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 241, in _send
    resp = await self.http_session.request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 982, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host s3.us-east-2.amazonaws.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')]
[FMWK][14:59:00][INFO] Sync done: 0 indexed, 0  deleted. (12 seconds)
[FMWK][14:59:00][CRITICAL] Cannot connect to host s3.us-east-2.amazonaws.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')]
Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1097, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1127, in _create_connection_transport
    await waiter
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 534, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 975, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/runner.py", line 196, in poll
    await self._one_sync(connector, es, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/runner.py", line 134, in _one_sync
    await connector.sync(es, self.idling, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/byoc.py", line 505, in sync
    await self.data_provider.ping()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/sources/s3.py", line 93, in ping
    self.bucket_list = await s3.list_buckets()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 140, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 160, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 89, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 197, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 156, in _do_get_response
    http_response = await self._send(request)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 241, in _send
    resp = await self.http_session.request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 982, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host s3.us-east-2.amazonaws.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')]
[FMWK][14:59:30][DEBUG] Polling every 30 seconds
[FMWK][14:59:30][DEBUG] Found 4 connectors
[FMWK][14:59:30][DEBUG] Connector vwHawoUB7sPvIFr5-5Dz of type elastic-crawler not supported, ignoring
[FMWK][14:59:30][DEBUG] Connector -QCdwoUB7sPvIFr5TCdc of type mongodb not supported, ignoring
[FMWK][14:59:30][DEBUG] Connector status is Status.CONNECTED
[FMWK][14:59:30][DEBUG] scheduler is disabled
[FMWK][14:59:30][DEBUG] Scheduling is disabled for s3
[FMWK][14:59:30][INFO] *** Connector baEuzIUB28qdopNRCptS HEARTBEAT
[FMWK][14:59:30][DEBUG] Connector status is Status.ERROR
[FMWK][14:59:30][DEBUG] scheduler is disabled
[FMWK][14:59:30][DEBUG] Scheduling is disabled for s3
[FMWK][14:59:30][INFO] *** Connector pfVWy4UBbYkBe3GCEisS HEARTBEAT

Our developers guide does not indicate any certificate setup steps that are necesssary.

awscli was installed on the Mac via https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html and the direct aws commands are running fine:

$ aws s3api list-objects --bucket ent-search
{
    "Contents": [
        {
            "Key": "_a/",
            "LastModified": "2023-01-19T01:37:46+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_a/_b/",
            "LastModified": "2023-01-19T01:37:53+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_a/_c/",
            "LastModified": "2023-01-19T01:37:58+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_a/_c/_d/",
            "LastModified": "2023-01-19T01:38:07+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_a/_c/_d/NOTICE.txt",
            "LastModified": "2023-01-19T01:38:26+00:00",
            "ETag": "\"c9c24cc0c0cdb2e0db1cdb93910406da\"",
            "Size": 73,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_test/",
            "LastModified": "2023-01-19T01:36:49+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_test/Knowledge - Innovation Hub.pdf",
            "LastModified": "2023-01-19T01:37:33+00:00",
            "ETag": "\"58196ae0524146cc96b6f0087b2e63dc\"",
            "Size": 512094,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_test/LICENSE.txt",
            "LastModified": "2023-01-19T01:37:32+00:00",
            "ETag": "\"e9a7d89ed55eb238cf0e85327684a082\"",
            "Size": 11348,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        },
        {
            "Key": "_test/de73deba02c60c2f.png",
            "LastModified": "2023-01-19T01:37:34+00:00",
            "ETag": "\"d041c61586583800fa1f8265f3b2733e\"",
            "Size": 7753,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "f158d34bbd60d91ff67d9fae7eeaf9a64788a2f80ffa44223fc7e6b8011dfd92"
            }
        }
    ]
}
$ aws s3 cp s3://ent-search/_a/_c/_d/NOTICE.txt .
download: s3://ent-search/_a/_c/_d/NOTICE.txt to ./NOTICE.txt

Also tried setting ca_bundle directly in ~/.aws/credentials file:

ca_bundle = /Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/certifi/cacert.pem

This results in a different error:

[FMWK][15:09:07][DEBUG] Syncing 's3'
[FMWK][15:09:07][INFO] *** Connector pfVWy4UBbYkBe3GCEisS HEARTBEAT
[FMWK][15:09:07][INFO] Sync starts, Job id: o_pLzIUBbYkBe3GCB9mP
[FMWK][15:09:07][DEBUG] Pinging the Datasource `Amazon S3` backend
[FMWK][15:09:29][ERROR] Error while connecting to AWS.
Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1097, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1127, in _create_connection_transport
    await waiter
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 637, in _on_handshake_complete
    raise handshake_exc
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 682, in _process_write_backlog
    ssldata = self._sslpipe.do_handshake(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 116, in do_handshake
    self._sslobj = self._context.wrap_bio(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 527, in wrap_bio
    return self.sslobject_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 866, in _create
    sslobj = context._wrap_bio(
ssl.SSLError: Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/sources/s3.py", line 93, in ping
    self.bucket_list = await s3.list_buckets()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 140, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 160, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 89, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 197, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 156, in _do_get_response
    http_response = await self._send(request)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 241, in _send
    resp = await self.http_session.request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 984, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host s3.us-east-2.amazonaws.com:443 ssl:default [Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)]
[FMWK][15:09:29][INFO] Sync done: 0 indexed, 0  deleted. (22 seconds)
[FMWK][15:09:29][CRITICAL] Cannot connect to host s3.us-east-2.amazonaws.com:443 ssl:default [Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)]
Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1097, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1127, in _create_connection_transport
    await waiter
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 637, in _on_handshake_complete
    raise handshake_exc
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 682, in _process_write_backlog
    ssldata = self._sslpipe.do_handshake(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 116, in do_handshake
    self._sslobj = self._context.wrap_bio(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 527, in wrap_bio
    return self.sslobject_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 866, in _create
    sslobj = context._wrap_bio(
ssl.SSLError: Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/runner.py", line 196, in poll
    await self._one_sync(connector, es, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/runner.py", line 134, in _one_sync
    await connector.sync(es, self.idling, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/byoc.py", line 505, in sync
    await self.data_provider.ping()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/sources/s3.py", line 93, in ping
    self.bucket_list = await s3.list_buckets()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 140, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/client.py", line 160, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 89, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 197, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 156, in _do_get_response
    http_response = await self._send(request)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 241, in _send
    resp = await self.http_session.request(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/aiohttp/connector.py", line 984, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host s3.us-east-2.amazonaws.com:443 ssl:default [Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)]
[FMWK][15:09:59][DEBUG] Polling every 30 seconds
[FMWK][15:10:00][DEBUG] Found 4 connectors
[FMWK][15:10:00][DEBUG] Connector vwHawoUB7sPvIFr5-5Dz of type elastic-crawler not supported, ignoring
[FMWK][15:10:00][DEBUG] Connector -QCdwoUB7sPvIFr5TCdc of type mongodb not supported, ignoring
[FMWK][15:10:00][DEBUG] Connector status is Status.CONNECTED
[FMWK][15:10:00][DEBUG] scheduler is disabled
[FMWK][15:10:00][DEBUG] Scheduling is disabled for s3
[FMWK][15:10:00][INFO] *** Connector baEuzIUB28qdopNRCptS HEARTBEAT
[FMWK][15:10:00][DEBUG] Connector status is Status.ERROR
[FMWK][15:10:00][DEBUG] scheduler is disabled
[FMWK][15:10:00][DEBUG] Scheduling is disabled for s3
[FMWK][15:10:00][INFO] *** Connector pfVWy4UBbYkBe3GCEisS HEARTBEAT
[FMWK][15:10:00][CRITICAL] ConflictError(409, 'version_conflict_engine_exception', '[pfVWy4UBbYkBe3GCEisS]: version conflict, required seqNo [326], primary term [6]. current document has seqNo [328] and primary term [6]')
Traceback (most recent call last):
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/runner.py", line 196, in poll
    await self._one_sync(connector, es, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/runner.py", line 134, in _one_sync
    await connector.sync(es, self.idling, sync_now)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/byoc.py", line 477, in sync
    await self.sync_doc()
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/byoc.py", line 317, in sync_doc
    await self.index.save(self)
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/connectors/byoc.py", line 108, in save
    return await self.client.update(
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/elasticsearch/_async/client/__init__.py", line 4484, in update
    return await self.perform_request(  # type: ignore[return-value]
  File "/Users/pius/Elastic/_products/Enterprise_Search/connectors-python-8.6/lib/python3.10/site-packages/elasticsearch/_async/client/_base.py", line 321, in perform_request
    raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
elasticsearch.ConflictError: ConflictError(409, 'version_conflict_engine_exception', '[pfVWy4UBbYkBe3GCEisS]: version conflict, required seqNo [326], primary term [6]. current document has seqNo [328] and primary term [6]')

I filed the ConflictError in a separate issue (https://github.com/elastic/connectors-python/issues/358) for this looks like an unrelated issue from the S3 connection/certificate problem.

ppf2 commented 1 year ago

I am able to get around this by running, courtesy of https://github.com/aio-libs/aiohttp/issues/955#issuecomment-679998400

$ /System/Volumes/Data/Applications/Python\ 3.10/Install\ Certificates.command
 -- pip install --upgrade certifi
Collecting certifi
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Installing collected packages: certifi
Successfully installed certifi-2022.12.7
 -- removing any existing file or link
 -- creating symlink to certifi certificate bundle
 -- setting permissions
 -- update complete

We may want to document this caveat either in the section where we talk about Python requirements, or in a troubleshooting section in case others are running into the same issue.