search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.72k stars 3.78k forks source link

[connector-builder] SSL certificate failed into Docker container - Tutorial APILayer #36162

Closed AlyssonCora closed 2 months ago

AlyssonCora commented 3 months ago

Platform Version

0.50.50

What step the error happened?

Other

Revelant information

Hello! I tried to create a connector using the UI builder. I went to do the Apilayer tutorial and had an SSL certificate error when consulting it.

I tested it with curl on my machine's command line and it worked.

So, I connected it to the container using the command below and did a curl to confirm the error. docker exec -it airbyte-server /bin/bash

Below are example of local curl and curl running inside the container:

curl --location 'https://api.apilayer.com/exchangerates_data/latest' \
--header 'apikey: [API-KEY-FROM-APILAYER]'

I tested connecting with another API and it worked, both on airbyte and locally. But api.apilayer.com and other connectors like "Publics API" give the same SSL certificate verification error, only in docker conainer.

Do you have any idea how to solve this problem?

On Airbyte:

image image

On localhost:

image

Relevant log output

ERROR
Something went wrong in the connector. See the logs for more details. - Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='apilayer.com', port=443): Max retries exceeded with url: /exchangerates_data/latest (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 128, in read
    yield from self._read_stream(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 222, in _read_stream
    for record in record_iterator:
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh
    for record_data_or_message in stream_instance.read_full_refresh(configured_stream.cursor_field, logger, self._slice_logger):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 121, in read_full_refresh
    yield from self.read_records(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 104, in read_records
    yield from self.retriever.read_records(self.get_json_schema(), stream_slice)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 323, in read_records
    for stream_data in self._read_pages(record_generator, self.state, stream_slice):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 287, in _read_pages
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 420, in _fetch_next_page
    return self.requester.send_request(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 460, in send_request
    response = self._send_with_retry(request, log_formatter=log_formatter)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 506, in _send_with_retry
    return backoff_handler(user_backoff_handler)(request, log_formatter=log_formatter)  # type: ignore
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 534, in _send
    response: requests.Response = self._session.send(request)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='apilayer.com', port=443): Max retries exceeded with url: /exchangerates_data/latest (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))
ERROR
None - Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 249, in _read_stream
    yield from AirbyteEntrypoint(source).read(source.spec(self.logger), config, configured_catalog, {})
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 163, in read
    yield from self.source.read(self.logger, config, catalog, state)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 167, in read
    yield from super().read(logger, config, catalog, state)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 175, in read
    raise AirbyteTracedException(message=error_message)
airbyte_cdk.utils.traced_exception.AirbyteTracedException: None
AlyssonCora commented 2 months ago

Problem with security tools at the company where I work.