search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.24k stars 4.15k forks source link

Auth0 low code cdk is failing to connect and retrieve data #29789

Open vasilisgav opened 1 year ago

vasilisgav commented 1 year ago

Connector Name

Auth0

Connector Version

0.4.0

What step the error happened?

Configuring a new connector

Revelant information

Upon configuring auth0 connector with the authentication Method Oath2 Confidential Application the following error appears

"internalMessage" : "Request to https://****REDUCT_THIS*****.com/clients?per_page=5 failed with status code 404 and error message None",
  "externalMessage" : "Something went wrong in the connector. See the logs for more details.",

the request should have been https://REDUCT_THIS.com**/api/v2/**clients?per_page=5

Relevant log output

No response

Contribute

vasilisgav commented 1 year ago

Tagging @sajarin @marcosmarxm @btkcodedev

vasilisgav commented 1 year ago

Tried using the fix

   requester:
     type: HttpRequester
-    url_base: "{{ config['base_url'] }}"
+    url_base: "{{ config['credentials']['audience'] }}"

but now the error is

2023-08-24 06:34:17 INFO i.a.w.g.DefaultCheckConnectionWorker(run):115 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@6bf34e4[status=failed,message='Unable to connect to stream clients - Request to https://****REDUCT_THIS*****.com/api/v2/clients?per_page=5 failed with status code 400 and error message Bad HTTP authentication header format',additionalProperties={}]

Can you help me identify where the headers are created using the low cdk code?

marcosmarxm commented 1 year ago

@sajarin you're working on this correct?

sajarin commented 1 year ago

hey @vasilisgav apologies for the delay, can you confirm whether or not the changes in #29804 fixes the issues you encountered here?

vasilisgav commented 1 year ago

Hi @sajarin, thanks for the update! I upgraded version to 0.4.1 and testing the connection is not working. here are the logs:

2023-09-28 05:29:38 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-09-28 05:29:38 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-09-28 05:29:38 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-09-28 05:29:38 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2023-09-28 05:29:38 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-auth0:0.4.1 exists...
2023-09-28 05:29:38 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-auth0:0.4.1 was found locally.
2023-09-28 05:29:38 INFO i.a.w.p.DockerProcessFactory(create):130 - Creating docker container = source-auth0-check-7eb9ccde-99ee-421c-ac2b-05ff680efe82-0-ucntt with resources io.airbyte.config.ResourceRequirements@5caf99ff[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts null
2023-09-28 05:29:38 INFO i.a.w.p.DockerProcessFactory(create):175 - Preparing command: docker run --rm --init -i -w /data/7eb9ccde-99ee-421c-ac2b-05ff680efe82/0 --log-driver none --name source-auth0-check-7eb9ccde-99ee-421c-ac2b-05ff680efe82-0-ucntt --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e STRICT_COMPARISON_NORMALIZATION_WORKSPACES= -e WORKER_CONNECTOR_IMAGE=airbyte/source-auth0:0.4.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e STRICT_COMPARISON_NORMALIZATION_TAG=strict_comparison2 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e FEATURE_FLAG_CLIENT= -e AIRBYTE_VERSION=0.42.0 -e WORKER_JOB_ID=7eb9ccde-99ee-421c-ac2b-05ff680efe82 airbyte/source-auth0:0.4.1 check --config source_config.json
2023-09-28 05:29:38 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):107 - Reading messages from protocol version 0.2.0
2023-09-28 05:29:39 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):164 - Encountered an error trying to connect to stream clients. Error: 
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name does not resolve
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 235, in _get_request_options
    auth_options_method(),
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 39, in get_auth_header
    return {"Authorization": f"Bearer {self.get_access_token()}"}
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 44, in get_access_token
    token, expires_in = self.refresh_access_token()
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/auth/oauth.py", line 118, in refresh_access_token
    response_json = self._get_refresh_access_token_response()
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 93, in _get_refresh_access_token_response
    if e.response.status_code == 429 or e.response.status_code >= 500:
AttributeError: 'NoneType' object has no attribute 'status_code'
2023-09-28 05:29:39 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):164 - Check failed
2023-09-28 05:29:40 INFO i.a.w.g.DefaultCheckConnectionWorker(run):120 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@626b406b[status=failed,message="Unable to connect to stream clients - 'NoneType' object has no attribute 'status_code'"]
2023-09-28 05:29:40 INFO i.a.w.t.TemporalAttemptExecution(get):169 - Stopping cancellation check scheduling...
2023-09-28 05:29:40 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-09-28 05:29:40 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-09-28 05:29:40 INFO i.a.c.i.LineGobbler(voidCall):149 -
vasilisgav commented 1 year ago

In addition, for some reason 0.4.1 is not showing as the latest version, I had to manually change the version:

image
Daesgar commented 1 year ago

I'm currently finding the same issues that @vasilisgav described.

I had to manually put the 0.4.1 tag in the Airbyte database because it is not being shown in the connectors list.

The last error shared by him about the status code arises because the base URL is not correctly concatenated with the oauth/token path. It is needed to add a slash at the end of the base URL.

Anyway, after adding it and testing again, a new error arises because when it requests a token to the oauth/token, it returns a 403.

I'm guessing it is due to the authentication flow setting the grant_type to something different from client_credentials, but I cannot find exactly where the grant_type is configured.

2023-10-02 11:07:44 ERROR i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - Encountered an error trying to connect to stream clients. Error:                                                                                                             
 Traceback (most recent call last):                                                                                                                                                                                                                             
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 50, in check_availability                                                                                                                       
    get_first_record_for_slice(stream, stream_slice)                                                                                                                                                                                                            
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py", line 38, in get_first_record_for_slice                                                                                                                      
    return next(records_for_slice)                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 104, in read_records                                                                                                                                
    yield from self.retriever.read_records(stream_slice)                                                                                                                                                                                                        
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 304, in read_records                                                                                                                       
    for stream_data in self._read_pages(self._parse_records, self.state, stream_slice):                                                                                                                                                                         
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 282, in _read_pages                                                                                                                        
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)                                                                                                                                                                               
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 256, in _fetch_next_page                                                                                                                   
    return self.requester.send_request(                                                                                                                                                                                                                         
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 392, in send_request                                                                                                                         
    headers=self._request_headers(stream_state, stream_slice, next_page_token, request_headers),                                                                                                                                                                
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 251, in _request_headers                                                                                                                     
    headers = self._get_request_options(                                                                                                                                                                                                                        
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 235, in _get_request_options                                                                                                                 
    auth_options_method(),                                                                                                                                                                                                                                      
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 39, in get_auth_header                                                                                                            
    return {"Authorization": f"Bearer {self.get_access_token()}"}                                                                                                                                                                                               
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 44, in get_access_token                                                                                                           
    token, expires_in = self.refresh_access_token()                                                                                                                                                                                                             
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/auth/oauth.py", line 118, in refresh_access_token                                                                                                                                
    response_json = self._get_refresh_access_token_response()                                                                                                                                                                                                   
  File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry                                                                                                                                                                            
    ret = target(*args, **kwargs)                                                                                                                                                                                                                               
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 90, in _get_refresh_access_token_response                                                                                         
    response.raise_for_status()                                                                                                                                                                                                                                 
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status                                                                                                                                                              
    raise HTTPError(http_error_msg, response=self)                                                                                                                                                                                                              
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://{{MY_BASE_URL}}/oauth/token                                                                                                                                                  

During handling of the above exception, another exception occurred:                                                                                                                                                                                             

Traceback (most recent call last):                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py", line 42, in check_connection                                                                                                                            
    stream_is_available, reason = availability_strategy.check_availability(stream, logger, source)                                                                                                                                                              
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 56, in check_availability                                                                                                                       
    is_available, reason = self.handle_http_error(stream, logger, source, error)                                                                                                                                                                                
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py", line 89, in handle_http_error                                                                                                                        
    response_error_message = stream.parse_response_error_message(error.response)                                                                                                                                                                                
AttributeError: 'DeclarativeStream' object has no attribute 'parse_response_error_message'
vasilisgav commented 1 year ago

@Daesgar I created this one https://github.com/airbytehq/airbyte/issues/31858 to continue