airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.77k stars 4.04k forks source link

[source-okta] documentation on Okta domain unclear / links to misleading documentation #32966

Open sutru opened 10 months ago

sutru commented 10 months ago

Connector Name

source-okta

Connector Version

0.1.16

What step the error happened?

Configuring a new connector

Relevant information

https://docs.airbyte.com/integrations/sources/okta has a step to "Use API token from requirements and Okta domain". This leads to Okta documentation that tells you your domain should be used in format example.oktapreview.com or example.okta.com or example.okta-emea.com. The connector expects it to be in form example caused by line 46 in util.py

function:

 def get_url_base(config: Mapping[str, Any]) -> str:
    return config.get("base_url") or f"https://{config['domain']}.okta.com"

base_url is used when using OAuth; domain is used when using api key

This also prevents use of preview environments or other envs while using api keys.

Also in my case the calling method check_connection() line 344 of source.py eats the error instead returning "Failed to authenticate with the provided credentials" personally I feel this should return more info about the error i.e. the url any errors non-http related etc. but thats more a matter of personal opinion.

Loving Airbyte so far just from playing with it but I figured Id submit this in hopes of saving someone else some headache

Relevant log output

2023-11-29 23:41:42 INFO i.a.w.t.TemporalAttemptExecution(get):124 - Docker volume job log path: /tmp/workspace/592c653b-eb3b-49b4-9c92-8de85793e4ef/0/logs.log
2023-11-29 23:41:42 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.34
2023-11-29 23:41:42 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
2023-11-29 23:41:42 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-29 23:41:42 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-11-29 23:41:42 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-29 23:41:42 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-11-29 23:41:42 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-11-29 23:41:42 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-11-29 23:41:42 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-11-29 23:41:42 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-11-29 23:41:42 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-okta:0.1.16 exists...
2023-11-29 23:41:43 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-okta:0.1.16 was found locally.
2023-11-29 23:41:43 INFO i.a.w.p.DockerProcessFactory(create):140 - Creating docker container = source-okta-check-592c653b-eb3b-49b4-9c92-8de85793e4ef-0-bbkdi with resources io.airbyte.config.ResourceRequirements@161d5ea[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-11-29 23:41:43 INFO i.a.w.p.DockerProcessFactory(create):187 - Preparing command: docker run --rm --init -i -w /data/592c653b-eb3b-49b4-9c92-8de85793e4ef/0 --log-driver none --name source-okta-check-592c653b-eb3b-49b4-9c92-8de85793e4ef-0-bbkdi --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-okta:0.1.16 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.34 -e WORKER_JOB_ID=592c653b-eb3b-49b4-9c92-8de85793e4ef airbyte/source-okta:0.1.16 check --config source_config.json
2023-11-29 23:41:43 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):186 - Reading messages from protocol version 0.2.0
2023-11-29 23:41:44 ERROR i.a.w.i.VersionedAirbyteStreamFactory(internalLog):313 - Check failed
2023-11-29 23:41:44 INFO i.a.w.g.DefaultCheckConnectionWorker(run):117 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@187d499[status=failed,message='Failed to authenticate with the provided credentials',additionalProperties={}]
2023-11-29 23:41:44 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-11-29 23:41:44 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK --

Contribute

haroldsnyers commented 2 days ago

Any news on this ? I need to be able to set up okta domain with a different subdomain, it seems the UI is only accepting okta.com by default, while I should be able to put okta-preview.com or something else if the client has its own domain

'Unable to connect to stream users - HTTPSConnectionPool(host=\'https\', port=443): Max retries exceeded with url: /*.oktapreview.com.okta.com/api/v1/users?