search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.55k stars 4.01k forks source link

[training] [source-google-search-console] Error creating source with certain domains on OSS but not Cloud #34327

Closed rwask closed 8 months ago

rwask commented 8 months ago

Connector Name

source-google-search-console

Connector Version

v1.3.6

What step the error happened?

Configuring a new connector

Relevant information

User attempted to create a new Google Search Concole connector but having an issue with certain domains when creating the source connector on OSS but not on Cloud.

The issue is not reproducible by the user when attempting to create the same source in Airbyte Cloud, only with OSS does the source fail to be created.

On Airbyte OSS I get the following error: "InvalidSiteURLValidationError('The following URLs are not permitted: sc-domain:sipgate.io')"

The error is thrown on domains, such as: sc-domain:sipgate.io sc-domain:secure.live.sipgate.de sc-domain:sipgate.co.uk

However, these domains work: sc-domain:status.sipgate.de sc-domain:sipgate.de

Airbyte OSSd oes not accept these domains. The only difference is the authentication method. Airbyte Cloud uses OAuth whereas Airbyte OSS uses a service account.

See screenshot Airbyte OSS: Screenshot Airbyte OSS not working image

See image below provided by user regarding the same domains working on Airbyte Cloud: image

Relevant log output

024-01-16 09:28:39 platform > Docker volume job log path: /tmp/workspace/121f582f-12ab-4d65-b897-697fe50d4557/0/logs.log
2024-01-16 09:28:39 platform > Executing worker wrapper. Airbyte version: 0.50.41
2024-01-16 09:28:39 platform > Attempt 0 to save workflow id for cancellation
2024-01-16 09:28:39 platform >
2024-01-16 09:28:39 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-01-16 09:28:39 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-01-16 09:28:39 platform > ----- START check-orchestrator -----
2024-01-16 09:28:39 platform >
2024-01-16 09:28:39 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-01-16 09:28:39 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-01-16 09:28:39 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-01-16 09:28:39 platform > Checking if airbyte/source-google-search-console:1.3.6 exists…
2024-01-16 09:28:39 platform > airbyte/source-google-search-console:1.3.6 was found locally.
2024-01-16 09:28:39 platform > Creating docker container = source-google-search-console-check-121f582f-12ab-4d65-b897-697fe50d4557-0-ftokp with resources io.airbyte.config.ResourceRequirements@6186155b[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-01-16 09:28:39 platform > Preparing command: docker run --rm --init -i -w /data/121f582f-12ab-4d65-b897-697fe50d4557/0 --log-driver none --name source-google-search-console-check-121f582f-12ab-4d65-b897-697fe50d4557-0-ftokp --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-google-search-console:1.3.6 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.41 -e WORKER_JOB_ID=121f582f-12ab-4d65-b897-697fe50d4557 airbyte/source-google-search-console:1.3.6 check --config source_config.json
2024-01-16 09:28:39 platform > Reading messages from protocol version 0.2.0
2024-01-16 09:28:40 platform > Check failed
2024-01-16 09:28:41 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@4e200677[status=failed,message="InvalidSiteURLValidationError('The following URLs are not permitted: test.de')",additionalProperties={}]
2024-01-16 09:28:41 platform >
2024-01-16 09:28:41 platform > ----- END check-orchestrator -----
2024-01-16 09:28:41 platform >

Contribute

octavia-squidington-iii commented 8 months ago

Zendesk ticket #3984 has been linked to this issue.

marcosmarxm commented 8 months ago

Awesome.