search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.76k stars 3.8k forks source link

[source-Confluence] REST API encounters a 403 error even when after passing the test when creating source #33737

Open mw19930312 opened 6 months ago

mw19930312 commented 6 months ago

Connector Name

source-Confluence

Connector Version

0.2.0

What step the error happened?

During the sync

Relevant information

The sync failed due to airbyte_cdk.sources.declarative.exceptions.ReadException: Request to https://fortify-ai.atlassian.net/wiki/rest/api/audit?limit=25 failed with status code 403 and error message None. However, the credentials that I provided passes all tests when creating the source. mw_thu12_gmail_com_job_6860440_attempt_5_txt.txt

The source id is e3a60886-34a4-4f8e-8bbb-c3a8b54d4419 and the connection id is 0e1f0fac-6303-4e59-a69e-0e2d6bbc07aa

Relevant log output

"destinationWriteStartTime" : 1703212599454,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "recordsEmitted" : 0,
    "replicationEndTime" : 0,
    "replicationStartTime" : 1703212599378,
    "sourceReadEndTime" : 1703213222277,
    "sourceReadStartTime" : 1703212606173,
    "sourceStateMessagesEmitted" : 0
  },
  "streamStats" : [ ]
}
2023-12-22 02:47:06 replication-orchestrator > failures: [ {
  "failureOrigin" : "source",
  "failureType" : "system_error",
  "internalMessage" : "Request to https://fortify-ai.atlassian.net/wiki/rest/api/audit?limit=25 failed with status code 403 and error message None",
  "externalMessage" : "Something went wrong in the connector. See the logs for more details.",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 6860440,
    "from_trace_message" : true,
    "connector_command" : "read"
  },
  "stacktrace" : "Traceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 200, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 114, in run\n    yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 156, in read\n    yield from self.source.read(self.logger, config, catalog, state)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py\", line 136, in read\n    yield from super().read(logger, config, catalog, state)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 142, in read\n    raise e\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 122, in read\n    yield from self._read_stream(\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 197, in _read_stream\n    for record in record_iterator:\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 328, in _read_full_refresh\n    for record_data_or_message in record_data_or_messages:\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py\", line 104, in read_records\n    yield from self.retriever.read_records(stream_slice)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 304, in read_records\n    for stream_data in self._read_pages(self._parse_records, self.state, stream_slice):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 282, in _read_pages\n    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 256, in _fetch_next_page\n    return self.requester.send_request(\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py\", line 399, in send_request\n    return self._validate_response(response)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py\", line 496, in _validate_response\n    raise ReadException(error_message)\nairbyte_cdk.sources.declarative.exceptions.ReadException: Request to https://fortify-ai.atlassian.net/wiki/rest/api/audit?limit=25 failed with status code 403 and error message None\n",
  "timestamp" : 1703212613900
}, {
  "failureOrigin" : "source",
  "internalMessage" : "Source process exited with non-zero exit code 1",
  "externalMessage" : "Something went wrong within the source connector",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 6860440,
    "connector_command" : "read"
  },
  "stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromSrcAndWriteToDstRunnable$8(DefaultReplicationWorker.java:382)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
  "timestamp" : 1703213222287
} ]
2023-12-22 02:47:06 replication-orchestrator > Returning output...
2023-12-22 02:47:06 replication-orchestrator > 
2023-12-22 02:47:06 replication-orchestrator > ----- END REPLICATION -----
2023-12-22 02:47:06 replication-orchestrator > 
2023-12-22 02:47:06 replication-orchestrator > Writing async status SUCCEEDED for KubePodInfo[namespace=jobs, name=orchestrator-repl-job-6860440-attempt-4, mainContainerInfo=KubeContainerInfo[image=airbyte/container-orchestrator:dev-124bd07262, pullPolicy=IfNotPresent]]...
2023-12-22 02:47:10 platform > State Store reports orchestrator pod orchestrator-repl-job-6860440-attempt-4 succeeded
2023-12-22 02:47:12 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=5, totalCompleteFailures=5, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 13 minutes 30 seconds
2023-12-22 02:47:12 platform > Failing job: 6860440, reason: Job failed after too many retries for connection 0e1f0fac-6303-4e59-a69e-0e2d6bbc07aa

Contribute

mw19930312 commented 6 months ago

Any update?