airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.36k stars 4.16k forks source link

[source-s3] Set source to minio error `304 - Check failed` #46732

Open vitaaaaa1 opened 1 month ago

vitaaaaa1 commented 1 month ago

Connector Name

airbyte/source-s3

Connector Version

4.8.4

What step the error happened?

None

Relevant information

I used docker to install minio for testing

docker ps
15207ab104a4   minio/minio            "/usr/bin/docker-ent…"   13 days ago   Up 13 days   0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp   minio

These are the parameters set when airbyte connects to minio

Bucket:airtype
AWS Access Key ID:xxxxx
AWS Secret Access Key:xxxx
Endpoint:10.8.0.84:9000

image I tested successfully connecting to this bucket using the s3 connection tool, so rule out the minio secret_key permission problem

Relevant log output

2024-10-11 05:40:53 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CLAIM — (workloadId = 69589781-7828-43c5-9f63-8925b1c1ccc2_2a38bf9a-1901-4ae7-9da8-123898bea445_0_check) — (dataplaneId = local)
2024-10-11 05:41:03 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-10-11 05:41:03 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- START CHECK -----
2024-10-11 05:41:03 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-10-11 05:41:03 INFO i.a.c.ConnectorWatcher(processConnectorOutput):114 - Connector exited, processing output
2024-10-11 05:41:03 INFO i.a.c.ConnectorWatcher(getConnectorOutputStream):151 - Output file jobOutput.json found
2024-10-11 05:41:03 INFO i.a.c.ConnectorWatcher(processConnectorOutput):117 - Connector exited with exit code 0
2024-10-11 05:41:03 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):189 - Reading messages from protocol version 0.2.0
2024-10-11 05:41:03 WARN i.a.m.l.MetricClientFactory(getMetricClient):43 - MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2024-10-11 05:41:04 ERROR i.a.w.i.VersionedAirbyteStreamFactory(internalLog):304 - Check failed
2024-10-11 05:41:04 INFO i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):231 - Checking for optional control message...
2024-10-11 05:41:04 INFO i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):253 - Optional control message not found. Skipping...
2024-10-11 05:41:04 INFO i.a.c.ConnectorWatcher(saveConnectorOutput):162 - Writing output of 69589781-7828-43c5-9f63-8925b1c1ccc2_2a38bf9a-1901-4ae7-9da8-123898bea445_0_check to the doc store
2024-10-11 05:41:04 INFO i.a.c.ConnectorWatcher(markWorkloadSuccess):167 - Marking workload 69589781-7828-43c5-9f63-8925b1c1ccc2_2a38bf9a-1901-4ae7-9da8-123898bea445_0_check as successful
2024-10-11 05:41:04 INFO i.a.c.ConnectorWatcher(exitProperly):215 - Deliberately exiting process with code 0.
2024-10-11 05:41:04 INFO i.a.c.i.LineGobbler(voidCall):166 - 
2024-10-11 05:41:04 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- END CHECK -----

Contribute

vitaaaaa1 commented 1 month ago

I had this problem trying to use either 9000 or 9001 ports, or miniodurl with or without http

marcosmarxm commented 2 weeks ago

I've added the issue to the connector backlog, but it won't be prioritized for now. If this issue is affecting you, please comment or give it a 👍. This helps the team understand the number of users facing the same problem so that high-impact issues can be prioritized.