tabular-io / iceberg-kafka-connect

Apache License 2.0
203 stars 46 forks source link

The AWS Access Key Id you provided does not exist in our records #154

Closed NhatDuy11 closed 10 months ago

NhatDuy11 commented 11 months ago

Here is my iceberg sink connector configuration: { { "name": "IcebergSinkConnectorConnector_4", "config": { "iceberg.catalog.s3.secret-access-key": "###########", "iceberg.catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "iceberg.catalog.credential": "software.amazon.awssdk.auth.credentials.SimpleAwsCredentialsProvider", "bootstrap-servers": "localhost:9092", "iceberg.catalog.client.region": "us-east-1", "iceberg.catalog.uri": "thrift://hive:9083", "iceberg.s3a.endpoint": "http://192.168.1.206", "value.converter.schema.registry.url": "http://localhost:8081", "iceberg.catalog.warehouse": "s3a://data/warehouse", "key.converter.schema.registry.url": "http://localhost:8081", "iceberg.catalog.s3.access-key-id": "########", "name": "IcebergSinkConnectorConnector_4", "connector.class": "io.tabular.iceberg.connect.IcebergSinkConnector", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "io.confluent.connect.avro.AvroConverter", "topics": "TIME_ICEBERG", "iceberg.tables": "cnd.iceberg_time"

} } Can someone help me with this error: Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: EGWB54QFPR4KKS3N, Extended Request ID: Q0PLcS077oYiTKn1he01+Rjm57iKdtd98FQ5Ef4no1pMiBWLZM+nTeeTKGR4aFsOXVs6VYe4Q38=) currently i am using : iceberg-kafka-connect-runtime-hive-0.6.2 Thank you everyone for reading !

NhatDuy11 commented 11 months ago

currently i am using : iceberg-kafka-connect-runtime-hive-0.6.2

bryanck commented 11 months ago

You have set iceberg.s3a.endpoint, did you mean to set iceberg.catalog.s3.endpoint?

NhatDuy11 commented 10 months ago

@bryanck Hi, I also tried this case iceberg.s3.endpoint and got this error: Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])): Profile file contained no credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]

NhatDuy11 commented 10 months ago

@bryanck thank you ,I have sink the data into minio,because I am missing the config credentials of the aws service.

bryanck commented 10 months ago

Were you able to get this working?

bryanck commented 10 months ago

I'm going to close this, feel free to reopen or open a new ticket if you have new information.