search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.35k stars 4.16k forks source link

[destination-DynamoDB] Custom Endpoint fails with unknown signer type: AWSDynamodbSignerType #29977

Closed kevinmartin closed 5 days ago

kevinmartin commented 1 year ago

Connector Name

destination-dynamodb

Connector Version

0.1.7

What step the error happened?

Configuring a new connector

Revelant information

Steps to reproduce:

Example Docker Compose Project:

version: "3.9"
services:
  localstack:
    image: localstack/localstack
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - DEBUG=${DEBUG-}
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "./.localstack:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
    networks:
      - default
      - airbyte
networks:
  default:
  airbyte:
    name: airbyte_airbyte_internal
    external: true

Upon checking the connection, the Airbyte working is able to ping LocalStack correctly (a bad Endpoint value just causes it to timeout).

Source of the issue: https://github.com/search?q=repo%3Aairbytehq%2Fairbyte+AWSDynamodbSignerType&type=code

The AWSDynamodbSignerType isn't a valid signer type.

Relevant log output

Configuration check failed Could not connect to the DynamoDB table with the provided configuration. unknown signer type: AWSDynamodbSignerType

2023-08-29 22:03:46 INFO i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/c0f91771-f680-43af-b559-5aaa0c745c0b/0/logs.log
2023-08-29 22:03:46 INFO i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.50.20
2023-08-29 22:03:46 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):258 - Attempt 0 to save workflow id for cancellation
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_MEMORY_LIMIT: '50Mi'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_MEMORY_REQUEST: '25Mi'
2023-08-29 22:03:46 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-08-29 22:03:46 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-08-29 22:03:46 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-08-29 22:03:46 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-08-29 22:03:46 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/destination-dynamodb:0.1.7 exists...
2023-08-29 22:03:47 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/destination-dynamodb:0.1.7 was found locally.
2023-08-29 22:03:47 INFO i.a.w.p.DockerProcessFactory(create):143 - Creating docker container = destination-dynamodb-check-c0f91771-f680-43af-b559-5aaa0c745c0b-0-wezor with resources io.airbyte.config.ResourceRequirements@2429ea15[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-08-29 22:03:47 INFO i.a.w.p.DockerProcessFactory(create):196 - Preparing command: docker run --rm --init -i -w /data/c0f91771-f680-43af-b559-5aaa0c745c0b/0 --log-driver none --name destination-dynamodb-check-c0f91771-f680-43af-b559-5aaa0c745c0b-0-wezor --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/destination-dynamodb:0.1.7 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.20 -e WORKER_JOB_ID=c0f91771-f680-43af-b559-5aaa0c745c0b airbyte/destination-dynamodb:0.1.7 check --config source_config.json
2023-08-29 22:03:47 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):177 - Reading messages from protocol version 0.2.0
2023-08-29 22:03:47 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - Running destination under deployment mode: OSS
2023-08-29 22:03:47 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - Starting destination: io.airbyte.integrations.destination.dynamodb.DynamodbDestination
2023-08-29 22:03:47 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - integration args: {check=null, config=source_config.json}
2023-08-29 22:03:47 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - Running integration: io.airbyte.integrations.destination.dynamodb.DynamodbDestination
2023-08-29 22:03:47 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - Command: CHECK
2023-08-29 22:03:48 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-08-29 22:03:48 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):309 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-08-29 22:03:48 ERROR i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - Exception attempting to access the DynamoDB table: 
Stack Trace: java.lang.IllegalArgumentException: unknown signer type: AWSDynamodbSignerType
    at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:131)
    at com.amazonaws.auth.SignerFactory.getSignerByTypeAndService(SignerFactory.java:104)
    at com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:459)
    at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:430)
    at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:318)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.init(AmazonDynamoDBClient.java:392)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:382)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.<init>(AmazonDynamoDBClient.java:364)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder.build(AmazonDynamoDBClientBuilder.java:109)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder.build(AmazonDynamoDBClientBuilder.java:30)
    at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
    at io.airbyte.integrations.destination.dynamodb.DynamodbChecker.getAmazonDynamoDB(DynamodbChecker.java:76)
    at io.airbyte.integrations.destination.dynamodb.DynamodbChecker.attemptWriteAndDeleteDynamodbItem(DynamodbChecker.java:33)
    at io.airbyte.integrations.destination.dynamodb.DynamodbChecker.attemptDynamodbWriteAndDelete(DynamodbChecker.java:28)
    at io.airbyte.integrations.destination.dynamodb.DynamodbDestination.check(DynamodbDestination.java:33)
    at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:121)
    at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:97)
    at io.airbyte.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.java:87)
    at io.airbyte.integrations.destination.dynamodb.DynamodbDestinationRunner.main(DynamodbDestinationRunner.java:15)

2023-08-29 22:03:48 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - Completed integration: io.airbyte.integrations.destination.dynamodb.DynamodbDestination
2023-08-29 22:03:48 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):312 - Completed destination: io.airbyte.integrations.destination.dynamodb.DynamodbDestination
2023-08-29 22:03:48 INFO i.a.w.g.DefaultCheckConnectionWorker(run):117 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@11cf0d1b[status=failed,message=Could not connect to the DynamoDB table with the provided configuration. 
unknown signer type: AWSDynamodbSignerType,additionalProperties={}]
2023-08-29 22:03:48 INFO i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-08-29 22:03:48 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2023-08-29 22:03:48 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2023-08-29 22:03:48 INFO i.a.c.i.LineGobbler(voidCall):149 -

Contribute

octavia-squidington-iii commented 3 weeks ago

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

octavia-squidington-iii commented 5 days ago

This issue was closed because it has been inactive for 20 days since being marked as stale.