Open taylorcressy opened 4 years ago
I am running into the same problem. Looks like the ddb client is private so it doesnt look like we can pass a config to it.
Also ddb local theres no easy way to use it with ssl. One way is to setup a reverse proxy to terminate the ssl.
Has anyone found a workaround or solution yet?
@christopherhudy I originally did what you said and setup a proxy. This proved to be way too much effort.
So instead I rolled my own implementation of
ddbConf.set("mapred.input.format.class", "org.apache.hadoop.dynamodb.read.DynamoDBInputFormat") ddbConf.set("mapred.output.format.class", "org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat")
Which did the trick.
Yea i ended up ditching ddb local and mocked for unit testing. Then created actual integration tests with ddb. Thanks @taylorcressy
Hi there,
We are building out a spark structured streaming job and are attempting to run build-level integration tests with an embedded DynamoDB instance.
The problem we are running into is that it appears the connector only connects to Dynamo with HTTPS. And the embedded dynamo only accepts HTTP.
Is there a way to toggle off SSL for the connector?
Example code for setting up embedded Dynamo
Our job conf is:
This leads to the following exception: