Open SravaniMaddala opened 10 months ago
I am trying to use this connector with Spark 3.3 and EMR 6.11 but the RDD is always empty. Wondering if there is any version mismatch that is causing this or if I am missing something.
SparkSession sparkSession = SparkSession.builder().getOrCreate(); SparkContext sparkContext = sparkSession.sparkContext(); JavaSparkContext sc = new JavaSparkContext(sparkContext) JobConf jobConf = new JobConf(sc.hadoopConfiguration()); jobConf.set("dynamodb.input.tableName", "test-dynamo"); jobConf.set("mapred.output.format.class", "org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat"); jobConf.set("mapred.input.format.class", "org.apache.hadoop.dynamodb.read.DynamoDBInputFormat"); JavaPairRDD<Text, DynamoDBItemWritable> rows = sc.hadoopRDD(jobConf, DynamoDBInputFormat.class,Text.class, DynamoDBItemWritable.class); rows.count();
The unreleased master does seem to work with 3.3. I had the same experience with latest 4.16 release not working against 3.3.
I am trying to use this connector with Spark 3.3 and EMR 6.11 but the RDD is always empty. Wondering if there is any version mismatch that is causing this or if I am missing something.