When i tried to read data from redshift using iamrole, for some reason it isn't picking up the iamrole but instead using root.
Why is that happening?
My code:
val df: DataFrame = spark.read
.format("io.github.spark_redshift_community.spark.redshift")
.option("url", s"jdbc:redshift://redshifthostname:5439/ssl=true&?user=$dbUser&password=$dbPass")
.option("query", " select count(*) as cnt from tblname")
.option("tempdir", "s3dir")
.option("aws_iam_role", "iamrole")
.load()
Error
Exception in thread "main" java.sql.SQLException: FATAL: password authentication failed for user "root"
at com.amazon.redshift.util.RedshiftException.getSQLException(RedshiftException.java:56)
at com.amazon.redshift.Driver.connect(Driver.java:329)
at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
at io.github.spark_redshift_community.spark.redshift.JDBCWrapper.getConnector(RedshiftJDBCWrapper.scala:232)
at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.$anonfun$schema$1(RedshiftRelation.scala:62)
at scala.Option.getOrElse(Option.scala:189)
at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.schema$lzycompute(RedshiftRelation.scala:59)
at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.schema(RedshiftRelation.scala:58)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:438)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:297)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:286)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:286)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:221)
at com.rand.spark.LocalPi$.main(LocalPi.scala:76)
at com.rand.spark.LocalPi.main(LocalPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
When i tried to read data from redshift using iamrole, for some reason it isn't picking up the iamrole but instead using root.
Why is that happening?
My code: val df: DataFrame = spark.read .format("io.github.spark_redshift_community.spark.redshift") .option("url", s"jdbc:redshift://redshifthostname:5439/ssl=true&?user=$dbUser&password=$dbPass")
.option("query", " select count(*) as cnt from tblname")
.option("tempdir", "s3dir")
.option("aws_iam_role", "iamrole")
.load()
Error Exception in thread "main" java.sql.SQLException: FATAL: password authentication failed for user "root" at com.amazon.redshift.util.RedshiftException.getSQLException(RedshiftException.java:56) at com.amazon.redshift.Driver.connect(Driver.java:329) at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45) at io.github.spark_redshift_community.spark.redshift.JDBCWrapper.getConnector(RedshiftJDBCWrapper.scala:232) at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.$anonfun$schema$1(RedshiftRelation.scala:62) at scala.Option.getOrElse(Option.scala:189) at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.schema$lzycompute(RedshiftRelation.scala:59) at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.schema(RedshiftRelation.scala:58) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:438) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:297) at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:286) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:286) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:221) at com.rand.spark.LocalPi$.main(LocalPi.scala:76) at com.rand.spark.LocalPi.main(LocalPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
pom.xml dependencies: