Open RusabKhan opened 1 year ago
What happens when you try com.amazonaws.auth.EnvironmentVariableCredentialsProvider
instead?
.config('spark.hadoop.fs.s3a.aws.credentials.provider', 'com.amazonaws.auth.EnvironmentVariableCredentialsProvider')
With the above settings at first, it wasn't loading environment variables set through code
os.environ["AWS_ACCESS_KEY_ID"] = 'etc'
os.environ["AWS_SECRET_KEY"] = 'etc'
Then I set my variables through the export command in the terminal and retried and it resulted in the below error.
23/04/03 16:44:35 WARN AbstractS3ACommitterFactory: Using standard FileOutputCommitter to commit work. This is slow and potentially unsafe.
23/04/03 16:45:39 ERROR RedshiftWriter: Exception thrown during Redshift load; will roll back transaction
java.sql.SQLException: Exception thrown in awaitResult:
at io.github.spark_redshift_community.spark.redshift.JDBCWrapper.executeInterruptibly(RedshiftJDBCWrapper.scala:133)
at io.github.spark_redshift_community.spark.redshift.JDBCWrapper.executeInterruptibly(RedshiftJDBCWrapper.scala:109)
.......
Caused by: com.amazon.redshift.util.RedshiftException: ERROR: Query cancelled.
at com.amazon.redshift.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2601)
at com.amazon.redshift.core.v3.QueryExecutorImpl.processResultsOnThread(QueryExecutorImpl.java:2269)
The database is accessible and I can write to it via the psycopg2 library and even ran some copy commands after writing to s3 via boto3.
@RusabKhan I just turned on secrets scanning in the repository, and GitHub reported possibly secret leak when you initially open the issue. Please make sure you rotate your tokens if it's still active. Despite you edit it, GitHub basically has some logs of it.
Below is my code
This throws the exception
I have also tried using forward_s3_crdentials true. This role has full permission to redshift and s3. I have written multiple files and ran copy command manually via the psycopg2 library however with the spark it doesn't work.
Environment Spark 3.2.3 Scala 2.12 Pyspark 3.2.3 Java 11 Ubuntu