Hi, I am running spark streaming application on Kubernetes, where-in service-account has the aws IAM role to connect to aws services. The app is reading from kinesis and writing to kinesis stream.
But while running the app is not able to connect to aws with any of the configs below:
"spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
"spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
"spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.AWSSessionCredentialsProvider"
Tried both configs awsUseInstanceProfile true and false, with no luck.
The spark app is running fine my local system with DefaultAWSCredentialsProviderChain while writing to console, but having issues while running on Kubernetes. Could anyone please help on this?
Hi, I am running spark streaming application on Kubernetes, where-in service-account has the aws IAM role to connect to aws services. The app is reading from kinesis and writing to kinesis stream.
But while running the app is not able to connect to aws with any of the configs below: "spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider" "spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.DefaultAWSCredentialsProviderChain" "spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.AWSSessionCredentialsProvider"
Tried both configs awsUseInstanceProfile true and false, with no luck.
The spark app is running fine my local system with DefaultAWSCredentialsProviderChain while writing to console, but having issues while running on Kubernetes. Could anyone please help on this?