Is there a way to specify the "region" to be used in the credential signing request (specifically in the Authorization Header)? My use case would benefit from using a custom Kinesis endpoint as it will allow direct connect between the VPC the Spark Kinesis job will be running in and the VPC that the kinesis stream will be consumed from.
Using the "standard" AWS kinesis endpoint for us-east-1 works without issue, but using it means spark must first go out to the internet and then authenticate and consume the stream rather than just using direct-connect.
The direct-connect/custom endopointURL I am using is https://kinesis-ae1.hdw.r53.deap.tv and from debug logs I can see that the credential portion of the Authorization header being sent is using the ae1 portion of the URL.
I understand that this might have something to do with the implementation of the Kinesis Client Library, but wondering if there is any way to override this header within the spark application code itself?
For reference, here is the relevant readStream block from the PySpark app. Notice the region is explicitly specified:
Just to reiterate. Setting the endpointUrloption to https://kinesis.us-east-1.amazonaws.com allows the application to work without issues, but this is not ideal from a performance standpoint.
Hello,
Is there a way to specify the "region" to be used in the credential signing request (specifically in the Authorization Header)? My use case would benefit from using a custom Kinesis endpoint as it will allow direct connect between the VPC the Spark Kinesis job will be running in and the VPC that the kinesis stream will be consumed from.
Using the "standard" AWS kinesis endpoint for
us-east-1
works without issue, but using it means spark must first go out to the internet and then authenticate and consume the stream rather than just using direct-connect.The direct-connect/custom endopointURL I am using is
https://kinesis-ae1.hdw.r53.deap.tv
and from debug logs I can see that thecredential
portion of the Authorization header being sent is using theae1
portion of the URL.Full log:
Relevant line:
I understand that this might have something to do with the implementation of the Kinesis Client Library, but wondering if there is any way to override this header within the spark application code itself?
For reference, here is the relevant
readStream
block from the PySpark app. Notice the region is explicitly specified:Just to reiterate. Setting the
endpointUrl
option tohttps://kinesis.us-east-1.amazonaws.com
allows the application to work without issues, but this is not ideal from a performance standpoint.Thanks for any insight!