Open KissBalazs opened 1 year ago
I have discovered that the problem is that the sink actually trying to send data to AWS instead of the local kinesalite instance!( I have provided my current credentials, and it sent data to the stream I have created on AWS....) So it seems that the AWS URL configuration is not working for some reason.
I've just encountered this exact scenario - Did you get anywhere with this?
@Bunky after further looking into the docs, I realized the the current AWS Flink Kinesis connector version is buggy, and won't send the data to the expected outputstreams. If I upgrade the version it worked as expected, but that way I was unable to build and deploy in "per-job" mode in AWS, so I scrapped this approach, and now I write the results to STDOUT instead of sending them back to Flink to an output stream. My whole aim with this was to create a test environment for our AWS Kinesis app, and this approach suffice for me. See also my same question in here: https://stackoverflow.com/questions/73801996/apache-flink-kineisstreamsink-pkix-path-building-failed
I've sort of found a fix for this, however haven't had the chance to try it specifically for Kinesalite. I ended up changing to use kinesis-mock for use with docker, although I'd imagine similar steps can be made with Kinesalite.
Instead of using
outputProperties.setProperty(ConsumerConfigConstants.AWS_ENDPOINT, "http://localhost:4568");
in the producer, I swapped it out for
outputProperties.put("KinesisEndpoint", "localhost");
outputProperties.put("KinesisPort", "4567");
This seemed to then point correctly to my local Kinesis, only throwing an SSL connect error, which was resolved by pointing the producer to an SSL enabled port, which kinesis-mock provides out of the box (which is why in the above examples the port is different, there may just be a way to disable the SSL requirement, but I didn't look into it).
And for sake of clarity I have this working with Apache Flink 1.15.2
I have the following Flink application, running in a docker container, connected to a Kinesalite server:
My problem is that this code executes properly, and sets up the connections. I am able to read from the input stream, but it fails to write to the output stream: upon reaching the sink, the application gets stuck with the following error:
What can be the problem here? As you can see the two configuration is nearly identical. :\