search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.12k stars 831 forks source link

[Bug] - Balanced-Kinesis plugin fails to start checkpointer #457

Closed kamaroyl closed 2 years ago

kamaroyl commented 4 years ago

Any matching open bugs: No

Benthos Version: 3.18.0 calling benthos -c path_to_my_yaml returns:

"Failed to connect to kinesis_balanced: Failed to start Checkpointer: SerializationError: failed to unmarshal response error\n\tstatus code: 403, request id: fe7b59d5-d0c4-a717-a91e-d159f61c25ac\ncaused by: UnmarshalError: failed decoding error message\n\t00000000 3c 49 6e 76 61 6c 69 64 53 69 67 6e 61 74 75 72 |<InvalidSignatur|\n00000010 65 45 78 63 65 70 74 69 6f 6e 3e 0a 20 20 3c 4d |eException>. <M|\n00000020 65 73 73 61 67 65 3e 43 72 65 64 65 6e 74 69 61 |essage>Credentia|\n00000030 6c 20 73 68 6f 75 6c 64 20 62 65 20 73 63 6f 70 |l should be scop|\n00000040 65 64 20 74 6f 20 63 6f 72 72 65 63 74 20 73 65 |ed to correct se|\n00000050 72 76 69 63 65 3a 20 27 6b 69 6e 65 73 69 73 27 |rvice: 'kinesis'|\n00000060 2e 20 3c 2f 4d 65 73 73 61 67 65 3e 0a 3c 2f 49 |. .</I|\n00000070 6e 76 61 6c 69 64 53 69 67 6e 61 74 75 72 65 45 |nvalidSignatureE|\n00000080 78 63 65 70 74 69 6f 6e 3e 0a |xception>.|\n\ncaused by: invalid character '<' looking for beginning of value"} after a few minutes.

This seems to be indicative of underlying aws library being used with the wrong scoping of the credentials (https://docs.aws.amazon.com/general/latest/gr/signature-v4-troubleshooting.html)

Config: balanced_kinesis_config_yaml.txt

kamaroyl commented 4 years ago

Removing endpoint from the yaml file got it working.

Jeffail commented 4 years ago

Hey @kamaroyl, are you using the endpoint field in order to connect to a third party service? If so are you able to share which one?

kamaroyl commented 4 years ago

@Jeffail using the endpoint to reach the kinesis/dynamo instance from inside a firewalled off network; it's an fqdn spun up by an internal team.

kamaroyl commented 4 years ago

Found that using the env variable that's exposed by gokini, I can get the application to use the url I want.

Jeffail commented 4 years ago

Interesting, thanks @kamaroyl that should help narrow it down, I suspect the setting isn't being propagated but I'll need to dig into the library a little bit.

Jeffail commented 3 years ago

Just to update, I'm actively working on this in the background in an effort to unify the kinesis and kinesis_balanced inputs. I'm hoping to simplify the balancing logic considerably and that should fix oddities like this which I'm assuming is due to session collision.

kamaroyl commented 3 years ago

Fantastic news, thanks @Jeffail

Jeffail commented 3 years ago

Hey @kamaroyl, in the latest release there's a general purpose aws_kinesis input that is intended to replace kinesis_balanced.

This component is a total rewrite, no longer relying on a third party library, and both the dynamodb and kinesis components share a credentials session. The mechanism for checkpointing is also simplified quite a bit so I'm hoping that not only should we see fewer issues but they should be easier to diagnose as well.

The table format used by this component should be compatible with the previous input so you can drop it in without any manual intervention, but I'd recommend attempting a migration with a test pipeline first, and make a back up of your table if you're going to try this new input out.

patrobinson commented 3 years ago

Endpoint isn't something I realised we had to support in kinesis balanced, I'd be happy to add it but the component looks to be deprecated now.

@Jeffail I'd love to merge your changes back upstream, but they're quite considerable. I've spent a lot of time improving this library and would love to make any improvements available to others.