Protocol error in kinesis putRecords in aws sdk > 3.81.0

[cesarfd]

Describe the bug

This is a reopening of #3602 as the bug is definitely not fixed as of v3.121.0

Expected Behavior

putRecords() should work correctly.

Current Behavior

Connection fails, error msg gives "Protocol error".

Reproduction Steps

Please refer to https://github.com/aws/aws-sdk-js-v3/issues/3602#issuecomment-1134973121

Possible Solution

No response

Additional Information/Context

No response

SDK version used

3.82.0 or later

Environment details (OS name and version, etc.)

Please refer to https://github.com/aws/aws-sdk-js-v3/issues/3602#issuecomment-1134973121

[gugu]

I confirm that it is not fixed in 3.128.0 Here is what I have in Sentry (not much)

It can be kinesis problem and not SDK, related: https://github.com/awslabs/amazon-kinesis-client/issues/711

[gugu]

@cesarfd let's try to debug. Do you use some other other AWS service, which does not support http2, for example DynamoDB? This error happens if someone uses http2 protocol for the endpoint, which does not support http2. And my guess aws sdk accidentally upgrades protocol version of the other service, and it fails

[cesarfd]

I can confirm it fails with and without using a custom VPC endpoint for Kinesis Streams (I tried it just in case it made a difference).

One thing that I don't understand is why is this not failing to everyone? AFAIK there should be nothing special that has to be configured in the client side or in the AWS console to use http/2 with any of the services, right?

[gugu]

Here is my workaround:

    const usKinesis = new KinesisClient({ region: "us-east-1", requestHandler: new NodeHttpHandler() });

[gugu]

My guess it is because we are the only people who use both http2-enabled AWS kinesis and http1-only AWS services together

[cesarfd]

Here is my workaround:

    const usKinesis = new KinesisClient({ region: "us-east-1", requestHandler: new NodeHttpHandler() });

I can confirm the workaround works. At least we can force it to use classic http 1.1

[RanVaknin]

@gugu Thanks for posting the workaround.

I added a p2 tag to it. Since it has a workaround other more pressing issues will take precedence, but this will be added to the team's backlog.

Thanks again, Ran

[300LiterPropofol]

NodeHttpHandler https://github.com/aws/aws-sdk-js-v3/issues/3780#issuecomment-1184381160 Hi @gugu, thank you for the post. I have one dumb question, where does the new NodeHttpHandler() come from? What require() do I need to import to use this handler or is it built-in in Nodejs? Thank you!

[dorrogeray]

@300LiterPropofol

import {NodeHttpHandler} from '@aws-sdk/node-http-handler'

[jomaora]

I have the same error on v3.405.0, with NodeJS v18.17.0

const document = // DATA TO SEND

const kinesis = new KinesisClient({
  endpoint: 'http://localhost:4567',
  credentials: {
    secretAccessKey: 'local',
    accessKeyId: 'local'
  },
  region: 'eu-west-1'
});

const command = new PutRecordCommand({
        StreamName: 'streamName',
        Data: Buffer.from(JSON.stringify(document)),
        PartitionKey: document._id
});

const result = await kinesis.send(command);

  Error (NghttpError) {
    $metadata: {
      attempts: 1,
      totalRetryDelay: 0,
    },
    code: 'ERR_HTTP2_ERROR',
    errno: -505,
    message: 'Protocol error',
  }

[a8t]

^ Me too, sdk 3.405.0 and node 20.6.0

[godu]

I have the same error on v3.405.0, with NodeJS v18.17.0

const document = // DATA TO SEND

const kinesis = new KinesisClient({
  endpoint: 'http://localhost:4567',
  credentials: {
    secretAccessKey: 'local',
    accessKeyId: 'local'
  },
  region: 'eu-west-1'
});

const command = new PutRecordCommand({
        StreamName: 'streamName',
        Data: Buffer.from(JSON.stringify(document)),
        PartitionKey: document._id
});

const result = await kinesis.send(command);

  Error (NghttpError) {
    $metadata: {
      attempts: 1,
      totalRetryDelay: 0,
    },
    code: 'ERR_HTTP2_ERROR',
    errno: -505,
    message: 'Protocol error',
  }

We found a workaround. It was because we used mhart/kinesalite to simulate Kinesis locally. Unfortunately, kinesalite doesn't support HTTP2 connect. We had to configure kinesis client to use HTTP1.1 instead.

import {NodeHttpHandler} from '@smithy/node-http-handler';
import {KinesisClient} from '@aws-sdk/client-kinesis';

const kinesis = new KinesisClient(
  KINESIS_ENDPOINT
    ? {
        endpoint: KINESIS_ENDPOINT,
        requestHandler: new NodeHttpHandler()
      }
    : {}
);

[a8t]

Sorry, i neglected to say that the same workaround worked for us:

import { KinesisClient } from '@aws-sdk/client-kinesis';
import { NodeHttpHandler } from '@smithy/node-http-handler';

const region = 'us-east-2';
const kinesisClient = new KinesisClient({
  region,
  requestHandler: new NodeHttpHandler(),
  maxAttempts: 5,
});

export default kinesisClient;

This workaround was mentioned earlier in the thread, but since then, AWS has moved some libraries under @smithy.

[cesarfd]

Still failing with @smithy/node-http-handler 2.1.10 and @aws-sdk/client-kinesis 3.465.0.

It only works when explicitly instantiating a NodeHttpHandler to the client.

[RanVaknin]

Hi there,

I tried to reproduce this with the latest version, but I'm not hitting this error. Kinesis supports both http2 and http1.1, and the default httphandler for this service uses http2 (likely to support streaming). The workaround suggested here is setting the request handler to the default protocol which 1.1. This is a totally viable workaround as the service supports both.

If anyone here can come up with a way to force this error we can take another look, but as it stands it is not reproducible.

Thanks, Ran~

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.