'Unkown TCP Option' from AWS AMI

blakelead commented 3 years ago

Traffic coming from pods inside one of our Kubernetes cluster have the following TCP frame:

Transmission Control Protocol, Src Port: 40959, Dst Port: 443, Seq: 0, Len: 0
    Source Port: 40959
    Destination Port: 443
    [Stream index: 275]
    [TCP Segment Len: 0]
    Sequence number: 0    (relative sequence number)
    Sequence number (raw): 742718731
    [Next sequence number: 1    (relative sequence number)]
    Acknowledgment number: 0
    Acknowledgment number (raw): 0
    1101 .... = Header Length: 52 bytes (13)
    Flags: 0x002 (SYN)
    Window size value: 62727
    [Calculated window size: 62727]
    Checksum: 0x8a7e [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    Options: (32 bytes), Maximum segment size, SACK permitted, Timestamps, No-Operation (NOP), Window scale, Unknown (0x1f)
        TCP Option - Maximum segment size: 8961 bytes
            Kind: Maximum Segment Size (2)
            Length: 4
            MSS Value: 8961
        TCP Option - SACK permitted
            Kind: SACK Permitted (4)
            Length: 2
        TCP Option - Timestamps: TSval 3376024531, TSecr 0
            Kind: Time Stamp Option (8)
            Length: 10
            Timestamp value: 3376024531
            Timestamp echo reply: 0
        TCP Option - No-Operation (NOP)
            Kind: No-Operation (1)
        TCP Option - Window scale: 7 (multiply by 128)
            Kind: Window Scale (3)
            Length: 3
            Shift count: 7
            [Multiplier: 128]
        TCP Option - Unknown
            Kind: Unknown (31)
            Length: 12
            Payload: 000011000000c0ba778b

The last TCP option is causing our on-premises Firewall receiving traffic from the pods to block the request containing this unknown (31) option.

This option is reserved by IANA's spec and should not be used: https://www.iana.org/assignments/tcp-parameters/tcp-parameters.xhtml#tcp-parameters-1

I tried multiple AMIs: ami-04b959013b0e014ca, ami-038341f2c72928ada, ami-00056da53ebd12c2e, ami-0bd4849feacfbb8a2 and I still get the same unknown option.

So my questions are:

What is the use for this option?
Are the AMIs responsible for it?
If so how can I get rid of it?

Environment:

AWS Region: eu-west-1
Instance Type(s): m5.large
EKS Platform version : eks.1
Kubernetes version : 1.19
AMI Version: ami-04b959013b0e014ca
Kernel : Linux ip-10-43-94-102.eu-west-1.compute.internal 5.4.95-42.163.amzn2.x86_64 #1 SMP Thu Feb 4 12:50:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Release information :

BASE_AMI_ID="ami-0a1ccc021b9016ec9"
BUILD_TIME="Wed Mar 10 19:47:23 UTC 2021"
BUILD_KERNEL="5.4.95-42.163.amzn2.x86_64"
ARCH="x86_64"

cartermckinnon commented 1 year ago

Are the AMIs responsible for it?

We don't have any reason to think this is related to the AMI; but let us know if this is still an issue on the latest AMI's or if you root-caused it.

kanabeat commented 9 months ago

Hi @blakelead,

So my questions are:

What is the use for this option?

Are the AMIs responsible for it?

If so how can I get rid of it?

Did you finally find which component was causing the unknown TCP option to be added?

I have the same problem with an on premises cluster here.

blakelead commented 9 months ago

@kanabeat unfortunately no.

kanabeat commented 9 months ago

After digging deeper into this, it appears that the TCP option is added by a security component in charge of threat detection/workload protection.

awslabs / amazon-eks-ami

'Unkown TCP Option' from AWS AMI #658