Error during WebSocket handshake: 'Upgrade' header is missing

ourfor commented 4 years ago

Kong 2.0.2

Server Config

{
"host": "api.dev.ourfor.top",
"created_at": 1584945910,
"connect_timeout": 60000,
"id": "46f07b22-d0e8-4f5e-a05f-eb3f57e4da54",
"protocol": "http",
"name": "api",
"read_timeout": 60000,
"port": 80,
"path": null,
"updated_at": 1584945910,
"retries": 5,
"write_timeout": 60000,
"tags": null,
"client_certificate": null
}

Route Config

{
"id": "de596b50-c3b9-48aa-98e2-5d3d7e245230",
"path_handling": "v0",
"paths": null,
"destinations": null,
"headers": null,
"protocols": [
    "http",
    "https"
],
"methods": null,
"snis": null,
"service": {
    "id": "46f07b22-d0e8-4f5e-a05f-eb3f57e4da54"
},
"name": null,
"strip_path": true,
"preserve_host": false,
"regex_priority": 0,
"updated_at": 1584945983,
"sources": null,
"hosts": [
    "demo.ourfor.top"
],
"https_redirect_status_code": 426,
"tags": null,
"created_at": 1584945983
}

Xnip2020-03-23_15-55-15

Xnip2020-03-23_15-56-23

http works ok, but can't connect websocket, what should I do?

bungle commented 4 years ago

I think it is your upstream that does not handle the request correctly. Kong will send: Connection: keep-alive, Upgrade, and your upstream websocket implementation just cannot parse that, aka it thinks there is no Upgrade in connection while there is? Am I correct? And Kong also sends Upgrade: websocket. We do have a test for this here:

https://github.com/Kong/kong/blob/master/spec/02-integration/05-proxy/03-upstream_headers_spec.lua#L166-L180

Can you check on your upstream what headers the request contains?

ourfor commented 4 years ago

I use tomcat and the log show, the onError execute

2020-03-23 08:00:15.046 ERROR 1 --- [nio-8080-exec-3] k.r.j.i.KClassImpl                       : LoginSocket: 出现异常
2020-03-23 08:00:15.046 ERROR 1 --- [nio-8080-exec-3] k.r.j.i.KClassImpl                       : java.io.EOFException

any good idea?

bungle commented 4 years ago

@ourfor,

If you change this line: https://github.com/Kong/kong/blob/2.0.2/kong/runloop/handler.lua#L1120

var.upstream_connection = "keep-alive, Upgrade"

TO

var.upstream_connection = "Upgrade"

Does it work then?

ourfor commented 4 years ago

I use docker image 😅

ourfor commented 4 years ago

build next branch source code and change

tomcat log

it seems like connection closed early

bungle commented 4 years ago

In docker image the files should be at:

/usr/local/share/lua/5.1/kong

But the issue is rather strange. Is there anything between:

client and kong?
kong and upstream?

ourfor commented 4 years ago

I change the file in docker image, got this error:

WebSocket connection to 'ws://demo.ourfor.top:8000/auth/wechat' failed: Error during WebSocket handshake: Unexpected response code: 404

tomcat log

23-Mar-2020 10:56:49.110 INFO [http-nio-8080-exec-8] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
    java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:567)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:502)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

before using kong, I had used apache and iis. work well with apache, same error with iis

ourfor commented 4 years ago

I use nodejs to test websocket.

server code

const Server = require('ws').Server
ws = new Server({port: 80})
ws.on('connection', ws => {
console.log('client connected')

ws.on('message', msg => {
    console.log(`client -> server: ${msg}`)
    ws.send('hello, I am server')
})
})

deploy: api.dev.ourfor.topTest

client code


const socket = new WebSocket('ws://api.dev.ourfor.top')

socket.onopen = () => { console.log('connected!') socket.send('hello, I am client.') }

socket.onmessage = ({data: msg}) => { console.log(server -> client: ${msg}) console.log(msg) }


before using kong, 
![image](https://user-images.githubusercontent.com/33711476/77316712-df503200-6d44-11ea-8f45-3ff5149d3559.png)

after using those config:
- `api service`:
```json
{
    "host": "api.dev.ourfor.top",
    "created_at": 1584966116,
    "connect_timeout": 60000,
    "id": "ffe00cc0-40ab-4109-843c-c9818efc0038",
    "protocol": "http",
    "name": "api",
    "read_timeout": 60000,
    "port": 80,
    "path": null,
    "updated_at": 1584966116,
    "retries": 5,
    "write_timeout": 60000,
    "tags": null,
    "client_certificate": null
}

route

{
"next": null,
"data": [
    {
        "id": "ad8eb330-6f5e-46fe-b8b6-38d7364b537e",
        "path_handling": "v0",
        "paths": null,
        "destinations": null,
        "headers": null,
        "protocols": [
            "http",
            "https"
        ],
        "methods": null,
        "snis": null,
        "service": {
            "id": "ffe00cc0-40ab-4109-843c-c9818efc0038"
        },
        "name": null,
        "strip_path": true,
        "preserve_host": false,
        "regex_priority": 0,
        "updated_at": 1584966133,
        "sources": null,
        "hosts": [
            "demo.ourfor.top"
        ],
        "https_redirect_status_code": 426,
        "tags": null,
        "created_at": 1584966133
    }
]
}

and client code change to:

const socket = new WebSocket('ws://demo.ourfor.top:8000')

socket.onopen = () => {
    console.log('connected!')
    socket.send('hello, I am client.')
}

socket.onmessage = ({data: msg}) => {
    console.log(`server -> client: ${msg}`)
    console.log(msg)
}

and test http

bungle commented 4 years ago

@ourfor I used my machine to proxy (using Kong 2.0.2) to: http://api.dev.ourfor.top/

proxying through kong

Using Chrome Version 80.0.3987.149 (Official Build) (64-bit). (also tested with Firefox and works too)

bungle commented 4 years ago

Here is the service I used:

{
    "client_certificate": null,
    "connect_timeout": 60000,
    "created_at": 1584974390,
    "host": "api.dev.ourfor.top",
    "id": "b6838ab6-a71e-48e2-aee2-b367a8be1eac",
    "name": null,
    "path": "/",
    "port": 80,
    "protocol": "http",
    "read_timeout": 60000,
    "retries": 5,
    "tags": null,
    "updated_at": 1584974390,
    "write_timeout": 60000
}

and here is the route:

{
    "created_at": 1584974415,
    "destinations": null,
    "headers": null,
    "hosts": null,
    "https_redirect_status_code": 426,
    "id": "c669e947-a9bc-45a3-8ab1-ea83703f0f34",
    "methods": null,
    "name": null,
    "path_handling": "v0",
    "paths": [
        "/"
    ],
    "preserve_host": false,
    "protocols": [
        "http",
        "https"
    ],
    "regex_priority": 0,
    "service": {
        "id": "b6838ab6-a71e-48e2-aee2-b367a8be1eac"
    },
    "snis": null,
    "sources": null,
    "strip_path": true,
    "tags": null,
    "updated_at": 1584974415
}

ourfor commented 4 years ago

the problem is that I have a static website and api service in one server. I use blog.ourfor.top to visit my static website and use api.ourfor.top to provide data. How should I configure?

api service contains http and websocket

bungle commented 4 years ago

Do you know why it works on my machine? I think it should work on yours as well. I don’t see why it won’t work with static website.

ourfor commented 4 years ago

set service path to /, it worked. thank you!

service:

curl -i -X POST \
--url http://localhost:8001/services \
--data "name=api" \
--data "url=http://api.dev.ourfor.top" \
--data "path=/"

route

curl -i -X POST \
--url http://localhost:8001/services/api/routes \
--data "paths[]=/" \
--data "hosts[]=demo.ourfor.top"

test nodejs

test tomcat

bungle commented 4 years ago

@ourfor, great! thanks for getting back. I am closing this as it turned out to be a small configuration issue.

ourfor commented 4 years ago

[UPDATE] It seems like can't work well with tomcat

service

{
"host": "api.dev.ourfor.top",
"created_at": 1585711143,
"connect_timeout": 60000,
"id": "ed72b4f9-8e89-4211-b86f-cec1a56841b5",
"protocol": "http",
"name": "api",
"read_timeout": 60000,
"port": 80,
"path": "/",
"updated_at": 1585711351,
"retries": 5,
"write_timeout": 60000,
"tags": null,
"client_certificate": null
}

route

{
"next": null,
"data": [
    {
        "id": "d1287f25-4e1e-4d77-926d-1f5b0928febd",
        "path_handling": "v0",
        "paths": [
            "/"
        ],
        "destinations": null,
        "headers": null,
        "protocols": [
            "http",
            "https"
        ],
        "methods": null,
        "snis": null,
        "service": {
            "id": "ed72b4f9-8e89-4211-b86f-cec1a56841b5"
        },
        "name": null,
        "strip_path": true,
        "preserve_host": false,
        "regex_priority": 0,
        "updated_at": 1585711151,
        "sources": null,
        "hosts": [
            "demo.ourfor.top"
        ],
        "https_redirect_status_code": 426,
        "tags": null,
        "created_at": 1585711151
    }
]
}

websocket


const socket1 = new WebSocket('ws://api.dev.ourfor.top/auth/wechat')
socket1.onopen = () => {
console.log('socket1 connected')
}

const socket2 = new WebSocket('ws://demo.ourfor.top:8000/auth/wechat') socket2.onopen = () => { console.log('socket2 connected') }



result is `WebSocket connection to 'ws://demo.ourfor.top:8000/auth/wechat' failed: Error during WebSocket handshake: 'Upgrade' header is missing` 😂

ourfor commented 4 years ago

follow https://github.com/Kong/kong/issues/5714#issuecomment-602500322, it worked

elvis824 commented 4 years ago

@bungle I also encountered the same issue with Tomcat. With Kong v2.0.3, websocket connections from both Chrome or Firefbox would be dropped by Tomcat 9.0.29 if it is behind Kong.

Following the changes you mentioned in https://github.com/Kong/kong/issues/5714#issuecomment-602500322, or downgrading Kong to v2.0.0, everything works again.

dwin commented 4 years ago

Also having this issue with Java Spring, looking into resolving with https://github.com/Kong/kong/issues/5714#issuecomment-602500322

zcpwillam commented 4 years ago

I got the same problem, the client parsed upgrade response, and when I checked the headers of the upgrade response, there was no key named "upgrade", which caused the following validation exception;

dwin commented 4 years ago

The Upgrade header is still there, but it is not in the order that these servers expect and I think this is an issue on the upstream side that it doesn't inspect more than one value for each header key.

This is the Dockerfile we use to fix it, we're not actually building Kong, just updating a line in the handler for this header.

FROM alpine:3.12 AS helper

WORKDIR /scratch

RUN apk add --no-cache sed
# copy handler file from Kong image to modify
COPY --from=kong:2.1.0-alpine /usr/local/share/lua/5.1/kong/runloop/handler.lua .
# Apply modification to Websocket handling https://github.com/Kong/kong/issues/5714#issuecomment-602514521
RUN sed -i 's/var.upstream_connection = "keep-alive, Upgrade"/var.upstream_connection = "Upgrade"/' handler.lua

FROM kong:2.1.0
# Copy modified file in
COPY --from=helper /scratch/handler.lua /usr/local/share/lua/5.1/kong/runloop/handler.lua
# no other changes

jasine commented 3 years ago

same issue

zonybob commented 3 years ago

same issue

YouYue123 commented 3 years ago

This handler.lua fix works for me. If the connection in header is "Keep-alive, Upgrade", socket.io won't handle it.

My env:

Backend: socket.io 3.0.5 on nodeJS 14 Frontend: socket.io-client 3.0.5

wayne-pq commented 3 years ago

@dwin 3q ，Solve my problem with tomcat9 and kong 2.2.1

missops commented 3 years ago

k8s内代理websocket一样的问题，版本：2.1.0

按@重新打下镜像解决。

The Upgrade header is still there, but it is not in the order that these servers expect and I think this is an issue on the upstream side that it doesn't inspect more than one value for each header key.

This is the Dockerfile we use to fix it, we're not actually building Kong, just updating a line in the handler for this header.
FROM alpine:3.12 AS helper

WORKDIR /scratch

RUN apk add --no-cache sed
# copy handler file from Kong image to modify
COPY --from=kong:2.1.0-alpine /usr/local/share/lua/5.1/kong/runloop/handler.lua .
# Apply modification to Websocket handling https://github.com/Kong/kong/issues/5714#issuecomment-602514521
RUN sed -i 's/var.upstream_connection = "keep-alive, Upgrade"/var.upstream_connection = "Upgrade"/' handler.lua

FROM kong:2.1.0
# Copy modified file in
COPY --from=helper /scratch/handler.lua /usr/local/share/lua/5.1/kong/runloop/handler.lua
# no other changes

k8s kong ingress 一样的问题，按这个重新打镜像后解决。版本：2.1.0

Kong / kong

Error during WebSocket handshake: 'Upgrade' header is missing #5714