search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
15.83k stars 2.96k forks source link

Daemon panic: concurrent write to websocket connection #6197

Closed Kycklingar closed 4 years ago

Kycklingar commented 5 years ago

Version information:

go-ipfs version: 0.4.20-rc1-495f12c02 Repo version: 7 System version: amd64/linux Golang version: go1.12.2

Type:

panic

Description:

Daemon is ready
panic: concurrent write to websocket connection

goroutine 77927234 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc005d79740, 0xc0112c2101, 0x0, 0x0, 0x0, 0xc00264cd10, 0xc00264cd10)
    /home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x77a
github.com/gorilla/websocket.(*messageWriter).Close(0xc005d79740, 0x155ea80, 0xc00264cd28)
    /home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x56
github.com/gorilla/websocket.(*Conn).prepWrite(0xc002691e40, 0x2, 0x10000c00ae47800, 0x0)
    /home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x20d
github.com/gorilla/websocket.(*Conn).NextWriter(0xc002691e40, 0x2, 0x8f30e9, 0xc00025e350, 0x10, 0x92658d)
    /home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x39
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc002691e40, 0x2, 0xc0045b7000, 0x14, 0x1000, 0x10, 0xc012dfce00)
    /home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x73
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc005d79530, 0xc0045b7000, 0x14, 0x1000, 0xc00264ceb8, 0x8f3978, 0x10)
    /home/kyck/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.0.2/conn.go:74 +0x59
bufio.(*Writer).Flush(0xc016ea16c0, 0xc016ea16c0, 0xc011bbef60)
    /usr/local/go/src/bufio/bufio.go:590 +0x7b
github.com/multiformats/go-multistream.delimWriteBuffered(0x7feef2b2b208, 0xc005d79680, 0xc011bbef60, 0x12, 0x12, 0x1, 0x0)
    /home/kyck/go/pkg/mod/github.com/multiformats/go-multistream@v0.0.1/multistream.go:64 +0xb9
github.com/multiformats/go-multistream.handshake.func1(0xc0072fdc20, 0x7feef2b2b150, 0xc005d79680)
    /home/kyck/go/pkg/mod/github.com/multiformats/go-multistream@v0.0.1/client.go:49 +0xab
created by github.com/multiformats/go-multistream.handshake
    /home/kyck/go/pkg/mod/github.com/multiformats/go-multistream@v0.0.1/client.go:48 +0x84
Stebalien commented 5 years ago

Dammit! I thought that was fixed.

Stebalien commented 5 years ago

Are you seeing any other stack traces? I'm having trouble reproducing this.

I'm pretty sure this isn't a concurrent write/close issue. It looks like a concurrent write issue when negotiating the security transport but I'm not sure how this is possible.

inetic commented 5 years ago

For what it's worth, we're seeing this. But that's IPFS v0.4.19 and on Android.

Stebalien commented 5 years ago

Thanks. Unfortunately, I'm still going to need to do some work to figure out what's actually causing this.

Kycklingar commented 5 years ago

Are you seeing any other stack traces? I'm having trouble reproducing this.

Nope, that's all I got. It's only ever happened twice, both on the same day.

songjiayang commented 5 years ago

@Kycklingar Same with me, no panic now.

Stebalien commented 5 years ago

Interesting... I wonder if this can only be triggered by a non-ipfs websocket client (someone may have been scanning).

songjiayang commented 5 years ago

@Stebalien seem look like it , but ipfs node should not panic.

Stebalien commented 5 years ago

I agree. I'm just trying to reproduce it so I can fix it.

Stebalien commented 5 years ago

@inetic can you reproduce this reliably? Could you try building https://github.com/ipfs/go-ipfs/pull/6263?

hsn10 commented 4 years ago

I have this crash daily, running with --enable-pubsub-experiment --enable-namesys-pubsub --enable-mplex-experiment

go-ipfs version: 0.4.22-
panic: concurrent write to websocket connection

goroutine 42561526 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc00dad2540, 0x1, 0x0, 0x0, 0x0, 0x1, 0xc00c5fb900)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x781
github.com/gorilla/websocket.(*messageWriter).Close(0xc00dad2540, 0x11f8649, 0x3)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x5d
github.com/gorilla/websocket.(*Conn).prepWrite(0xc00740adc0, 0x2, 0x8aa4bd8, 0x3f)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x214
github.com/gorilla/websocket.(*Conn).NextWriter(0xc00740adc0, 0x2, 0xc01d5dafc0, 0x8aa4bd8, 0x0, 0x1000000014e1fc0)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x40
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc00740adc0, 0x2, 0xc01d5dafc0, 0x20, 0x40, 0x10, 0xc018931de0)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x7a
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc00dd0ff50, 0xc01d5dafc0, 0x20, 0x40, 0x10946c0, 0x11d1be0, 0x2ab4620)
        pkg/mod/github.com/libp2p/go-ws-transport@v0.0.4/conn.go:74 +0x60
bytes.(*Buffer).WriteTo(0xc00db3a6c0, 0x79393c8, 0xc00dad2450, 0x2ab4620, 0xc00db3a6c0, 0x8ba401)
        /usr/lib/go/src/bytes/buffer.go:242 +0xbf
io.copyBuffer(0x79393c8, 0xc00dad2450, 0x14c4920, 0xc00db3a6c0, 0x0, 0x0, 0x0, 0xc018bd7370, 0x438d5d, 0x1382308)
        /usr/lib/go/src/io/io.go:384 +0x355
io.Copy(...)
        /usr/lib/go/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x1200771, 0xa, 0x7d5a580, 0xc00dad2450, 0xc00ec41d40)
        pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:27 +0x16b
created by github.com/multiformats/go-multistream.SelectProtoOrFail
        pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:23 +0x9c
MatthewSteeples commented 4 years ago

I also see this crash just running ipfs daemon --enable-pubsub-experiment --enable-namesys-pubsub

go-ipfs version: 0.4.22- Repo version: 7 System version: arm/linux Golang version: go1.12.7

Running on a Raspberry Pi 3B+

Stebalien commented 4 years ago

@MatthewSteeples could you try running go-ipfs master? We believe this may be fixed.

bonedaddy commented 4 years ago

This issue is also happening with general usage of libp2p websockets. I'm getting this issue outside of go-ipfs as well. https://github.com/libp2p/go-ws-transport/issues/56 is my bug report on go-ws-transport

This is a bug within go-ws-transport and makes websockets completely unusable. I have a custom node that's crashing every 30 minutes due to bug.

OP opened report using go-ws-transport v0.0.2 and it's still present in the latest master branch

Stebalien commented 4 years ago

Damn. Our gateways were "fixed" by updating but apparently we need to figure out where the issue actually is.

ander7agar commented 4 years ago

I have the same error. It happened for the first time last Sunday, it had version 0.4.21. I updated to the latest version 0.4.22, and so far it has continued to happen 2 to 3 times per day.

Mithgol commented 4 years ago

It starts happening more often, almost every hour.

KireinaHoro commented 4 years ago

Confirms for 0.4.22 on Windows 10, installed from chocolatey. Encountered that over 4 times in a few hours...

Daemon is started with no other arguments. Seems like it should happen more frequently if the node's serving data.

pstroinski commented 4 years ago

Same here, happens once or twice a day, ipfs 0.4.22 on ubuntu 18.04: panic: concurrent write to websocket connection goroutine 36495687 [running]: github.com/gorilla/websocket.(messageWriter).flushFrame(0xbc615c0, 0x160d2801, 0x0, 0x0, 0x0, 0x8f06390, 0x0) pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x6e4 github.com/gorilla/websocket.(messageWriter).Close(0xbc615c0, 0x17ffdc2c, 0x844390f) pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x4c github.com/gorilla/websocket.(Conn).prepWrite(0x21241380, 0x2, 0x0, 0x77) pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x1aa github.com/gorilla/websocket.(Conn).NextWriter(0x21241380, 0x2, 0x8f06390, 0x2f, 0x0, 0x1000b01) pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x2d github.com/gorilla/websocket.(Conn).WriteMessage(0x21241380, 0x2, 0x1a765dc0, 0x22, 0x40, 0xa026aec, 0xe740c918) pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x4f github.com/libp2p/go-ws-transport.(Conn).Write(0xbc612a0, 0x1a765dc0, 0x22, 0x40, 0x8b0d7c0, 0x8c10080, 0xe740c918) pkg/mod/github.com/libp2p/go-ws-transport@v0.0.4/conn.go:74 +0x42 bytes.(*Buffer).WriteTo(0xbc8d520, 0xe7456828, 0xbc61500, 0xe740c918, 0xbc8d520, 0xe7456801, 0x1) /usr/lib/go/src/bytes/buffer.go:242 +0x9e io.copyBuffer(0xe7456828, 0xbc61500, 0x8f05fd0, 0xbc8d520, 0x0, 0x0, 0x0, 0xe71ceb0, 0xe71ceb8, 0x80a36f9, ...) /usr/lib/go/src/io/io.go:384 +0x2d6 io.Copy(...) /usr/lib/go/src/io/io.go:364 github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x8c318ae, 0xc, 0xe74567a8, 0xbc61500, 0xbf981c0) pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:27 +0x136 created by github.com/multiformats/go-multistream.SelectProtoOrFail pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:23 +0x71

auronsan commented 4 years ago

same with me too xD . I thought only me got this ipfs 0.4.22

ipfs_1        | Changing user to ipfs
ipfs_1        | ipfs version 0.4.22
ipfs_1        | Found IPFS fs-repo at /data/ipfs
ipfs_1        | Initializing daemon...
ipfs_1        | go-ipfs version: 0.4.22-
ipfs_1        | Repo version: 7
ipfs_1        | System version: amd64/linux
ipfs_1        | Golang version: go1.12.7
ipfs_1        | Swarm listening on /ip4/127.0.0.1/tcp/4001
ipfs_1        | Swarm listening on /ip4/172.25.0.2/tcp/4001
ipfs_1        | Swarm listening on /p2p-circuit
ipfs_1        | Swarm announcing /ip4/127.0.0.1/tcp/4001
ipfs_1        | Swarm announcing /ip4/172.25.0.2/tcp/4001
ipfs_1        | API server listening on /ip4/0.0.0.0/tcp/5001
ipfs_1        | WebUI: http://0.0.0.0:5001/webui
ipfs_1        | Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
ipfs_1        | Daemon is ready
ipfs_1        | panic: concurrent write to websocket connection
ipfs_1        | 
ipfs_1        | goroutine 4704739 [running]:
ipfs_1        | github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc008b83a10, 0xc007486c01, 0x0, 0x0, 0x0, 0x1, 0xc007486cf0)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x77a
ipfs_1        | github.com/gorilla/websocket.(*messageWriter).Close(0xc008b83a10, 0x1, 0x1)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x56
ipfs_1        | github.com/gorilla/websocket.(*Conn).prepWrite(0xc0046ae6e0, 0x2, 0xc007486d58, 0x982cc5)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x20d
ipfs_1        | github.com/gorilla/websocket.(*Conn).NextWriter(0xc0046ae6e0, 0x2, 0x1, 0xc007486e10, 0x97af39, 0x100000000000008)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x39
ipfs_1        | github.com/gorilla/websocket.(*Conn).WriteMessage(0xc0046ae6e0, 0x2, 0xc00d99cdc0, 0x22, 0x40, 0x10, 0xc00e387b20)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x73
ipfs_1        | github.com/libp2p/go-ws-transport.(*Conn).Write(0xc008b83680, 0xc00d99cdc0, 0x22, 0x40, 0x163dfa0, 0x1795ea0, 0x7f8f66e2b380)
ipfs_1        |     /go/pkg/mod/github.com/libp2p/go-ws-transport@v0.0.4/conn.go:74 +0x59
ipfs_1        | bytes.(*Buffer).WriteTo(0xc008696a20, 0x7f8f66e3a9a8, 0xc008b83980, 0x7f8f66e2b380, 0xc008696a20, 0xe25e01)
ipfs_1        |     /usr/local/go/src/bytes/buffer.go:242 +0xb8
ipfs_1        | io.copyBuffer(0x7f8f66e3a9a8, 0xc008b83980, 0x1a98d00, 0xc008696a20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x9a59d6)
ipfs_1        |     /usr/local/go/src/io/io.go:384 +0x34e
ipfs_1        | io.Copy(...)
ipfs_1        |     /usr/local/go/src/io/io.go:364
ipfs_1        | github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x17c705d, 0xc, 0x7f8f66e3a8f0, 0xc008b83980, 0xc0011b3260)
ipfs_1        |     /go/pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:27 +0x164
ipfs_1        | created by github.com/multiformats/go-multistream.SelectProtoOrFail
ipfs_1        |     /go/pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:23 +0x95

I think this is reference from gorilla websocket https://github.com/googollee/go-socket.io/issues/182

jaller94 commented 4 years ago

Version information:

ipfs version 0.4.22 from the Manjaro Linux repo.

go-ipfs version: 0.4.22-4e981576b-dirty Repo version: 7 System version: amd64/linux Golang version: go1.12.8

Description:

  1. I ran ipfs daemon.
  2. I ran ipfs mount.
  3. Within 2 hours, the IPFS daemon crashed. This left the mount points in the status "/ipns: Transport endpoint is not connected".

Stacktrace:

panic: concurrent write to websocket connection

goroutine 2156447 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc001605530, 0x1, 0x0, 0x0, 0x0, 0xc0001325ee, 0xc00732ccf0)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:591 +0x77c
github.com/gorilla/websocket.(*messageWriter).Close(0xc001605530, 0x558fde865d9a, 0x10)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:709 +0x58
github.com/gorilla/websocket.(*Conn).prepWrite(0xc001c86dc0, 0x2, 0x0, 0xc00732cd00)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:459 +0x20f
github.com/gorilla/websocket.(*Conn).NextWriter(0xc001c86dc0, 0x2, 0x7f3828b036f8, 0xc00ab750e0, 0xc00ab750e0, 0x1007f3828b036f8)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:494 +0x3b
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc001c86dc0, 0x2, 0xc02075d500, 0x22, 0x40, 0x10, 0xc002116500)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:755 +0x75
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc0016052f0, 0xc02075d500, 0x22, 0x40, 0x558fdfb012c0, 0x558fdfc59200, 0x7f3828ac68c0)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/libp2p/go-ws-transport/conn.go:74 +0x5b
bytes.(*Buffer).WriteTo(0xc005c99800, 0x7f38280f0460, 0xc0016054a0, 0x7f3828ac68c0, 0xc005c99800, 0x558fded14b01)
    /usr/lib/go/src/bytes/buffer.go:242 +0xba
io.copyBuffer(0x7f38280f0460, 0xc0016054a0, 0x558fdfca2a20, 0xc005c99800, 0x0, 0x0, 0x0, 0x558fdee010ce, 0x2, 0x2)
    /usr/lib/go/src/io/io.go:384 +0x350
io.Copy(...)
    /usr/lib/go/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x558fdf3d17fd, 0xc, 0x7f38280f03a8, 0xc0016054a0, 0xc009b9d5c0)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:27 +0x166
created by github.com/multiformats/go-multistream.SelectProtoOrFail
    /build/go-ipfs/src/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:23 +0x97
aschmahmann commented 4 years ago

I was running into this issue very frequently on Windows 10 with go-ipfs 0.4.22 (e.g. more than once per hour), but after upgrading to master this occurs much less frequently. While the issue is not actually fixed, it might be an easy stop gap for those of us having issues.

go-ipfs version: 0.5.0-dev-410e7e878-dirty
Repo version: 7
System version: amd64/windows
Golang version: go1.13
djdv commented 4 years ago

Adding my logs to this thread in case they're useful. Context: started the daemon, added 2 directories (~80GB each), wanted to make sure they were pinned along with a 3rd hash, so I ran the following which eventually resulted in a panic.

The dirty portions of the commit relate to changes in badger and reuseport (to build on this platform). Nothing else should be out of the ordinary. (ref: https://github.com/ipfs/go-ipfs/issues/6642#issue-493470642)

go-ipfs version: 0.5.0-dev-410e7e878-dirty
Repo version: 7
System version: amd64/solaris
Golang version: go1.12.10

ipfs pin add --progress X Y Z
panic: concurrent write to websocket connection

goroutine 2952978 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc0093b90e0, 0xc0093b9501, 0x0, 0x0, 0x0, 0xfffffd7fef2e4d98, 0x0)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:610 +0x62f
github.com/gorilla/websocket.(*messageWriter).Close(0xc0093b90e0, 0x40, 0xc0009cfd00)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:724 +0x62
github.com/gorilla/websocket.(*Conn).beginMessage(0xc01a2f8f20, 0xc0093b9530, 0x2, 0xc0009cfda0, 0x786589)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:473 +0x244
github.com/gorilla/websocket.(*Conn).NextWriter(0xc01a2f8f20, 0x2, 0xfffffd7fee3caf28, 0x38, 0x200, 0x1c0)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:513 +0x53
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc01a2f8f20, 0x2, 0xc016a87fc0, 0x20, 0x40, 0x10, 0xc00bc4bc50)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:766 +0x73
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc00a2a3b90, 0xc016a87fc0, 0x20, 0x40, 0x10924a0, 0x11d3000, 0xfffffd7feecc90e8)
        src/github.com/ipfs/go-ipfs/vendor/github.com/libp2p/go-ws-transport/conn.go:74 +0x59
bytes.(*Buffer).WriteTo(0xc0093b9440, 0xfffffd7feeb91958, 0xc00b200120, 0xfffffd7feecc90e8, 0xc0093b9440, 0x927401)
        /usr/lib/golang/1.12/src/bytes/buffer.go:242 +0xb8
io.copyBuffer(0xfffffd7feeb91958, 0xc00b200120, 0x14cc680, 0xc0093b9440, 0x0, 0x0, 0x0, 0xc00bc4b340, 0x437cd6, 0x1382758)
        /usr/lib/golang/1.12/src/io/io.go:384 +0x34e
io.Copy(...)
        /usr/lib/golang/1.12/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x1201aa0, 0xa, 0xfffffd7feeb918c0, 0xc00b200120, 0xc01bf6e240)
        src/github.com/ipfs/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:27 +0x164
created by github.com/multiformats/go-multistream.SelectProtoOrFail
        src/github.com/ipfs/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:23 +0x95
Error: unexpected EOF
Stebalien commented 4 years ago

@vyzo going to cook up some patches to help debug this.

vyzo commented 4 years ago

https://github.com/libp2p/go-ws-transport/issues/56 adds instrumentation (and a lock) to catch concurrent writes and print stack traces. Can you try it out and report?

Stebalien commented 4 years ago

@vyzo could you create a branch of go-ipfs that updates go-ws-transport?

vyzo commented 4 years ago

I guess I could do that.

djdv commented 4 years ago

Follow up: (still go-ipfs 410e7e878, but with https://github.com/libp2p/go-ws-transport/commit/e2dded5874d31da646da3d4a4176d90e0df00fb8 vendored) Started the daemon, tried accessing hashes of mine through the gateway, eventually this happened.

Edit: gist with more logging after running for more time.

https://gist.github.com/djdv/1d6cd8c552d2d2983cfe11f7918058d2

vyzo commented 4 years ago

I just pushed a debug/issue-6197 branch which has the patch too.

UltraNote commented 4 years ago

go-ipfs version: 0.4.22 and go-ipfs version: 0.5.0-dev

ipfs_panic

zquestz commented 4 years ago

Also seeing this on my node:

panic: concurrent write to websocket connection goroutine 154743888 [running]: github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc02f8aeb40, 0x20613ed80877501, 0x0, 0x0, 0x0, 0xc01822a4d0, 0xdc2fc7) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x77a github.com/gorilla/websocket.(*messageWriter).Close(0xc02f8aeb40, 0x40, 0xc01822a548) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x56 github.com/gorilla/websocket.(*Conn).prepWrite(0xc00a9cdce0, 0x2, 0x0, 0x0) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x20d github.com/gorilla/websocket.(*Conn).NextWriter(0xc00a9cdce0, 0x2, 0x7f0fdc9430e8, 0x20, 0x200, 0x100) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x39 github.com/gorilla/websocket.(*Conn).WriteMessage(0xc00a9cdce0, 0x2, 0xc030ee7900, 0x22, 0x40, 0x10, 0xc01adfb0d0) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x73 github.com/libp2p/go-ws-transport.(*Conn).Write(0xc02f8ae840, 0xc030ee7900, 0x22, 0x40, 0x163dfa0, 0x1795ea0, 0x7f10199f5cc0) /go/pkg/mod/github.com/libp2p/go-ws-transport@v0.0.4/conn.go:74 +0x59 bytes.(*Buffer).WriteTo(0xc02c3e8150, 0x7f101805ca88, 0xc02f8aeab0, 0x7f10199f5cc0, 0xc02c3e8150, 0xe25e01) /usr/local/go/src/bytes/buffer.go:242 +0xb8 io.copyBuffer(0x7f101805ca88, 0xc02f8aeab0, 0x1a98d00, 0xc02c3e8150, 0x0, 0x0, 0x0, 0xc01ad51500, 0x9a59d6, 0x1946a18) /usr/local/go/src/io/io.go:384 +0x34e io.Copy(...) /usr/local/go/src/io/io.go:364 github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x17c705d, 0xc, 0x7f101805c9f0, 0xc02f8aeab0, 0xc001bb30e0) /go/pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:27 +0x164 created by github.com/multiformats/go-multistream.SelectProtoOrFail /go/pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:23 +0x95
vyzo commented 4 years ago

Can you try running with the debug/issue-6197 branch? It should eliminate the panic and also print stack traces to help us debug the issue.

zquestz commented 4 years ago

Can you post an image on docker hub? If so I would happily test it. Right now I am just using 0.5.0-dev on master.On Nov 19, 2019 12:28 PM, vyzo notifications@github.com wrote:Can you try running with the debug/issue-6197 branch? It should eliminate the panic and also print stack traces to help us debug the issue.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

jpeletier commented 4 years ago

@zquestz Here you go: epiclabs/go-ipfs:issue-6197 https://hub.docker.com/repository/docker/epiclabs/go-ipfs

jpeletier commented 4 years ago

@vyzo Here is a stack trace of issue-6197 that printed out after some time running:

WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
goroutine 159924 [running]:
runtime/debug.Stack(0xc0007e9d08, 0x9e3b61, 0x1a60e88)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
    /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/libp2p/go-ws-transport.(*Conn).Write.func1(0xc0019091c0)
    /home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/conn_native.go:82 +0x3d
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc0019091c0, 0xc001fb5500, 0x34, 0x40, 0x34, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/conn_native.go:90 +0x18b
github.com/libp2p/go-libp2p-secio.(*etmWriter).WriteMsg(0xc0019093c0, 0xc00349af50, 0x10, 0x10, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:64 +0x2ee
github.com/libp2p/go-libp2p-secio.readWriteMsg.func1(0xc000e33620, 0x7f8991ec8418, 0xc001cb54a0, 0xc00349af50, 0x10, 0x10)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:252 +0x4f
created by github.com/libp2p/go-libp2p-secio.readWriteMsg
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:251 +0xab
goroutine 159804 [running]:
runtime/debug.Stack(0x9f6391, 0xc0004c98d8, 0x2)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
    /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/libp2p/go-ws-transport.(*Conn).Close(0xc0019091c0, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/conn_native.go:98 +0x1cd
github.com/libp2p/go-libp2p-secio.(*secureSession).runHandshake(0xc0007ed800, 0x1beb560, 0xc0012f1620, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/protocol.go:131 +0x29c
github.com/libp2p/go-libp2p-secio.newSecureSession(0x1beb560, 0xc0012f0c00, 0xc000042210, 0x22, 0x1bf3240, 0xc00093ed20, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/protocol.go:95 +0x1c5
github.com/libp2p/go-libp2p-secio.(*Transport).SecureOutbound(0xc0001f0360, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0xc0001f0360, 0x0, 0x0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/transport.go:40 +0x9b
github.com/libp2p/go-conn-security-multistream.(*SSMuxer).SecureOutbound(0xc0001f2190, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0x17a5100, 0x1818560, 0xc0004c9a00, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-conn-security-multistream@v0.1.0/ssms.go:56 +0xf2
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).setupSecurity(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0x1bcf500, 0xc00093c6e0, 0x1c011e0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:108 +0x10c
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).upgrade(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x1bee720, 0xc0002d2038, 0x1bfd8e0, 0xc002663da0, 0xc002127950, 0x22, 0x0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:86 +0x29e
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).UpgradeOutbound(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x1bee720, 0xc0002d2038, 0x1bfd8e0, 0xc002663da0, 0xc002127950, 0x22, 0x0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:57 +0xcf
github.com/libp2p/go-ws-transport.(*WebsocketTransport).Dial(0xc0002d2038, 0x1beb560, 0xc0012f0c00, 0x1c011e0, 0xc0020cb280, 0xc002127950, 0x22, 0xc002262580, 0xc0028f9450, 0xdf84755bf, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/websocket.go:63 +0xed
github.com/libp2p/go-libp2p-swarm.(*Swarm).dialAddr(0xc00091f080, 0x1beb560, 0xc0012f0c00, 0xc002127950, 0x22, 0x1c011e0, 0xc0020cb280, 0x2790e60, 0x9e330a, 0xc0012f0c00, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.2/swarm_dial.go:462 +0x219
github.com/libp2p/go-libp2p-swarm.(*dialLimiter).executeDial(0xc000167e50, 0xc002414a80)
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.2/limiter.go:218 +0x1a9
created by github.com/libp2p/go-libp2p-swarm.(*dialLimiter).addCheckFdLimit
    /home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.2/limiter.go:168 +0x5fc
lazyweirdo commented 4 years ago

Same issue here, go-ipfs 0.4.22 , swarm, api and read only gateway open ...

asiniscalchi commented 4 years ago

same here ... gp-ipfs 0.4.22 and 0.4.18

Stebalien commented 4 years ago

This should be fixed by https://github.com/ipfs/go-ipfs/pull/6804.