Open adsb-related-code opened 6 years ago
May or may not still be an issue. Reddit reports not leaking ..
I've created a fork that uses a stream client I was already working on writing, and writes to all the clients from a single goroutine. I did quite a bit of rearranging as well. If you want to check out the dev branch of my fork and see what you think, I'd appreciate any feedback.
Will do!
I'll give it a go in production with about 150 incoming connections. How do I pull it down?
I pushed a couple updates, you should be able to clone the repo, checkout dev and build it now.
Ok let's see if I can do this without breaking something
Looking for so far!
Currently running on pub-vrs.adsbexchange.com 32001
I'll have more time this week to look at that code. I'm very new to Go, as you could probably tell from my terrible code.
62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 6.1 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 7.0 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 8.1 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 9.1 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 10 MB ========= ========== ==========
Glad to hear it's working! I'm looking at how to do the auto-reconnect in #4, I'll maybe have something for that tomorrow.
We need to run 5 of them at once to cover all the ports using a bash script with a do while - if they crash it sleeps for 10 seconds and launches it again.
If it loses the TCP stream coming into it - os.Exit(1) is fine. Not pretty, but it will work.
I'll keep some stats on it.
32001 (1 second updates for VRS) is the busiest stream, usually around 100-150 connections and 500 Mb/s - 700 Mb/s.
Thanks!
There's still a lot of orphan go routines even in the update to the main code I pushed today.
It looks like the sendData maybe? Does it 'freeze' if it can't send or some odd TCP error happens.
github.com/dbudworth/greak
https://gist.github.com/adsbxchange/ad95965de88063a71fa147235fdde2f1
I think I'm on to something. If a client has a slow connection then the go routines just pile up doing the 'slow send'.
Like a sort of denial of service attack .. go just keeps stacking memory waiting for the routines to complete.
Have you been able to try my rewritten version? I'm using a single goroutine per client connection with a channel, and slow readers just drop messages if the channel is backed up.
Ahh ok let me give it a try.
I ran it overnight and it crashed at somepoint with memory overflow
From: Collin Kreklow notifications@github.com Sent: Wednesday, August 29, 2018 3:57:36 PM To: adsbxchange/tcp-relay-pub-vrs Cc: James Stanford; Author Subject: Re: [adsbxchange/tcp-relay-pub-vrs] Orphan go routines and memory usage ... (#6)
Have you been able to try my rewritten version? I'm using a single goroutine per client connection with a channel, and slow readers just drop messages if the channel is backed up.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/adsbxchange/tcp-relay-pub-vrs/issues/6#issuecomment-417133866, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ag9chvxx7PtkqOiL3Z0fE6Vks73rTGIsks5uVxxggaJpZM4WQn3x.
What am I doing incorrectly with git?
linux:~/tcp-relay-pub-vrs$ git pull https://github.com/cjkreklow/tcp-relay-pub-vrs.git dev
From https://github.com/cjkreklow/tcp-relay-pub-vrs
* branch dev -> FETCH_HEAD
Already up to date.
linux:~/tcp-relay-pub-vrs$ go get github.com/cjkreklow/tcp-relay-pub-vrs/stream
linux:~/tcp-relay-pub-vrs$ go build tcp-relay.go
# command-line-arguments
./tcp-relay.go:89:8: client.ReconnectMax undefined (type *stream.StreamClient has no field or method ReconnectMax)
./tcp-relay.go:90:8: client.ReconnectDelay undefined (type *stream.StreamClient has no field or method ReconnectDelay)
oh hah ... derp .. it's late here
Ok I guess I need to pull the correct stream somehow .. I tried again this am to get it to compile
Yeah, that won't work since everything's on the dev branch. I assume you're still running Go 1.10, so try this: go get -d github.com/cjkreklow/tcp-relay-pub-vrs cd $HOME/go/src/github.com/cjkreklow/tcp-relay-pub-vrs git checkout dev go build
You should have a clean build that way.
Were you able to get it to build? If not I can merge the changes to master to make it easier to "go get".
I haven't had a chance to try it yet. I'll be home this evening and will give it a go. I assume it will work - I was able to do it initially.
From: Collin Kreklow notifications@github.com Sent: Saturday, September 1, 2018 8:51:57 AM To: adsbxchange/tcp-relay-pub-vrs Cc: James Stanford; Author Subject: Re: [adsbxchange/tcp-relay-pub-vrs] Orphan go routines and memory usage ... (#6)
Were you able to get it to build? If not I can merge the changes to master to make it easier to "go get".
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/adsbxchange/tcp-relay-pub-vrs/issues/6#issuecomment-417868494, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ag9chlze-0nipahCZL_VxSzJ3BT_rImUks5uWq0dgaJpZM4WQn3x.
I've gone through sync and added concurrency locks to the map
profiling still shows it creating every increasing amounts of goroutines ... ![screenshot_2018-08-28_17-24-53]
(https://user-images.githubusercontent.com/34561158/44758159-a8258a80-aae7-11e8-8f37-8b3bd87ded0f.png)