Orphan go routines and memory usage ...

[adsb-related-code]

I've gone through sync and added concurrency locks to the map

profiling still shows it creating every increasing amounts of goroutines ... ![screenshot_2018-08-28_17-24-53]

(https://user-images.githubusercontent.com/34561158/44758159-a8258a80-aae7-11e8-8f37-8b3bd87ded0f.png)

[adsb-related-code]

May or may not still be an issue. Reddit reports not leaking ..

[cjkreklow]

I've created a fork that uses a stream client I was already working on writing, and writes to all the clients from a single goroutine. I did quite a bit of rearranging as well. If you want to check out the dev branch of my fork and see what you think, I'd appreciate any feedback.

[adsb-related-code]

Will do!

[adsb-related-code]

I'll give it a go in production with about 150 incoming connections. How do I pull it down?

[cjkreklow]

I pushed a couple updates, you should be able to clone the repo, checkout dev and build it now.

[adsb-related-code]

Ok let's see if I can do this without breaking something

[adsb-related-code]

Looking for so far!

Currently running on pub-vrs.adsbexchange.com 32001

I'll have more time this week to look at that code. I'm very new to Go, as you could probably tell from my terrible code.

62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 6.1 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 7.0 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 8.1 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 9.1 MB ========= ========== ========== ========= ========== ========== 62001 - # goroutines: 74 Client Connections: 70 Memory Acquired: 16 MB GC: true Last GC: 1535519336066545010 Next GC: 11 MB Heap Alloc: 10 MB ========= ========== ==========

[cjkreklow]

Glad to hear it's working! I'm looking at how to do the auto-reconnect in #4, I'll maybe have something for that tomorrow.

[adsb-related-code]

We need to run 5 of them at once to cover all the ports using a bash script with a do while - if they crash it sleeps for 10 seconds and launches it again.

If it loses the TCP stream coming into it - os.Exit(1) is fine. Not pretty, but it will work.

I'll keep some stats on it.

32001 (1 second updates for VRS) is the busiest stream, usually around 100-150 connections and 500 Mb/s - 700 Mb/s.

Thanks!

[adsb-related-code]

There's still a lot of orphan go routines even in the update to the main code I pushed today.

It looks like the sendData maybe? Does it 'freeze' if it can't send or some odd TCP error happens.

github.com/dbudworth/greak

https://gist.github.com/adsbxchange/ad95965de88063a71fa147235fdde2f1

[adsb-related-code]

I think I'm on to something. If a client has a slow connection then the go routines just pile up doing the 'slow send'.

Like a sort of denial of service attack .. go just keeps stacking memory waiting for the routines to complete.

[cjkreklow]

Have you been able to try my rewritten version? I'm using a single goroutine per client connection with a channel, and slow readers just drop messages if the channel is backed up.

[adsb-related-code]

Ahh ok let me give it a try.

I ran it overnight and it crashed at somepoint with memory overflow

---

From: Collin Kreklow notifications@github.com Sent: Wednesday, August 29, 2018 3:57:36 PM To: adsbxchange/tcp-relay-pub-vrs Cc: James Stanford; Author Subject: Re: [adsbxchange/tcp-relay-pub-vrs] Orphan go routines and memory usage ... (#6)

Have you been able to try my rewritten version? I'm using a single goroutine per client connection with a channel, and slow readers just drop messages if the channel is backed up.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/adsbxchange/tcp-relay-pub-vrs/issues/6#issuecomment-417133866, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ag9chvxx7PtkqOiL3Z0fE6Vks73rTGIsks5uVxxggaJpZM4WQn3x.

[adsb-related-code]

What am I doing incorrectly with git?

linux:~/tcp-relay-pub-vrs$ git pull https://github.com/cjkreklow/tcp-relay-pub-vrs.git dev
 From https://github.com/cjkreklow/tcp-relay-pub-vrs
 * branch            dev        -> FETCH_HEAD
Already up to date.
linux:~/tcp-relay-pub-vrs$ go get github.com/cjkreklow/tcp-relay-pub-vrs/stream 
linux:~/tcp-relay-pub-vrs$ go build tcp-relay.go 
# command-line-arguments
./tcp-relay.go:89:8: client.ReconnectMax undefined (type *stream.StreamClient has no field or method ReconnectMax)
./tcp-relay.go:90:8: client.ReconnectDelay undefined (type *stream.StreamClient has no field or method ReconnectDelay)

[adsb-related-code]

oh hah ... derp .. it's late here

[adsb-related-code]

Ok I guess I need to pull the correct stream somehow .. I tried again this am to get it to compile

[cjkreklow]

Yeah, that won't work since everything's on the dev branch. I assume you're still running Go 1.10, so try this: go get -d github.com/cjkreklow/tcp-relay-pub-vrs cd $HOME/go/src/github.com/cjkreklow/tcp-relay-pub-vrs git checkout dev go build

You should have a clean build that way.

[cjkreklow]

Were you able to get it to build? If not I can merge the changes to master to make it easier to "go get".

[adsb-related-code]

I haven't had a chance to try it yet. I'll be home this evening and will give it a go. I assume it will work - I was able to do it initially.

---

From: Collin Kreklow notifications@github.com Sent: Saturday, September 1, 2018 8:51:57 AM To: adsbxchange/tcp-relay-pub-vrs Cc: James Stanford; Author Subject: Re: [adsbxchange/tcp-relay-pub-vrs] Orphan go routines and memory usage ... (#6)

Were you able to get it to build? If not I can merge the changes to master to make it easier to "go get".

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/adsbxchange/tcp-relay-pub-vrs/issues/6#issuecomment-417868494, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ag9chlze-0nipahCZL_VxSzJ3BT_rImUks5uWq0dgaJpZM4WQn3x.