search

cloudfoundry-community / firehose-to-syslog

Send firehose events from Cloud Foundry to syslog.
MIT License
44 stars 58 forks source link

CPU usage maxing out with 4.0.0 #164

Closed aleliaert closed 6 years ago

aleliaert commented 6 years ago

CPU usage used to be at around 20%, now seeing around 100% with same workload.

I'm curious if this is expected behavior? We're not seeing any data loss so maybe the extra CPU resources are being consumed checking something at a high frequency?

Thanks!

shinji62 commented 6 years ago

Interesting,

At least is seems data lost are gone. I will try to re-enable profiling sometime next week. The code and dependency change a lot so I will try to track where is the cpu be consume.

Btw are you using the stats server ?

Thanks

aleliaert commented 6 years ago

On data loss, the new version has helped us quite a bit. We have run into some cases where it's reporting missed logs, but it keeps running versus encountering the i/o timeout reading from the firehose and crashing as before.

Do you have any advice on calculating max memory consumption versus buffer size? Is it likely that the app has hit its highest memory usage when the "Missed Logs" error is observed?

On stats, we have cf ssh-ed into a container running an f2s instance and have played around with the endpoints it offers. Lots of details in the /debug/pprof endpoints, can you recommend some that shed light on the CPU usage?

Thanks!

shinji62 commented 6 years ago

For the buffer size is in number of enveloppe, which have a MAX size of 64KB so I think it's a tradeoff.

/debug/pprof is use for profiling. Getting a sample of your env could be nice. you can check here https://golang.org/pkg/net/http/pprof/

shinji62 commented 6 years ago

Ok, I think this is due to a "busy" loop or infinite loop when reading the buffer.

Not sure what to do about that right now.

aleliaert commented 6 years ago

@shinji62 - I've been wanting to learn the go language and also contribute to open-source CF efforts. Maybe this is my opportunity... Some questions before I dig in:

aleliaert commented 6 years ago

I opened a PR for this :)

shinji62 commented 6 years ago

Already merge and release so closing.