AFSocketServerConnector in junixsocket-jetty 2.7.0 causes GC pauses

kohlschutter / junixsocket

Unix Domain Sockets in Java 7 and newer (AF_UNIX), AF_TIPC, AF_VSOCK, and more

Apache License 2.0

433 stars 114 forks source link

AFSocketServerConnector in junixsocket-jetty 2.7.0 causes GC pauses #138

Closed kevink-sq closed 1 year ago

kevink-sq commented 1 year ago

Describe the bug

This is a follow-up to https://github.com/kohlschutter/junixsocket/issues/135 as it was found rolling out junixsocket to all services in prod caused gc spikes leading to p99/max latency. Cannot share heap dumps but the symptoms were as follows:

Services would see spikes in p99 latency growing over time. In a span of 24 hours post deploy, spikes in latency would be seen growing 30s, 1min, 3min, 5min, etc. Max we saw was 6 min.
These spikes would correlate with gc pause times (with g1gc). Memory seems to getting reclaimed but gc ends up running longer each time.
These services are running jetty 10.

We had to rollback usage of junixsocket across our services.

kohlschuetter commented 1 year ago

@kevink-sq Can you please give a little more detail, it's not actionable without additional information. Do you know which junixsocket objects were causing these GC pause times?

What Java VM on what platform are you using? What are the VM arguments? (which garbage collector, etc.). Can you post the output of junixsocket-selftest?

kohlschuetter commented 1 year ago

@kevink-sq Please try the latest 2.7.1-SNAPSHOT (2.7.1-20230828.103553 or newer). Please note that I can't release 2.7.1 on a hunch, so verifying that snapshot version is imperative if you want a proper fix.

kohlschuetter commented 1 year ago

@kevink-sq Please verify with 2.7.1.

kohlschuetter commented 1 year ago

Please verify with 2.7.2, just released. Thanks again for reporting, @kevink-sq !

kevink-sq commented 11 months ago

Hi @kohlschuetter,

We did extensive prod testing with junixsocket 2.7.2 and found no regression. Its been baking in prod for the last 3 weeks now and things look fine.

2.7.1 however did have an issue with NativeUnixSocket.poll timing out but that seems to have been fixed in #139.

Overall thanks for the work on junixsocket. We initially went on this path because we found jnr-unixsocket to have load issues during high concurrency and our infra could not support JEP 380.

kohlschuetter commented 11 months ago

@kevink-sq Thanks for the feedback, @kevink-sq, really appreciated!