search

krasserm / akka-persistence-kafka

A replicated Akka Persistence journal backed by Apache Kafka
Apache License 2.0
201 stars 59 forks source link

Lots of open files #9

Closed danielhopkins closed 10 years ago

danielhopkins commented 10 years ago

I'm running on the HEAD commit (3646d8795eb78e0aa44f8418ed2d70f742ebaf77) and there seem to be lots of open socket connections between kafka and the persistence application.

On os x at least I run this periodically:

pgrep -f kafka-server | xargs lsof -i 4 -P -p | grep 9092 | wc -l

During load testing I'm running into to "too many open files" which I think might be related.

Could there be a connection leak in there?

krasserm commented 10 years ago

Thanks for reporting, there actually were connection leaks in the code which should now be fixed with d92383758b00b888b0c338c938a4b70dad54dcc1. Can you please verify?

Also, too many topics in Kafka (= too many persistent actors) can also cause a too many open files (see also this question on Quora and this discussion on akka-user).

danielhopkins commented 10 years ago

looks good to me, under heavy load it's only using a couple of connections now

danielhopkins commented 10 years ago

feel free to close this and thanks for the fix

krasserm commented 10 years ago

You're welcome, thank you for your valuable feedback.