search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Possible memory leak in component.py for pending task ids and commands #60

Open ecanzonieri opened 9 years ago

ecanzonieri commented 9 years ago

Pyleus enqueues commands and task ids into a pending unlimited queue. This can be dangerous and lead to a memory leak. We should define a reasonable max size for these data structures and use a blocking queue.

Migrated from PYLEUS-139.

imcom commented 8 years ago

Is there anyone still managing this issue? or please someone loop me in? has this bug fixed already? I am seeing memory leak in Pyleus bolts and bolts hang situation ... need immediate help from Pyleus team

poros commented 8 years ago

These are the two lines of code this issue refers to: https://github.com/Yelp/pyleus/blob/develop/pyleus/storm/component.py#L154

This behavior was inherited from the Python implementation of a component that Storm itself suggests. I believe that writing to a queue without a maximum size is a problem, but I am not sure you can call it a memory leak. The memory is not "leaking", it's just that your upstream component is producing tuples much faster than your downstream component is able to process and they queue up in memory until it goes OOM. This also happens to pure Java Storm topologies (at least, it was the case the last time I checked).

Nevertheless, every contribution is much appreciated, so feel free to submit a pull request if you are able to figure out a reasonable size limit for these queues.

imcom commented 8 years ago

hi @poros , thanks for you response, this piece of information is of utter importance. As I am dealing with Pyleus/Storm memory "leaking"/leaking issues. I will definitely look into this and hope I could find something interesting )

imcom commented 8 years ago

Storm has throttling on its spout when the pending queue is full, but what I think is if we do similar limitation on Pyleus side, it would make things more complicated for developers. Instead, if memory "leaking" happens due to this implementation, then it should be considered as a performance issue that should be addressed in topology structure design or implementation. The memory "leaking" is a lot more obvious than slow topology, so people have better chance to refine and measure their work.