search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Kill the topology before shutting down in local mode. #153

Closed dbrand666 closed 8 years ago

dbrand666 commented 8 years ago

The shutdown process tended to hang if the topology was misbehaving. Killing the topology first seems to work better.

poros commented 8 years ago

I already tried out this approach several versions ago and then removed it in https://github.com/Yelp/pyleus/commit/03842dc2547ede46a3858c25698abae1e2dc5dab.

The reason why I removed this code was because every time pyleus tried to kill the the topology and failed, the cluster shutdown failed as well, leaving a lot of temporary files in /tmp. And that was happening so many times that soon filled up a good portion of the disk space. Calling only the cluster shutdown worked well most of the times, instead.

This was many Storm versions ago, though. Would you mind checking if this still happens with your change?

dbrand666 commented 8 years ago

Yeah. Let's close this. I haven't seen the behavior you describe but it also doesn't seem to fix all the shutdown issues consistently. I think the real problem is upstream.