search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

Storm 0 10 0 #170

Open mzbyszynski opened 8 years ago

mzbyszynski commented 8 years ago

Hey Guys,

I've started trying to get pyleus to compile and deploy using storm 0.10.0. I don't think it is safe to merge yet, but I wanted to get the PR up there in case anyone else wants to help test this out. So far my testing has been going well, but if anyone else feels like pitching in that would be great.

I haven't done that much local testing yet, so I could use help testing that.

All feedback welcome!

Thanks,

Marc

grant-traynor commented 8 years ago

How are you getting on with this? Would really like to get involved, we're upgrading our platform and possibly moving from the Petrel toolkit to this one. Have you managed to configure a development environment to build / deploy topologies in Storm 0.10.0? How did you deal with the deployment / pip download process?

Would be great to hear from you?

mzbyszynski commented 8 years ago

Hi @grant-traynor. I've been deploying our topologies to 0.10.0 since before I submitted this PR and so far I haven't found any issues. To deploy, I've manually installed my locally built version of pyleus in my project's venv and then run a local pypi server that servers my locally built pyleus version as described in the development tips section of the documentation.

Have you been having trouble getting that working?

grant-traynor commented 8 years ago

Hi @mzbyszynski. I haven't yet had a crack at setting up as per the dev tips. Will try that now. Thanks for the pointer.

dearli commented 8 years ago

Hi,there: I'm very happy to find this issue.I have using pyleus which deploied with storm 0.10.0 for long. When i use it ,I find many heartbeat timeout errors occured,and then worker will be shutdown.I wonder if it is because of pyleus image my own pom.xml in topology_builder folder is as below:(a part) image

mzbyszynski commented 8 years ago

@dearli The only time I've gotten these timeouts is when I had a bolt that was slow and so the heartbeat requests were backing up, so the heartbeats weren't being processed within the default heartbeat timeout limit. At least I think that is what was happening...

grant-traynor commented 8 years ago

It does seem to be related to (1) speed of bolt / spout and (2) overall load on the topology. If the topology is flooded with tuples, then the heartbeats / acks seem to get delayed. Adjust this setting in your storm.yaml config file:

How long for a heartbeat until supervisor considers that worker dead and

tries to restart it supervisor.worker.timeout.secs: 90

Cheers, Grant.

On Wed, Jan 27, 2016 at 1:27 PM, Marc Zbyszynski notifications@github.com wrote:

@dearli https://github.com/dearli The only time I've gotten these timeouts is when I had a bolt that was slow and so the heartbeat requests were backing up, so the heartbeats weren't being processed within the default heartbeat timeout limit. At least I think that is what was happening...

— Reply to this email directly or view it on GitHub https://github.com/Yelp/pyleus/pull/170#issuecomment-175351878.

Grant TRAYNOR Home : +61 (0)2 4942 5802 Mobile : +61 (0)428 408 558

imcom commented 8 years ago

Hi dudes, what's the current status for supporting Storm 0.10.x ? This PR has been here for a while. As of saying... Storm released 1.0 .... and it provides really some nice features ... any ideas?