search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

Switching to parallel-ssh instead of fabric for SSH bits #394

Open dan-blanchard opened 7 years ago

dan-blanchard commented 7 years ago

Fabric is not guaranteed to work well when using the @parallel decorator and it seems I run into issues with it flaking out when updating virtualenvs more and more frequently. parallel-ssh is built to handle much larger clusters and seems much more robust.

sureshsarda commented 7 years ago

I would like to help, but I might need some help getting started.

westover commented 7 years ago

Can I make a suggestion of using Ansible? We are managing our deployments using Ansible and it supports parallel execution

tanaysoni commented 7 years ago

Hi @dan-blanchard. I'd like to take this up. Wondering if you evaluated any other alternatives like Invoke(or Fabric 2.x)?

dan-blanchard commented 7 years ago

Invoke doesn't handle SSH connections, so it's not really a valid alternative here. We need to be to more reliably connect to a bunch of nodes in parallel via SSH. parallel-ssh seems like a solid library for doing exactly that. We could probably switch from fabric to invoke for the local bits (like managing the environment).

As for Fabric 2, it's still not officially out on PyPI (or out of beta), so I'd like to wait for a release before trying that. Also, it remains to be seen if they'll have actually fixed the concurrency issues that are in the current version.