Merge output from multiple servers

[GoogleCodeExporter]

I'm happy user of pssh. I used it yesterday to setup 10 servers. I run pssh 
with -i option to watch server output. Servers are identical except IP 
addresses and hostnames.

Some commands produce a page of output (like apt-get install). So command on 10 
servers produce 10 copies of almost identical output. Lot of noise. It would be 
helpful, if pssh merged output of commands while output is the same.

Consider command that outputs:

===
updating bla-bla
installing something useful
time is 14:30:22
===

pssh -i with three servers could output:

===
srv1:
srv2:
srv3:
updating bla-bla
installing bla-bla
srv1:
time is 14:30:21
srv2:
time is 14:30:22
srv3:
error happened
===

I think pssh should merge only identical heads of servers output (not tail, not 
middle).

Original issue reported on code.google.com by stepan.k...@gmail.com on 6 Mar 2011 at 11:18

[GoogleCodeExporter]

I agree that there's often redundancy in the output, but I don't think there is 
any reasonable general-purpose way to address this.  However, it sounds like 
something like this would be helpful in a lot of special-case situations.  If I 
were trying to implement this, I think I would write a wrapper script that 
behaved roughly as follows:

1) Make a temporary directory (say $tmpdir).
2) Run pssh with -o and -e pointing to $tmpdir/out and $tmpdir/err.
3) When pssh terminates, run the case-specific comparison/consolidation code.
4) Output the summary.

I think you'll find that this isn't too hard to write, and it would be easy to 
customize for your specific needs.  If you think that this would be helpful to 
include with pssh, I would be happy to add it to a "contrib" directory with 
pssh.

Original comment by amcna...@gmail.com on 7 Mar 2011 at 5:07

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

A long time ago, in a galaxy far away, I wrote a tool called ... pssh.  It did 
very very similar things to this tool, except it was written in Perl.  One of 
the things its users loved the most was precisely its ability to merge output 
from multiple servers.  The algorithm I used was the obvious one: do a search 
in each host's output for per-host data (i.e. hostname and IP address) and 
replace them with generic stubs ("[hostname]" and "[IP]" respectively).  Then 
calculate the MD5 digest of each munged output, group them according to this 
digest, and display each group of output, clearly indicating at the top of each 
which hosts belong to that group.

The output was collected in memory rather than on disk.  From a scalability 
perspective this was perhaps not the best approach, although in practice it was 
never a problem even when running across hundreds of machines.  Perhaps this 
time round I would go for on-disk collection, but on the strict condition that 
the tool should not *require* users to (a) think of a suitable output directory 
and (b) have to delete it after each run.  In other words it should default to 
a safely generated temporary directory, and default to cleaning up that 
directory unless the user explicitly specifies an output directory or 
explicitly asks for it not to be cleaned up.  This is because users want to be 
able to do multiple lightning quick comparisons across machines (e.g. pssh ... 
cat /etc/hosts), so any given pssh should not require extra commands (such as 
rm -rf $tmpdir) to be run afterwards.  But on the flip side, the user *will* 
sometimes want to grep -r through the output directory, so --no-cleanup should 
be an option when no output directory is explicitly specified.

I would recommend that output grouping be implemented as part of the core code 
rather than as a wrapper.  It is simply another output format, and so should be 
available via a CLI option.

BTW I have fallen out of love with Perl and returned from that galaxy far away. 
 So needless to say, I would also love to see this feature implemented by this 
project.

Original comment by adam.spi...@gmail.com on 3 Mar 2012 at 11:34

[GoogleCodeExporter]

adam.spiers, the output grouping idea sounds pretty cool to me. I probably 
don't have time to work on this right now, but if you're interested in working 
on it, I'd be happy to answer any questions you have. With the "-i" option, 
we're already storing all of the output in RAM, and in practice, I don't think 
this would be the biggest scalability problem to worry about.

Anyway, I think it's a nice idea, and it wouldn't be all that hard to 
implement. If you've fallen out of love with Perl, it might be a ripe time to 
fall in love with Python. :)

Original comment by amcna...@gmail.com on 3 Mar 2012 at 4:13

[GoogleCodeExporter]

I don't have time to work on it either, unfortunately :-/  I sent you an email 
with more details though :)

Actually I fell in love with Ruby way back in 2002.  I've had many flings with 
Python since then, and I'm quite fond of it, but it will never develop into a 
serious relationship after experiencing Ruby ;-)

Original comment by adam.spi...@gmail.com on 3 Mar 2012 at 4:17