PDSH is truncating very long lines

[GoogleCodeExporter]

I have very long lines of xml strings that I would like to capture with pdsh 
from multiple hosts. pdsh seems to trim from the head of each line. 

Here is how I can show the issue without cluttering the message. When I run wc 
on remote server, I get 93K chars:

$ pdsh -N -w ssh:server-1 "grep b429d9114463de0b721e4760995665a6 
/var/log/server.log* | grep VISUAL | grep cmd | wc "
      4      52   93311

When I run wc on my localhost, I see only 30K chars:

$ pdsh -N -w ssh:server-1 "grep b429d9114463de0b721e4760995665a6 
/var/log/videoid/server.log* | grep VISUAL | grep cmd " | wc 
      3       4   30956

Original issue reported on code.google.com by nejat...@gmail.com on 3 Sep 2011 at 12:23

[GoogleCodeExporter]

Oh, that doesn't sound good. I'll try to work on this issue this weekend.

Thanks,
mark

Original comment by mark.gro...@gmail.com on 3 Sep 2011 at 12:45

• Changed state: Accepted

[GoogleCodeExporter]

Can you give more details about how pdsh is trimming the output? I tried some 
very simple tests with long lines of output, a single long line of output, etc 
and
have not been able to reproduce this issue.

If your data isn't sensitive it might help a lot to attach the output of 
grep STRING /var/log/server.log* | grep VISUAL, or perhaps at first an example 
of
an original line of output, and one that has been mangled by pdsh.

Thanks!

Original comment by mark.gro...@gmail.com on 3 Sep 2011 at 3:58

[GoogleCodeExporter]

Sorry, I _was_ able to reproduce this issue.

Currently, pdsh truncates lines at 8K.
I'm working on a fix.

Original comment by mark.gro...@gmail.com on 4 Sep 2011 at 1:20

[GoogleCodeExporter]

Could you try the two attached patches? Together they should allow lines up to 
128K.

Original comment by mark.gro...@gmail.com on 7 Sep 2011 at 3:50

Attachments:

• 0001-Increase-output-cbuf-max-size-to-128K.patch
• 0002-Use-a-dynamic-buffer-for-output-to-avoid-truncation.patch

[GoogleCodeExporter]

Yes, patches fixed the issue.

Thank you very much,
Nejat

Original comment by nejat...@gmail.com on 7 Sep 2011 at 6:52

[GoogleCodeExporter]

Thanks for reporting the issue. This will be fixed in pdsh-2.27

Original comment by mark.gro...@gmail.com on 7 Sep 2011 at 7:37

• Changed state: Fixed

[GoogleCodeExporter]

This issue was updated by revision 8117d3ac974c.

Add a test that reproduces the user observation that pdsh is truncating
very long lines. Lines of length > 8191 characters are probably required
to reproduce this behavior, which appears to be due to a static buffer
used in pdsh output routines, along with a maximum circular buffer size
of 8192 characters. Both issues will need to be resolved before this
test passes.

Original comment by mark.gro...@gmail.com on 8 Sep 2011 at 4:37

[GoogleCodeExporter]

This issue was closed by revision c37f0b533b93.

Original comment by mark.gro...@gmail.com on 8 Sep 2011 at 4:37