search

bradybd / pdsh

Automatically exported from code.google.com/p/pdsh
GNU General Public License v2.0
0 stars 0 forks source link

PDSH is truncating very long lines #36

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 

I have very long lines of xml strings that I would like to capture with pdsh 
from multiple hosts. pdsh seems to trim from the head of each line. 

Here is how I can show the issue without cluttering the message. When I run wc 
on remote server, I get 93K chars:

$ pdsh -N -w ssh:server-1 "grep b429d9114463de0b721e4760995665a6 
/var/log/server.log* | grep VISUAL | grep cmd | wc "
      4      52   93311

When I run wc on my localhost, I see only 30K chars:

$ pdsh -N -w ssh:server-1 "grep b429d9114463de0b721e4760995665a6 
/var/log/videoid/server.log* | grep VISUAL | grep cmd " | wc 
      3       4   30956

Original issue reported on code.google.com by nejat...@gmail.com on 3 Sep 2011 at 12:23

GoogleCodeExporter commented 9 years ago 
Oh, that doesn't sound good. I'll try to work on this issue this weekend.

Thanks,
mark

Original comment by mark.gro...@gmail.com on 3 Sep 2011 at 12:45

GoogleCodeExporter commented 9 years ago 
Can you give more details about how pdsh is trimming the output? I tried some 
very simple tests with long lines of output, a single long line of output, etc 
and
have not been able to reproduce this issue.

If your data isn't sensitive it might help a lot to attach the output of 
grep STRING /var/log/server.log* | grep VISUAL, or perhaps at first an example 
of
an original line of output, and one that has been mangled by pdsh.

Thanks!

Original comment by mark.gro...@gmail.com on 3 Sep 2011 at 3:58

GoogleCodeExporter commented 9 years ago 
Sorry, I _was_ able to reproduce this issue.

Currently, pdsh truncates lines at 8K.
I'm working on a fix.

Original comment by mark.gro...@gmail.com on 4 Sep 2011 at 1:20

GoogleCodeExporter commented 9 years ago 
Could you try the two attached patches? Together they should allow lines up to 
128K.

Original comment by mark.gro...@gmail.com on 7 Sep 2011 at 3:50

Attachments:

GoogleCodeExporter commented 9 years ago 
Yes, patches fixed the issue.

Thank you very much,
Nejat

Original comment by nejat...@gmail.com on 7 Sep 2011 at 6:52

GoogleCodeExporter commented 9 years ago 
Thanks for reporting the issue. This will be fixed in pdsh-2.27

Original comment by mark.gro...@gmail.com on 7 Sep 2011 at 7:37

GoogleCodeExporter commented 9 years ago 
This issue was updated by revision 8117d3ac974c.

Add a test that reproduces the user observation that pdsh is truncating
very long lines. Lines of length > 8191 characters are probably required
to reproduce this behavior, which appears to be due to a static buffer
used in pdsh output routines, along with a maximum circular buffer size
of 8192 characters. Both issues will need to be resolved before this
test passes.

Original comment by mark.gro...@gmail.com on 8 Sep 2011 at 4:37

GoogleCodeExporter commented 9 years ago 
This issue was closed by revision c37f0b533b93.

Original comment by mark.gro...@gmail.com on 8 Sep 2011 at 4:37