pdsh 2.25 doesn't load genders module

[GoogleCodeExporter]

I built pdsh with the following:

./configure     --prefix /u06/pdsh-2.25   \
                --without-dshgroups       \
               --without-exec             \
                --without-netgroup        \
                --without-machines        \
                --without-rsh             \
                --with-genders            \
                --with-readline           \
                --with-ssh                \
                CFLAGS="-O2 -Wl,-E -I/usr/local/include -I/usr/include" \
                LDFLAGS="-L/usr/local/lib/hpux32 -L/usr/lib/hpux32"  \
                LIBS="-lcext"

wilma:/home/root/pdsh-2.25# /u06/pdsh-2.25/bin/pdsh -V
pdsh-2.25 (+readline)
rcmd modules: ssh
misc modules: (none)

It finds the ssh module but not the genders module:

wilma:/home/root/pdsh-2.25# ll /u06/pdsh-2.25/lib/pdsh
total 432
-rw-r--r--   1 root       sys          29756 Apr 20 13:16 genders.a
-rwxr-xr-x   1 root       sys            893 Apr 20 13:16 genders.la
-r-xr-xr-x   1 root       sys          76256 Apr 20 13:16 genders.so
-rw-r--r--   1 root       sys          13828 Apr 20 13:16 sshcmd.a
-rwxr-xr-x   1 root       sys            850 Apr 20 13:16 sshcmd.la
-r-xr-xr-x   1 root       sys          72940 Apr 20 13:16 sshcmd.so

Previously when I built with machine and dshgroups it loaded those modules 
without problem, but not genders.  Any ideas?  config.log uploaded.

Original issue reported on code.google.com by bwir...@gmail.com on 20 Apr 2011 at 6:45

Attachments:

• config.log

[GoogleCodeExporter]

I can't think of a reason for this behavior off the top of my head. We will have
to do some debugging. There should at least be some sort of error if the .so 
fails
to load (I think).

Can you try installing some of the other "misc" modules (--with-dshgroups for 
instance) and see if that module fails to load? If it does, then we'll know the
problem is specific to the genders.so module, if not, then there is a generic
misc/* module loading problem.

Thanks,
mark 

Original comment by mark.gro...@gmail.com on 20 Apr 2011 at 7:56

• Changed state: Accepted

[GoogleCodeExporter]

BTW, have you tried running

 make check

in the pdsh build tree? That might expose more HP-UX build issues.

mark

Original comment by mark.gro...@gmail.com on 20 Apr 2011 at 7:58

[GoogleCodeExporter]

When I built pdsh with the machines, dshgroups and genders modules, machines 
and dshgroups loaded fine according to pdsh -L and pdsh -V.  The issue does 
appear to be specific to genders.

I uploaded the make check output.  It did not appear able to run the genders 
tests.

Original comment by bwir...@gmail.com on 20 Apr 2011 at 8:21

Attachments:

• makecheck.0420111515

[GoogleCodeExporter]

Ok, looks like there could be a case where a failed dlopen() might not
generate any error message in pdsh. Can you try the attached patch? There will
be lots of extra output, but perhaps we will see the error for genders.{la,so}

As for the testsuite. Egads it appears to be completely broken on HP-UX. I will
study the output further and see if there is anything to do there...

Original comment by mark.gro...@gmail.com on 20 Apr 2011 at 9:09

Attachments:

• pdsh-error-on-lt_dlopen-failure.patch

[GoogleCodeExporter]

I have opened issue 22 for the testsuite failures.

If you wouldn't mind, could you just run the "basic" test by doing the 
following?

(from top level src dir)
 make
 cd tests
 ./t0001-basic.sh

And paste the results into issue 22?

That would be very helpful, thanks!
mark

Original comment by mark.gro...@gmail.com on 20 Apr 2011 at 9:17

[GoogleCodeExporter]

I applied the patch and compiled and the output of pdsh -V is:

[user@wilma bin]$ ./pdsh -V
pdsh@wilma: /tmp/pdsh/pdshbuild/lib/pdsh/sshcmd.a: can't open the module
pdsh@wilma: /tmp/pdsh/pdshbuild/lib/pdsh/genders.so: can't open the module
pdsh@wilma: /tmp/pdsh/pdshbuild/lib/pdsh/genders.la: can't open the module
pdsh@wilma: /tmp/pdsh/pdshbuild/lib/pdsh/genders.a: can't open the module
pdsh-2.25 (+readline)
rcmd modules: ssh
misc modules: (none)

I have attached the configure/make(pdsh.install.16.21) and the make 
check(check.pdsh).  the make check has less error when run as a user and not 
root, a lot less.

Thanks,

Sean

Original comment by iffla...@gmail.com on 20 Apr 2011 at 9:31

Attachments:

• pdsh.install.16.21
• check.pdsh

[GoogleCodeExporter]

Hm, the error

 pdsh@wilma: /tmp/pdsh/pdshbuild/lib/pdsh/genders.so: can't open the module

is sadly not very helpful. It appears that instead of printing errors directly
from the underlying dynamic loader, ltdl is substituting its own very generic
errors.

I wonder if we add BIND_VERBOSE to shl_load() (assuming ltdl is using shl_load)
if we'll get more errors?

Can you try another patch? Thanks for being patient.

Original comment by mark.gro...@gmail.com on 20 Apr 2011 at 10:01

Attachments:

• pdsh-shl_load-verbose.patch

[GoogleCodeExporter]

BTW, what was the last "working" version of pdsh on this platform?

Original comment by mark.gro...@gmail.com on 20 Apr 2011 at 10:05

[GoogleCodeExporter]

The last and current working version on this platform (HPUX 11.23) is pdsh 2.7 
provided in the HP Distributed System Administration Utilities (DSAU).  

We'll get the patch installed today and report back.  Thanks.   

Original comment by bwir...@gmail.com on 21 Apr 2011 at 12:50

[GoogleCodeExporter]

[user@wilma bin]$ ll /usr/local/lib/hpux32/libgenders.so.2
lrwx------   1 root       sys             17 Apr 20 09:18 
/usr/local/lib/hpux32/libgenders.so.2 -> libgenders.so.2.0
[user@wilma bin]$ ./pdsh -V
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/sshcmd.a' is not a 
valid load module: Bad magic number
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/sshcmd.a' is not a 
valid load module: Bad magic number
pdsh@wilma: /home/user/BUILD/pdshbuild/lib/pdsh/sshcmd.a: can't open the module
/usr/lib/hpux32/dld.so: Unable to find library 'libgenders.so.2'.
/usr/lib/hpux32/dld.so: Unable to find library 'libgenders.so.2'.
pdsh@wilma: /home/user/BUILD/pdshbuild/lib/pdsh/genders.so: can't open the 
module
/usr/lib/hpux32/dld.so: Unable to find library 'libgenders.so.2'.
/usr/lib/hpux32/dld.so: Unable to find library 'libgenders.so.2'.
/usr/lib/hpux32/dld.so: Unable to find library 'libgenders.so.2'.
/usr/lib/hpux32/dld.so: Unable to find library 'libgenders.so.2'.
pdsh@wilma: /home/user/BUILD/pdshbuild/lib/pdsh/genders.la: can't open the 
module
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/genders.a' is not 
a valid load module: Bad magic number
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/genders.a' is not 
a valid load module: Bad magic number
pdsh@wilma: /home/user/BUILD/pdshbuild/lib/pdsh/genders.a: can't open the module
pdsh-2.25 (+readline)
rcmd modules: ssh
misc modules: (none)
[user@wilma bin]$

Original comment by iffla...@gmail.com on 21 Apr 2011 at 2:19

[GoogleCodeExporter]

>pdsh@wilma: /home/user/BUILD/pdshbuild/lib/pdsh/genders.so: can't open the 
module
> /usr/lib/hpux32/dld.so: Unable to find library 'libgenders.so.2'.
>  

The permissions on libgenders.so.2 don't look right:
  lrwx------   1 root       sys   

It appears that only root has read permission, such that if you are running
as a normal user you won't be able to open the file. Can you try changing
perms to 755 and see if that fixes the problem?

mark

Original comment by mark.gro...@gmail.com on 21 Apr 2011 at 2:39

[GoogleCodeExporter]

Changing the permissions on libgenders.so.2 make no difference.  However, 
defining the LD_LIBRARY_PATH environment variable with /usr/local/lib/hpux32 
now allows pdsh to load the genders module:

wilma:/home/root/cfengine-3.1.5b3# /u06/pdsh-2.25/bin/pdsh -V
pdsh-2.25 (+readline)
rcmd modules: ssh
misc modules: genders

I do not understand why LD_LIBRARY_PATH needs to be defined since we have built 
other applications that depend on /usr/local/lib/hpux32 and do not seem to have 
problems finding libraries.

The problem we have now is that although pdsh -g with a single attr works (pdsh 
-g linux) a complex query (pdsh -g "linux&&dla")  produces a "no remote hosts 
defined" error.  The nodeattr command works fine with this same query.

Any ideas?

Original comment by bwir...@gmail.com on 21 Apr 2011 at 6:52

[GoogleCodeExporter]

With the verbose output patch installed here is the output with a complex query:

[user@wilma bin]$ ./pdsh -g 'dla&&linux' uptime
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/sshcmd.a' is not a 
valid load module: Bad magic number
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/sshcmd.a' is not a 
valid load module: Bad magic number
pdsh@wilma: /home/admin/user/pdshbuild/lib/pdsh/sshcmd.a: can't open the module
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/genders.a' is not 
a valid load module: Bad magic number
/usr/lib/hpux32/dld.so: '/home/user/BUILD/pdshbuild/lib/pdsh/genders.a' is not 
a valid load module: Bad magic number
pdsh@wilma: /home/user/BUILD/pdshbuild/lib/pdsh/genders.a: can't open the module
/usr/lib/hpux32/dld.so: Invalid NULL library name; shared library pathname 
expected
pdsh@wilma: no remote hosts specified

Even if i specify -F /etc/genders it gives the same results.

Thanks,
Sean

Original comment by iffla...@gmail.com on 21 Apr 2011 at 7:03

[GoogleCodeExporter]

Ok, thanks for getting down to the solution of the libgenders load problem.
This second issue is similar to Issue 1:

The genders module is trying to dlopen() libgenders and see if the 
genders_query()
symbol is supported at runtime (rather than having a compile time check). When 
it
calls dlopen() (or equivalent) pdsh passes NULL as the filename, which on most
implementations is taken to mean "return the global symbol table". Since 
libgenders
is loaded in the global symbol table this works fine. However, on some systems
dlopen(NULL) is not allowed, as we see here. You could test this theory by 
applying
the attached patch, which kind of reverts the initial patch to fix issue 1. 
However,
some systems don't like dlopen("libgenders.so") because they may not have the
base .so link set up (i.e. they only have the libgenders.so.2 link).

I still haven't closed issue 1 because I haven't decided what to do about this.
Most likely I'll move genders_query support detection to a compile-time check,
or just assume presence of genders_query and remove support for older versions
of libgenders.

mark

Original comment by mark.gro...@gmail.com on 21 Apr 2011 at 8:31

Attachments:

• pdsh-genders-dont-dlopen-null.patch

[GoogleCodeExporter]

This issue was updated by revision r1322.

This commit should fix the second of the two issues open against
the genders module on HP-UX.

Original comment by mark.gro...@gmail.com on 22 Apr 2011 at 2:32

[GoogleCodeExporter]

Looks like this was fixed with a combination of the fix applied for issue 1, 
and the use of +b /path/to/genders with the HP-UX linker.

Original comment by mark.gro...@gmail.com on 27 Apr 2011 at 6:16

• Changed state: Fixed

[GoogleCodeExporter]

This fix and the other HP-UX issues you guys brought up were included in
pdsh-2.26 which I just posted to google code. Thanks for all your help
in tracking down the bugs!

Also, I fixed a lot of testsuite issues (re: Issue 22), if you ever have a
chance to retry the testsuite on hp-ux and report back, that would be great!

Original comment by mark.gro...@gmail.com on 2 May 2011 at 1:15