Alteration of tests directory contents in source tree causes rebuild of Hyperion

[srorso]

When any modification of the tests directory in the source tree is made, Hyperion is rebuilt in its entirety on the next make check. The full rebuild may also occur on a make. Issue affects UNIX builds only; Windows is not affected.

[jphartmann]

As far as ls in /tmp and dot in path, I should think that would not be good for anyone, even guest. Point here is that anyone can put a file in /tmp.

[srorso]

The matter I am struggling with is that crypto is located in /crypto/.libs/dyncrypt.so. I think I have it sorted though.

After make install

Directory /usr/local/lib contains shared libraries loaded by the runtime loader, and executables have that path prefixing the Hercules shared libraries to be loaded as part of the initial load of the executable. A loader library load trace (LD_DEBUG=libs) confirms that all such shared libraries are loaded prior to the Hercules main routine in bootstrap.c receiving control.

Directory /usr/local/lib/hercv4 has dynamically loaded shared libraries, including dyncrypt, dyngui, and the device handlers. This directory name may be overridden by the modpath command, the HERCULES_LIB environment variable, or the -p/--dynpath command line options.

Here is a trace of the open and mmap requests for crypto (line numbers added)

1. open("crypto/dyncrypt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2. open("crypto/dyncrypt.la", O_RDONLY)    = -1 ENOENT (No such file or directory)
3. open("/usr/local/lib/hercv4/dyncrypt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4. open("/usr/local/lib/hercv4/dyncrypt.la", O_RDONLY) = 5
5. mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd8d1a4b000
6. open("/usr/local/lib/hercv4/dyncrypt.so", O_RDONLY|O_CLOEXEC) = 5
7. mmap(NULL, 2284872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7fd8ce7f1000

crypto/dyncrypt is coded in hdl.c. The first three attempts by ltdl.c fail. The fourth succeeds, and ldtl.c parses the .la file. The first mmap is just working storage. Line 6 is the open for the .so file, named on the dlname= line in the .la file. The mmap() on 7. maps the instruction space from dyncrypt.so and corresponds to the total size of the two page-aligned LOAD sections displayed by objdump -x of dyncrypt.so.

Before make install

Directory /.libs contains shared libraries loaded by the run time loader at execution start and dynamically loaded device drivers, dynamic instructions & gui, and compression, and this directory is included in the RPATH included in the relinked lt-hercules. Directory /decNumber/.libs contains libdecNumber.so, and that directory is also named in RPATH.

Crypto is in /crypto/.libs/dyncrypt.so, and that directory is not mentioned...much.

Here is a trace of the open and mmap requests for crypto when make install has not been done and Hercules is being run from the build directory:

1. open("crypto/dyncrypt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2. open("crypto/dyncrypt.la", O_RDONLY)    = 5
3. mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8d0dd1000
4. open("crypto/.libs/dyncrypt.so", O_RDONLY|O_CLOEXEC) = 5
5. mmap(NULL, 2284872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7fe8cdb77000

Note that the .la file is found in the ./crypto subdirectory of the build directory on the second attempt, and things proceed from there. It is not clear to me why ltdl.c makes its first try for the .so in the .libs directory. I bet another few hours reading ltdl.c would do the trick.

Back to After make install

Here is where it gets interesting. We shall run Hercules (installed version) while the build directory is the current directory.

open("/usr/local/lib/tls/x86_64/libherc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/libherc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/x86_64/libherc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libherc.so", O_RDONLY|O_CLOEXEC) = 3
mmap(NULL, 9653616, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f0cdebe9000
[...]
open("crypto/dyncrypt", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("crypto/dyncrypt.la", O_RDONLY)    = 5
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3a94acd000
open("crypto/.libs/dyncrypt.so", O_RDONLY|O_CLOEXEC) = 5
mmap(NULL, 2284872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7f3a91873000

Note that libherc.so is retrieved from the install directory (first five lines) but dyncrypt.so is retrieved from the build directory.

Now that I understand how this works at run time, I can turn my attentions to John's original question, namely, why is a compile/link time dependency enforced on a shared library?

[srorso]

Progress...

...why is a compile/link time dependency enforced on a shared library?

Because that is what automake does when object files. libraries (.a) or shared libraries (.so) are included in a prog_LDADD variable. See the GNU Automake manual section on Linking for the following quote:

If prog_DEPENDENCIES is not supplied, it is computed by Automake. The automatically-assigned value is the contents of prog_LDADD, with most configure substitutions, -l, -L, -dlopen and -dlpreopen options removed.

Okay, automake is working as documented even if we do not like the result at the moment. Go back three paragraphs in the same section and see the following:

prog_LDADD is inappropriate for passing program-specific linker flags (except for -l, -L, -dlopen and -dlpreopen). [Emphasis added]

Ah, one can put search libraries in the LDADD variable, as opposed to just linking the shared library with the main executable object file. And GNU linker 2.9 documentation says that the linker will search .so libraries listed in the -l option before .a libraries of the same name:

On systems which support shared libraries, ld may also search for libraries with extensions other than .a. Specifically, on ELF and SunOS systems, ld will search a directory for a library with an extension of .so before searching for one with an extension of .a.

And if a library is included as a search library (-l option), that library is not made a dependency of the program being linked.

So I made the following changes to Makefile.am to add and use a tools_search variable:

if OPTION_DYNAMIC_LOAD
tools_ADDLIBS = $(HERCLIBS2) $(LDADD)
tools_search =  -lhercs     \
                -lhercu     \
                -lherct     \
                -lhercd     \
                -lherc
else
tools_ADDLIBS = $(HERCLIBS2) $(LDADD) hdlmain.o
endif

if BUILD_HERCIFC
hercifc_SOURCES     = hercifc.c
hercifc_LDADD        = $(tools_ADDLIBS)
hercifc_LDFLAGS  = $(tools_LD_FLAGS)
endif

dasdinit_SOURCES     = dasdinit.c
dasdinit_LDADD   = $(tools_search)
dasdinit_LDFLAGS     = $(tools_LD_FLAGS)

Hercules and utilities built ok on Debian 8.6, make check worked, and utilities that used $(tools_search) operated at least to generate the startup and help information. A review of the generated Makefile showed that those utilities that included the shared libraries as search libraries did not include a dependency on the libraries in $(tools_search).

Touch version.c followed by a make check rebuilt only those utilities not changed to $(tools_search).

I am not certain how this will work on other platforms, particularly Mac. But Solaris (successor to SunOS), BSD systems, and other modern (ELF) GNU/Linux systems should be fine. And Mac's status as a successor to a BSD system may work.

And in retrospect, I should have looked at the other contents of $(LDADD) to determine whether they should be included in the executable or searched. That shall be phase two.

Best Regards, Steve Orso

[srorso]

A less invasive way to do this, likely with less of an impact on portability, would be to explicitly specify the dependencies for each utility rather than moving the shared libraries from the link to search libraries of the link, thus:

dasdinit_SOURCES     = dasdinit.c
dasdinit_LDADD   =  $(tools_ADDLIBS)
dasdinit_LDFLAGS     = $(tools_LD_FLAGS)
dasdinit_DEPENDENCIES   =

The issue of dependencies other than those detected automatically by automake/make remains to be investigated.

I suppose getting the tabs to line up is a hopeless task.