Alteration of tests directory contents in source tree causes rebuild of Hyperion

[srorso]

When any modification of the tests directory in the source tree is made, Hyperion is rebuilt in its entirety on the next make check. The full rebuild may also occur on a make. Issue affects UNIX builds only; Windows is not affected.

[srorso]

Apparently, when make encounters a subdirectory with no Makefile, make assumes that if anything has changed in that subdirectory, everything in the parent (current level) directory must be rebuilt.

While the tests subdirectory has a Makefile.am, autoconfigure/automake never creates a Makefile for that directory. If you build out of source and make any change in the tests directory (rename a test, edit one, create a new one), Hyperion gets completely re-built.

To get tests/Makefile created from tests/Makefile.am, one must add tests/Makefile to the AC_OUTPUT statement at the end of configure.ac, and that's the correction for this issue.

The result is a tests directory in the build tree that contains a single file, the Makefile. And no rebuild of Hyperion if one changes a test in the source tree.

The tests directory was created by commit 821e565 on 17 Dec 2005, and a tests/Makefile.am was created at that time. Configure.ac, which had been initially created in 2001 and updated from time to time, was not updated in the same commit to create a Makefile from Makefile.am. Spot checks of commits affecting configure.ac failed to find one where tests/Makefile was added. John's commit 1bd401f added it today.

In addition, the June 2016 commit ba647a0 introduced an error in the tests/Makefile.am (trailing \ on the last line of the file). This causes autogen.sh to abort...if it processes tests/Makefile.am. The fact that people have been doing autogen.sh since then without encountering this abort demonstrates that tests/Makefile.am was not part of the build.

I thank John for testing this correction on his system and for making the commit.

[jphartmann]

Now that there is a tests/Makefile, would it not install a bunch of .core files on make install? I think that is what happens with the first variable being set. I have no idea what happens to the EXTRA_DIST files.

Fish said that all of this was required to build the .zip file, but as there was no Makefile to do it, that must be another piece of baloony?

The tests directory is for testing before installing; I don't think anything from this directory should be installed.

[srorso]

Good point.  I will test....

[srorso]

The .core files are installed in the same place as the html files. Certainly not needed.

Make dist fails immediately with:

srorso@debian64:~/Hercules/x86_64/hyperion$ make dist
make  dist-gzip am__post_remove_distdir='@:'
make[1]: Entering directory '/home/srorso/Hercules/x86_64/hyperion'
make[1]: *** No rule to make target 'bldlvlck', needed by 'distdir'.  Stop.
make[1]: Leaving directory '/home/srorso/Hercules/x86_64/hyperion'
Makefile:2448: recipe for target 'dist' failed
make: *** [dist] Error 2

So some other work would be needed to get make dist functional before we can consider the impact of the variables in tests/Makefile.am. No one has raised the issue, so make dist gets a low priority for now. But I believe Fish was correct in stating that the EXTRA_DIST value in tests/Makefile.am is necessary to get the tests directory contents included in the dist tarball.

I believe that tests/Makefile should not be committed to the repo; it is recreated each time autogen.sh is run. I also think that Makefile.am in the top level Hyperion directory needs tests added to the SUBDIRS variable, after crypto (line 17).

Some additional testing, then I will commit an update to tests/Makefile.am and Makefile.am

[jphartmann]

Steve, Makefile is not the same as makefile.

[srorso]

True enough; Debian/CentOS catch me on this too from time to time. Post updated.

Movement of the .core files to EXTRA_DIST generates a much shorter Makefile (almost did it again). It just contains the .core file build rule and related comments. The change also eliminates the copy of the .core files to /usr/local/share/hercules/.

Commit tomorrow.

[jphartmann]

I think more is needed. I still get everything compiled. I did create a symbolic link in .libs to the crypto module.

[srorso]

Hi John:

Thank you for testing. I will continue to look at this.

Best regards, Steve Orso

+1 610 217 7050

On Dec 3, 2016, at 8:23 AM, John P. Hartmann notifications@github.com wrote:

I think more is needed. I still get everything compiled. I did create a symbolic link in .libs to the crypto module.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[jphartmann]

Let me clarify. I did get everything compiled once; it is not every time I add a test case.

[jphartmann]

It looks like I no longer can reproduce after Steve's latest commit. But let us leave this open for a while.

[jphartmann]

I committed a change to dyncrypt and now it recompiles everything.

[jphartmann]

I think the culprit is _dynamic _version. When it detects a change in commit it will touch version.h (in the source directory!) and that will cause a rebuild. Another FishFest.

[srorso]

Hi John:

You’re right….and there are some subtleties that explain the intermittent behavior…..In addition to a commit changing the version string, consider the following sequence.

After a make/make check (and all the stuff that goes before it) on a clean newly cloned local repository, try:

1) Touch ieee.c, then make check.

Everything dependent on ieee.c gets built. That’s a lot of things but it is not the whole of Hercules. Makes sense. Apparently make uses timestamps to determine if things have been modified. Git does not, and therein hangs a tale.

2) Add some whitespace to ieee.c, then make check. Everything gets built. Well, everything that uses version.h. Make skips decNumber.

Because git uses diffs, not timestamps, it does not treat the repository as having been modified when ieee.c is touched. Adding whitespace leads git to believe the repo has been modified. That leads _dynamic_version to append -modified to the build string, and that triggers a complete rebuild.

3) Remove the whitespace added to ieee.c in 2) , then make check. Everything gets built.

Because the repo has been restored to its unmodified state, it is no longer “modified” and that string is removed from the version. Version.h gets changed again, and everything is rebuilt.

4) Add the whitespace back to ieee.c, then make check to set up the next test. Sorry, everything gets rebuilt.

5) Add more whitespace to ieee.c, then make check. Only the things dependent on ieee.c are rebuilt.

The repository is modified in 4), and remains modified in 5). So Version.h is not changed. Because “-modified” is already part of the commit string; additional repo changes do not poison version.h. Unless of course the net effect of the additional changes are to restore the repo to the unmodified state.

I recall arguing (in the best sense of the word) that there was value in having an indication of a modified repository in the Hercules version string. While I still feel that way, the price for that value is too high and the ‘modified’ string should be, at least for the moment, abandoned.

My argument extends only as far as “Hercules has been modified.” And git a) likely does not give us a tool to limit the modified test to only those things in the repo that interest us (documentation and test case changes should not trigger a build), and b) rebuilding Hercules instead of doing an incremental build is a pretty high price for the value of “modified” in the version string.

I suppose one could modify _dynamic_version to parse the list of changed files generated by git diff-index HEAD to eliminate false positives created by, say, documentation changes. Git diff-index does allow specification of paths, which limits the test to only files matching the paths, thus:

git diff-index –quiet HEAD .c .h decNumber/.c decNumber/.h

And so on. But that introduces the opportunity for false negatives, which are far worse than false positives.

One could also separate the version string printed by Hercules at start-up be separated from the module eye-catchers, and only the printed version string module should be messed with by the successor to _dynamic_version. But I will suggest that such activities be delegated to the Department of Redundancy Department and that, having delegated, we can move on to more fruitful pursuits.

This issue is much less likely to be reported in Windows even though _dynamic_version.cmd uses the same tests to determine if a repository has been modified. In Windows, running a full regression test, the equivalent of make check, is not done using nmake. As a result, the repository is not tested for modifications before running the full regression test.

John: Excellent catch!!!

Best Regards,

Steve Orso

From: John P. Hartmann [mailto:notifications@github.com] Sent: Thursday, December 29, 2016 4:03 AM To: hercules-390/hyperion hyperion@noreply.github.com Cc: Stephen Orso stephen.orso@yahoo.com; State change state_change@noreply.github.com Subject: Re: [hercules-390/hyperion] Alteration of tests directory contents in source tree causes rebuild of Hyperion (#168)

I think the culprit is _dynamic _version. When it detects a change in commit it will touch version.h (in the source directory!) and that will cause a rebuild. Another FishFest.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/168#issuecomment-269600629 , or mute the thread https://github.com/notifications/unsubscribe-auth/AQWuNkdrijchlKZcq3rvI_PmjACW4-rgks5rM3dIgaJpZM4LBdwS . https://github.com/notifications/beacon/AQWuNjT-5egr7eH4_jYRz3RLpyZsGabZks5rM3dIgaJpZM4LBdwS.gif

[jphartmann]

The git messages now include count of add/delete/modify files, as well as the number of untracked files.

We still have the blind rebuilding of the device managers: CC version.lo CCLD libhercu.la CCLD libherct.la CCLD libhercd.la

I'm not going to do anything about this until Enrico falls under a #9 bus.

[srorso]

The issue here is that we are interested in identifying changes to Hercules code as updates to commitinfo.h/version.c. Those changes have the potential for changing the way Hercules works and are relevant to diagnostic efforts. The tools at hand, though, identify changes to the repository, which includes lots of things that are not code. So updates to anything in the tests directory (or readmes, or anything else non-code), causes git to record the change as a modified or untracked file. That in turn updates commitinfo.h, which in turn triggers a rebuild of version.c and a re-link of all shared modules that reference version.c.

This issue is seen mostly with changes to tests directory contents because those changes are often followed by make check, but I have been able to reproduce this issue on a Debian system by changing a file in the directory msvc.makefile.includes. That directory's contents have no bearing on the open source build process.

I am not sure a change of build tools will help here. We need to narrow the scope of the checks performed using available git tools to look at modifications to Hercules code versus the entire repository. If that is possible.

[srorso]

Perhaps, rather than counting all changed files in the repository, GetGitHash and GetGitHash.cmd could be changed to count only changed c source files and headers. This would be relatively easy to do, and it means that changes to documentation, shell scripts, test cases, and such would not trigger a rebuild/relink, and the version.c/commitinfo.h pair would reflect the status of the files that actually turn into Hercules executables.

[jphartmann]

All executable utilitites have a dependency on these files:

HERCLIBS2 = \
   libhercs.la \
   libhercu.la \
   libherct.la \
   libhercd.la \
   libherc.la   

This is correct for static linking, but for dynamic libraries.

Dynamic library dependencies are via the .h file that defines the interface. Only when that file changes must the modules be rebuilt.

This unnecessary link step takes about fifteen minutes on an Atom processor, so it is something I would like to avoid, now that _dynamic_version has been shot down.

[jphartmann]

As for filtering changed files, I guess one could restrict it to the directories that contain executables. I'm not particularly keen on specific file types (extensions); I'm not sure what this list should be.

Take Makefile.am. I should think that any modification to this file should trigger a flag.

[srorso]

Hi John:

Because the header dependencies are handled automagically by the makefile generated by autotools, it should be possible to address this, at least for the executable utilities, by excluding $HERCLIBS2 from the dependency list for those utilities when dynamic libraries are being built.

I'll give it a try...I'm doing lots of builds anyway.

[srorso]

It appears the fact that the libherc*.la shared libraries are implicitly loaded by the executable utilities creates an unavoidable link-time dependency between the executable utilities and the required shared libraries. If the utilities issued explicit dlopen calls and created a table of the functions in each required shared library, the link-time dependency could be eliminated.

Not all of the executable utilities require all of the shared libraries, so with some effort it might be possible to reduce the number of rebuilt utilities. Or maybe not, so this does not appear to be a fruitful path.

Windows has conceptually identical capabilities for shared libraries with respect to implicit and explicit loading.

The device handlers (hdt*.la), crypto, compression, GUI, and dynamic instruction support appear to be the only Hercules shared libraries that are explicity loaded.

I am using the following resources for reference information; please feel free to pass along better sources:

libtools: https://www.gnu.org/software/libtool/manual/html_node/Dlpreopening.html#Dlpreopening Windows: https://msdn.microsoft.com/en-us/library/9yd93633.aspx

It appears that excluding directories from the scope of modified files detected by GetGitHash and its Windows counterpart may be the most direct approach. This would take maybe 3 lines of change in GetGitHash and perhaps 10-15 in the Windows script. I shall propose exclusion of the following directories:

tests/ man/ html/ scripts/ utils/

The remaining directories in the repository either contain code that gets built or scripts that build code.

I am testing the open source changes now (easy to add/remove directories), and will commit with a collection of other build changes to eliminate autogen.sh warnings.

[jphartmann]

I hate to contradict someone who expresses himself with the aplomb of a bishop.

[/usr/data/src/hercules/x86_64/hyperion] file .libs/libhercu.*
.libs/libhercu.la:  symbolic link to `../libhercu.la' 
.libs/libhercu.lai: libtool library file, 
.libs/libhercu.so:  ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=2220093ff9e50b243d7edc354dba5819df3e6f53, not stripped

.so is the library. .la and '.lai` are shell scripts to link or relink the library.

The only unusual thing is that objdump -T dasdcat does not show the name of the library referenced for the functions in the Hercules library; and I cannot see any section that looks like one where autolib could stash the information about the name of the library.

dlopen() is mandated by POSIX.1-2001. What you quote is related to systems that aren't POSIX.

On a modern UNIX, there is no need to relink anything when using a shared/dynamic object. dlopen() does its thing and there is nothing libtool can do about it.

[jphartmann]

Using the strings command on dasdcat in the build directory I can see the string /home/john/lib, but for the libraries I see no path, which is the way it is supposed to be; and you confirm that with objdump -x.

In the normal way of things one would add a non-standard path to /etc/ld/ld.so.conf; not something that is particularly attractive to my thinking since that is system-wide. However, the performance issue of searching all of these (there is a plethora of library directories nowadays) is addressed by ld.so.cache.

An alternative is the LD_LIBRARY_PATH environment variable to specify a non-standard location.

The information about unresolved entries that must be resolved at run time are added to the executable by ld, not the compiler. Perhaps you meant build time.

Anyhow my point remains: There is no need to relink anything if a .so file is rebuilt unless a recompile has been triggered by an interface change, which is a .h file dependency; make handles this very well on its own.

[ivan-w]

On 3/19/2017 9:39 PM, John P. Hartmann wrote:

Using the |strings| command on |dasdcat| in the build directory I can see the string |/home/john/lib|, but for the libraries I see no path, which is the way it is supposed to be; and you confirm that with |objdump -x|.

It's a one time thing that only happens when exec() is invoked. Maybe a few microseconds or so on a modern system... ONCE !

In the normal way of things one would add a non-standard path to |/etc/ld/ld.so.conf|; not something that is particularly attractive to my thinking since that is system-wide. However, the performance issue of searching all of these (there is a plethora of library directories nowadays) is addressed by |ld.so.cache|.

Yes.. The issue is that it requires root acccess and that yes, it is system wide. It is not suitable when, like me you may have several tens, hundreds or thousands different versions of hercules installed (think git bisect).

/etc/ld.so.conf (or whatever variant exists for you distribution) is really used for system installed utilities... especially those that tend to be invoked repetitively by shell scripts or whatnot (think ls, cat, etc..).

An alternative is the |LD_LIBRARY_PATH| environment variable to specify a non-standard location.

Which has a great number of pitfalls, including that those environment variables get dropped during execution of SETUID executables (for obvious security reasons), and it requires non-portable wrappers to execute the programs (which is something done when you execute hercules from the build directory - and is portable because it is built by libtool which has a understanding of the underlying OS and architecture).

The information about unresolved entries that must be resolved at run time are added to the executable by |ld|, not the compiler. Perhaps you meant build time.

I was talking about runtime via dlopen(), NOT execution time via late loading/linkediting done by ld.so. This requires a recompile because of how dlopen() is used under hercules (it specifies an absolute path constructed from the --prefix flag passed to configure).

Anyhow my point remains: There is no need to relink anything if a |.so| file is rebuilt unless a recompile has been triggered by an interface change, which is a |.h| file dependency; make handles this very well on its own.

This would require that the build procedure can guarantee the libraries are ALREADY installed at the expected location (what is in configure --prefix=)... which it cannot do at this point.

I understand your concern (you have a slow system which takes an inordinate amount of time to relink hercules and you'd like a selective build).

The only solution (I see) would be to have the various libraries as separate projects (or maybe subprojects) in order to only rebuild the part/subpart affected by a change (but the dependency tree would be extremely complex to create).

Unfortunately, although those libraries MAY have a somewhat stable external interface (API), their inner working are VERY dependent on the internal hercules structures, and there is currently no versioning system implemented.

--Ivan

[srorso]

Hi John,

I hate to contradict someone ...

I am grateful you have. I hate to be wrong, and when I am, I value highly learning of it quickly. So thanks.

In the meantime, I shall commit the GetGitHash(.cmd) changes to address your pain point. Not perfect, perhaps, but I hope a step forward.

And thanks to both you and Ivan for the additional information...It is nice to have a puzzle.

Best Regards, Steve Orso

[jphartmann]

With respect, Ivan, I don't think you have realised that behind the libtool obfuscation there are some very real and working Linux objects that do not need to be installed or in any other way messed with.

The following shows that dasdcat in the top build directory is indeed a shell script to do a link, but if you execute the real executable in .libs, it works right out of the box once you tell it where to find the libraries. This is the first time I ever used that program; there cannot have been any behind-my-back executable.

[/usr/data/src/hercules/x86_64/hyperion] file dasdcat
dasdcat: Bourne-Again shell script, ASCII text executable, with very long lines
[/usr/data/src/hercules/x86_64/hyperion] .libs/dasdcat
.libs/dasdcat: error while loading shared libraries: libhercs.so: cannot open shared object file: No such file or directory
[/usr/data/src/hercules/x86_64/hyperion] LD_LIBRARY_PATH=.libs .libs/dasdcat
HHC01413I Hercules utility dasdcat - DASD cat program; version 4.0.0 (4.0.0.8728)

The only executable that needs to be run, once as you say, in the top build directory is hercules itself. For some reason there is no hercules executable in .libs.

I never install Hercules. This prevents me from picking up stale libraries.

For building and testing of the tests suite I run out of a directory in my own source tree. Crypto was a problem since it lives in crypto/.libs and the load code does not inspect LD_LIBRARY_PATH, but I could overcome it thus (before Steve's change to have an architecture level):

[/home/john/src/hyperion] more onetest
#!/bin/sh
./runtest -f $1 -d . \
        -p /usr/data/src/hercules/dbg/.libs \
        -h /usr/data/src/hercules/dbg/ \
        -l /usr/data/src/hercules/dbg/crypto/.libs/dyncrypt.so

So I repeat: libtool is a simple pass-through to the operating system's facilities. Automake/autoconf is what knows how to build a shared object on a POSIX compliant UNIX.

[ivan-w]

On 3/20/2017 3:38 PM, John P. Hartmann wrote:

With respect, Ivan, I don't think you have realised that behind the |libtool| obfuscation there are some very real and working Linux objects that do not need to be installed or in any other way messed with.

The following shows that |dasdcat| in the top build directory is indeed a shell script to do a link, but if you execute the real executable in .libs, it works right out of the box once you tell it where to find the libraries. This is the first time I ever used that program; there cannot have been any behind-my-back executable.

|[/usr/data/src/hercules/x86_64/hyperion] file dasdcat dasdcat: Bourne-Again shell script, ASCII text executable, with very long lines [/usr/data/src/hercules/x86_64/hyperion] .libs/dasdcat .libs/dasdcat: error while loading shared libraries: libhercs.so: cannot open shared object file: No such file or directory [/usr/data/src/hercules/x86_64/hyperion] LD_LIBRARY_PATH=.libs .libs/dasdcat HHC01413I Hercules utility dasdcat - DASD cat program; version 4.0.0 (4.0.0.8728)| |So you are specific on linux and on YOUR specific distribution...

libtool with make install works.... everywhere.

|

||

The only executable that needs to be run, once as you say, in the top build directory is hercules itself. For some reason there is no hercules executable in .libs.

I /never/ install Hercules. This prevents me from picking up stale libraries.

I always install hercules for that particular reason... To ensure I NEVER pick stale libraries ! To leave the specifics of how shared libraries is implemented on such and such unix or linux distribution... Messing with LD_LIBRARY_PATH (or whatever environment variable exists for your target system) is a guaranteed possibility to run into loading the wrong libraries.

For building and testing of the tests suite I run out of a directory in my own source tree. Crypto was a problem since it lives in |crypto/.libs| and the load code does not inspect LD_LIBRARY_PATH, but I could overcome it thus (before Steve's change to have an architecture level):

|[/home/john/src/hyperion] more onetest #!/bin/sh ./runtest -f $1 -d . \ -p /usr/data/src/hercules/dbg/.libs \ -h /usr/data/src/hercules/dbg/ \ -l /usr/data/src/hercules/dbg/crypto/.libs/dyncrypt.so |

So I repeat: |libtool| is a simple pass-through to the operating system's facilities. Automake/autoconf is what knows how to build a shared object on a POSIX compliant UNIX.

I don't think so.. Have you ever try building hercules on say... AIX ? HPUX ? MacOSX ? FreeBSD ? NetBSD ? Hurd [1]? Each of them are posix compliant system, but POSIX does NOT specify how shared libraries are built. Posix only specifies that shared libraries may be loaded with dlopen (and in the case of MacOSX for example, implicitly loaded shared libraries (via execution) and explicitly loaded shared libraries (via runtime dlopen) are different object types). libtool HIDES all these internals.

libtoolize inserts the necessary invocation of libtool in makefile.am with generic terms which are then converted to specific flags, commands, etc that are appropriate for the target environment.

Automake/autoconf does NOT know how to build shared libraries or loadable modules for any specific target. Libtool does (and that is why libtool exists).

--Ivan [1] Ok, that one doesn't work !

[srorso]

Commit e901ecd excludes non-build/non-source directories from the calculation of new/changed files since last commit used to create commitinfo.h. Verified on Windows-10, Debian 8.6, and FreeBSD 11 (with non-GNU sed). (Would that be old sed?)

[ivan-w]

I'll try...

But I personally restart from scratch after every git pull or source modification... so I'm not a very good test subject. (Usually takes less than 2 minutes between make & make install for me)

--Ivan

On 3/20/2017 4:01 PM, Stephen Orso wrote:

Commit e901ecd https://github.com/hercules-390/hyperion/commit/e901ecd5f5e5bb81e2439647ca98f2552107df8c excludes non-build/non-source directories from the calculation of new/changed files since last commit used to create commitinfo.h. Verified on Windows-10, Debian 8.6, and FreeBSD 11 (with non-GNU sed). (Would that be old sed?)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/168#issuecomment-287786530, or mute the thread https://github.com/notifications/unsubscribe-auth/ABjMW-7e3bFLZwuqJRtJxEvpNnymQA3Nks5rnpTLgaJpZM4LBdwS.

[jphartmann]

Steve, FreeBSD sed is derived from the original BSD sed. I don't know where it differs from AT&T sed. But anything that works with BSD sed should also work with GNU sed. All of them should be POSIX compliant, but with various extensions it would appear. You might be able to find the spec at X/open (or however they spell it now) to see what the standard says.

[srorso]

Hi John: I develop open source on GNU/Linux Debian; the FreeBSD test helps me avoid GNU/bash extensions. I've been following along in the conversation between you and Ivan; I'm going to build a Solaris 11 machine to see exactly how they work and differ.

I can now describe how Debian (GNU/Linux) handles shared libraries at run time....the dependency issue is an automake matter.

All of this is very educational.....not sure when Hercules was last built on Solaris, so that might be interesting too. Another opportunity to make mistakes and learn from them.

[jphartmann]

Solaris may be a tad ambitios.

[ivan-w]

On 3/20/2017 6:09 PM, Stephen Orso wrote:

Hi John: I develop open source on GNU/Linux Debian; the FreeBSD test helps me avoid GNU/bash extensions. I've been following along in the conversation between you and Ivan; I'm going to build a Solaris 11 machine to see exactly how they work and differ.

I can now describe how Debian (GNU/Linux) handles shared libraries at run time....the dependency issue is an automake matter.

True... The dependency is... The intricate inner workings of how shared libraries are built, how they are installed and how hints regarding location of the shared libraries or modules isn't.

Therefore I think this is why dependencies in automake is based on a worst case scenario (if in doubt, relink all, just in case, using "libtool install").

Hopefully no library or module has any cross dependency (which isn't supported on all systems/archs anyway) - however to overcome this (something which is needed in some case) is when dlopen may also be used (including dlopen of oneself - allowing some form of interface reflection to overcome module cross dependency).

All of this is very educational.....not sure when Hercules was last built on Solaris, so that might be interesting too. Another opportunity to make mistakes and learn from them.

Thank you Stephen for all the efforts you put into this,

--Ivan

[jphartmann]

This is FreeBSD 10.2. The .la files are not there, but the bootstrap is still there.

[john@FB102 /pool/data/src/hercules/amd64/hyperion]$ find . -name 'dasd
dasdcat      dasdconv     dasdcopy     dasdinit     dasdisup     dasdload     dasdls       dasdpdsu     dasdseq      dasdtab.lo   
dasdcat.o    dasdconv.o   dasdcopy.o   dasdinit.o   dasdisup.o   dasdload.o   dasdls.o     dasdpdsu.o   dasdseq.o    dasdutil.lo  
[john@FB102 /pool/data/src/hercules/amd64/hyperion]$ find . -name 'dasdcat*' -ls
147488       27 -rw-rw-r--    1 john                             john                                12496 Mar 20 21:49 ./.deps/dasdcat.Po
147491       13 -rwxrwxr-x    1 john                             john                                 5390 Mar 20 21:49 ./dasdcat
147489      243 -rw-rw-r--    1 john                             john                               123200 Mar 20 21:49 ./dasdcat.o
147490      145 -rwxrwxr-x    1 john                             john                                72208 Mar 20 21:49 ./.libs/dasdcat
[john@FB102 /pool/data/src/hercules/amd64/hyperion]$ file dasdcat
dasdcat: POSIX shell script, ASCII text executable, with very long lines
[john@FB102 /pool/data/src/hercules/amd64/hyperion]$ file .libs/dasdcat
.libs/dasdcat: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.2, not stripped
[john@FB102 /pool/data/src/hercules/amd64/hyperion]$ LD_LIBRARY_PATH=.libs:decNumber/.libs .libs/dasdcat
HHC01413I Hercules utility dasdcat - DASD cat program; version 4.0.0 (4.0.0.8729)

[srorso]

Here is what happens with a Libtool dasdcat. With apologies if this was obvious to everyone but me.

During make, a shell script is created in the build directory and an executable is created in the .libs subdirectory of the build directory. The executable lists all shared libraries to be loaded when execution starts, but path names are provided only for system shared libraries.

srorso@debian-8-64:~/Hercules/x86_64/hyperion$ ldd .libs/dasdcat
        linux-vdso.so.1 (0x00007ffdb768d000)
        libhercs.so => not found
        libhercu.so => not found
        libherct.so => not found
        libhercd.so => not found
        libherc.so => not found
        libdecNumber.so => not found
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1111ceb000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1111ad0000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f11118b9000)
        libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f11116a1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f11113a0000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f111119c000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f1110f8c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1110d6f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f11109c4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1111ef3000)
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ 

The executable .libs/dasdcat may be run as is so long as a path to the shared objects labeled 'not found' is provided. LD_LIBRARY_PATH does this. And if run from the .libs directory, the shared libraries are also found.

The shell script dasdcat looks for .libs/lt-dasdcat. If found, it is executed. If not, ld is run to create it using the shared libraries .libs. lt-dasdcat and dasdcat are very similar:

srorso@debian-8-64:~/Hercules/x86_64/hyperion$ ls -la dasdcat .libs/*dasdcat
-rwxr-xr-x 1 srorso srorso   5321 Mar 20 09:21 dasdcat
-rwxr-xr-x 1 srorso srorso 534192 Mar 20 09:21 .libs/dasdcat
-rwxr-xr-x 1 srorso srorso 534288 Mar 20 11:27 .libs/lt-dasdcat
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ 

The difference that the shared libraries now all have paths.

srorso@debian-8-64:~/Hercules/x86_64/hyperion$ ldd .libs/lt-dasdcat
        linux-vdso.so.1 (0x00007ffefa281000)
        libhercs.so => /home/srorso/Hercules/x86_64/hyperion/.libs/libhercs.so (0x00007fd5e6a5a000)
        libhercu.so => /home/srorso/Hercules/x86_64/hyperion/.libs/libhercu.so (0x00007fd5e6834000)
        libherct.so => /home/srorso/Hercules/x86_64/hyperion/.libs/libherct.so (0x00007fd5e662c000)
        libhercd.so => /home/srorso/Hercules/x86_64/hyperion/.libs/libhercd.so (0x00007fd5e63e1000)
        libherc.so => /home/srorso/Hercules/x86_64/hyperion/.libs/libherc.so (0x00007fd5e5ab0000)
        libdecNumber.so => /home/srorso/Hercules/x86_64/hyperion/decNumber/.libs/libdecNumber.so (0x00007fd5e589c000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd5e5694000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd5e5479000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fd5e5262000)
        libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007fd5e504a000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd5e4d49000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd5e4b45000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fd5e4935000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd5e4718000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd5e436d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd5e6c6a000)
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ 

Make install also relinks executables that have references to Hercules shared libraries. In this case, the shared library path names point to the install directory for the shared libraries, not the /.libs directory:

srorso@debian-8-64:~/Hercules/x86_64/hyperion$ which dasdcat
/usr/local/bin/dasdcat
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ ldd /usr/local/bin/dasdcat
        linux-vdso.so.1 (0x00007ffd05736000)
        libhercs.so => /usr/local/lib/libhercs.so (0x00007f1cd1520000)
        libhercu.so => /usr/local/lib/libhercu.so (0x00007f1cd12fa000)
        libherct.so => /usr/local/lib/libherct.so (0x00007f1cd10f2000)
        libhercd.so => /usr/local/lib/libhercd.so (0x00007f1cd0ea8000)
        libherc.so => /usr/local/lib/libherc.so (0x00007f1cd0577000)
        libdecNumber.so => /usr/local/lib/libdecNumber.so (0x00007f1cd0363000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1cd015b000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1ccff40000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f1ccfd29000)
        libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f1ccfb11000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1ccf810000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1ccf60c000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f1ccf3fc000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1ccf1df000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1ccee34000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1cd1730000)
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ 

The above examples were created on Debian; FreeBSD is similar. The library versioning support is different.

As time passes, the .libs directory accumulates lt-xxx versions of Hercules and executable utilities.

The dasdcat script link command looks like this:

cd /home/srorso/Hercules/x86_64/hyperion; 
{ test -z \"\${LIBRARY_PATH+set}\" || unset LIBRARY_PATH || { LIBRARY_PATH=; export LIBRARY_PATH; }; }; 
{ test -z \"\${COMPILER_PATH+set}\" || unset COMPILER_PATH || { COMPILER_PATH=; export COMPILER_PATH; }; }; 
{ test -z \"\${GCC_EXEC_PREFIX+set}\" || unset GCC_EXEC_PREFIX || { GCC_EXEC_PREFIX=; export GCC_EXEC_PREFIX; }; }; 
{ test -z \"\${LD_RUN_PATH+set}\" || unset LD_RUN_PATH || { LD_RUN_PATH=; export LD_RUN_PATH; }; }; 
PATH=\"/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games\"; 
export PATH; 
gcc -g -O2 -W -Wall -O2 -g -g3 -ggdb3 -march=native -minline-stringops-dynamically \
    -fomit-frame-pointer -Wl,--warn-common -o \$progdir/\$file dasdcat.o -pthread -pthread  \
    ./.libs/libhercs.so -L/home/srorso/Hercules/x86_64/s3fh/lib \
    ./.libs/libhercu.so \
    ./.libs/libherct.so \
    ./.libs/libhercd.so \
    ./.libs/libherc.so \
    /home/srorso/Hercules/x86_64/hyperion/.libs/libherct.so \
    /home/srorso/Hercules/x86_64/hyperion/.libs/libhercd.so \
    /home/srorso/Hercules/x86_64/hyperion/.libs/libhercu.so \
    /home/srorso/Hercules/x86_64/hyperion/.libs/libhercs.so \
    /home/srorso/Hercules/x86_64/hyperion/decNumber/.libs/libdecNumber.so \
    -lrt -lz -lresolv -lnsl -lm -ldl -lbz2 -lSoftFloat \
    -Wl,--rpath -Wl,/home/srorso/Hercules/x86_64/hyperion/.libs \
    -Wl,--rpath -Wl,/home/srorso/Hercules/x86_64/hyperion/decNumber/.libs \
    -Wl,--rpath -Wl,/usr/local/lib)

$file is set to lt-dasdcat. It is coded as a single line in the script; I have broken it up for readability.

I do see a hercules executable in the .libs directory:

srorso@debian-8-64:~/Hercules/x86_64/hyperion$ ls -la hercules .libs/*hercules
-rwxr-xr-x 1 srorso srorso   5280 Mar 20 09:21 hercules
-rwxr-xr-x 1 srorso srorso 564864 Mar 20 09:21 .libs/hercules
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ 

There are no "not found" shared libraries referenced, so no lt-hercules is generate. How this is detected and dealt with by the wrapper script is at present a mystery to me.

I remain confused about John's runtest example...the -p option tells Hercules where to search for dlopen()'d stuff but does not tell the run time linker how to resolve shared libraries to be loaded upon execution. Is there an LD_LIBRARY_PATH?

And none of this addresses why automake creates a dependency on shared libraries that forces a rebuild of the executable utilities. And it does; examination of the generated Makefile makes that clear. But I need to understand the result before I attempted to understand the process that creates it.

As always, correction of gaps/errors welcomed.

Best Regards, Steve Orso

[jphartmann]

There is an added wrinkle or two.

.libs/dasdcat is created by CCLD, that is, without any assistance from libtool. But of course that aookies to this specific instance, thoug I shall venture that it is the same on all Linuxen.

The runtest invokation starts hercules in the top source dir. This is in fact an autolib wrapper that links stuff in the build directory exactly so that you can run a make check in the build directory. This gets hercules an -rpath in the ./libs directory so we don't need LD_LIBRARY_PATH or its various synonyms.

The dependency rule to relink everything is written by whoever wrote Makefile.am, so it is not "trust me, I know what I am doing" of autotools. It is in my view misguided on a UNIX system.

As for device managers, they do not need relinking as it is the executable's -rpath that is operative and Hercules causes the other libraries to be loaded. When you load a device manager, all its unresolved symbols are defined and the loader does not look for additional libraries.

[ivan-w]

John,

CCLD DOES invoke libtool (try running "make V=1" and watch)....

--Ivan

[ivan-w]

from build directory :

rm dasdcat

make V=1 make[2]: Entering directory '/home/ivan/hercules/build/hy-gcc' /bin/bash ./libtool --tag=CC --mode=link gcc -g -O2 -W -Wall -O2 -O3 -march=native -fomit-frame-pointer -Wl,--warn-common -o dasdcat dasdcat.o libhercs.la libhercu.la libherct.la libhercd.la libherc.la -lrt -lz -lresolv -lnsl -lm -ldl -pthread -lbz2 -L/home/ivan/hercules/build/s3fh/lib -lSoftFloat -lrt -lz -lresolv -lnsl -lm -ldl -pthread -lbz2 -L/home/ivan/hercules/build/s3fh/lib -lSoftFloat gcc -g -O2 -W -Wall -O2 -O3 -march=native -fomit-frame-pointer -Wl,--warn-common -o .libs/dasdcat dasdcat.o -pthread -pthread ./.libs/libhercs.so -L/home/ivan/hercules/build/s3fh/lib ./.libs/libhercu.so ./.libs/libherct.so ./.libs/libhercd.so ./.libs/libherc.so /home/ivan/hercules/build/hy-gcc/.libs/libherct.so /home/ivan/hercules/build/hy-gcc/.libs/libhercd.so /home/ivan/hercules/build/hy-gcc/.libs/libhercu.so /home/ivan/hercules/build/hy-gcc/.libs/libhercs.so /home/ivan/hercules/build/hy-gcc/decNumber/.libs/libdecNumber.so -lrt -lz -lresolv -lnsl -lm -ldl -lbz2 -lSoftFloat -Wl,--rpath -Wl,/home/ivan/hercules/run/hy-gcc/lib creating dasdcat

--Ivan

[jphartmann]

The stuff shown here was configured with --prefix=$HOME

So libtool creates dasdcat and a bog standard gcc creates the install version of libs/dasdcat. The somewhat unconventional way to specify --rpath is only a red herring.

[Ivan, you have to use the browser interface to make any sense of this]

[/usr/data/src/hercules/x86_64/hyperion] file dasdcat
dasdcat: Bourne-Again shell script, ASCII text executable, with very long lines
[/usr/data/src/hercules/x86_64/hyperion] file .libs/dasdcat
.libs/dasdcat: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=a21324b95ee2e4c92468056c05b445cfba71864f, not stripped
[/usr/data/src/hercules/x86_64/hyperion] objdump -x .libs/dasdcat | grep RPATH
34:  RPATH                /home/john/lib
[/usr/data/src/hercules/x86_64/hyperion] ./dasdcat | head -n 5
HHC01413I Hercules utility lt-dasdcat - DASD cat program; version 4.0.0 (4.0.0.8729)
HHC01414I (C) Copyright 1999-2016 by Roger Bowler, Jan Jaeger, and others
HHC01414I Commit e901ecd5f5e5bb81e2439647ca98f2552107df8c.
HHC01415I Build date: Mar 22 2017 at 10:29:37
HHC02405I Usage:
[/usr/data/src/hercules/x86_64/hyperion] objdump -x .libs/dasdcat | grep RPATH
34:  RPATH                /home/john/lib

My interpretation is that the dasdcat script creates a temporary environment that has an runpath of .libs or a temporary executable somewhere, and runs the executable without disturbing .libs/dasdcat. Then make install does not need to relink anything.

[/usr/data/src/hercules/x86_64/hyperion] find . -name hercules -ls
2168753   80 -rwxrwxr-x   1 john     john        81566 Mar 22 10:32 ./.libs/hercules
2168757    8 -rwxrwxr-x   1 john     john         5292 Mar 22 10:32 ./hercules
[/usr/data/src/hercules/x86_64/hyperion] file hercules 
hercules: Bourne-Again shell script, ASCII text executable, with very long lines
[/usr/data/src/hercules/x86_64/hyperion] file .libs/hercules
.libs/hercules: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=c26283dada28476bbed65c494306996a554a37b1, not stripped
[/usr/data/src/hercules/x86_64/hyperion] objdump -x .libs/hercules | grep RPATH
35:  RPATH                /home/john/lib

So to do make check I should invoke hercules in the top build directory; and I should not need LD_LIBRARY_PATH. I need .libs/hercules only when I want to run gdb on hercules.

[srorso]

Regarding the following from yesterday:

I do see a hercules executable in the .libs directory:

srorso@debian-8-64:~/Hercules/x86_64/hyperion$ ls -la hercules .libs/*hercules
-rwxr-xr-x 1 srorso srorso   5280 Mar 20 09:21 hercules
-rwxr-xr-x 1 srorso srorso 564864 Mar 20 09:21 .libs/hercules
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ 

There are no "not found" shared libraries referenced, so no lt-hercules is generate. How this is detected and dealt with by the wrapper script is at present a mystery to me.

Upon reexamination and a comparison of the hercules and wrapper scripts, I see that hercules and dasdcat behave in exactly the same way. Wrapper script links .libs/.o if needed into .libs/lt-, then executes it. .libs/ has "not found" shared libraries and will run ok only if LD_LIBRARY_PATH is set up correctly:

srorso@debian-8-64:~/Hercules/x86_64/hyperion$ .libs/hercules
.libs/hercules: error while loading shared libraries: libherc.so: cannot open shared object file: No such file or directory
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ export LD_LIBRARY_PATH=.libs:decNumber/.libs:crypto/.libs
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ .libs/hercules
HHC01413I Hercules version 4.0.0 (4.0.0.8730)
HHC01414I (C) Copyright 1999-2016 by Roger Bowler, Jan Jaeger, and others
HHC01414I Commit 3514b339f52ed76f6bf2ee99b3bf0b6538e21b32.
HHC01415I Build date: Mar 22 2017 at 10:35:51
[...]
HHC00150I Crypto module loaded (c) Copyright 2003-2016 by Bernard van der Helm
[...]
HHC01603I quit
HHC01420I Begin Hercules shutdown
HHC00101I Thread id 00007f85da5ec700, prio 2147483647, name Processor CP00 ended
HHC01427I Main storage released
[...]
HHC00101I Thread id 00007f85ddb2c740, prio 2147483647, name Control panel ended
HHC00101I Thread id 00007f85da4eb700, prio 2147483647, name Timer ended
srorso@debian-8-64:~/Hercules/x86_64/hyperion$ 

Please accept my apologies for the confusion caused by my earlier post.

[srorso]

I traced execution of the dasdcat script using /bin/basd -x -v dasdcat. It does link dasdcat.o to lt-dasdcat with link options that include -rpath to the .libs directory if the .libs directory does not contain lt-dasdcat.

The relink command in the hercules shell script differs from that in dasdcat in that the hercules command includes decNumber. The hercules script does not include any references to the crypto subdirectory, which leads me to believe that the source has a hard-coded subdirectory path for the dlopen of the crypto module.

[ivan-w]

Stephen,

The location of the modules to be loaded by dlopen() are not 'hard coded' per se..

However, this is compiled in and is defined based on the "prefix" set during the configure run.

The directory where the loadable modules reside can be changed with the hercules "modpath" console command.

--Ivan

[jphartmann]

Yes, the loading of the crypto module is hardwired/automatic. I complained a while back that if you did not use the default location, you could load crypto or the device managers, but not both. Fish fixed that, at least for the locations he could think of, but to use crypto in general, I had to load the module using the -l option with the absolute path on the hercules command, as shown above. I did have to try a lot of ways because the diagnostic did not include any indication of the path hercules tried and failed on.

Ivan, the problem with -p is that it supports one directory path only. Had it been like a normal path variable where directory paths are separated by colon/semicolon, things would have been much different.

[ivan-w]

John,

I think that's fixable !

I must admit, I never gave it much thought since I always go through the src->build->run (where all that pertains to path location and whatnot is not an issue).

However, I understand that this is not ideal (for example for a "make test" which requires running from the build directory).

Of course we run into the issue of how a multi-directory "path" is coded between POSIX and Windows (and the complications it implies).. but it should be manageable.

Then there is also the situation where you have stale modules... (that load, but aren't actually the latest modules). Currently I think it relies on the size of the SYSBLK, but there isn't actually any version control.

I would however like that the version after "make install" be completely standalone (as it currently is) as it allows one to have multiple versions available (and then it's just simply a matter where the PATH environment variable points to).

--Ivan

[jphartmann]

Ivan, no need to fix anything for me, now that I understand.

I shall change make check not to require any special treatment. Running hercules out of the top build directory will do what is desired without further ado if my understanding is correct.

Funnily the handling of windows vs UNIX is taken care of by Python for Harold's ASMA.

How can you have a stale module? If the SYSBLK changes, so will the h file and that will be a dependency of the .h which will trigger regeneration. If module A passes SYSBLK to module B which passes it on to C without using the .h file, this would be a transparent pass-through and should not make module B stale.

After install, you have a stand-alone environment. It will not have any makefille and make check is not applicable.

You install in into a unique path; I build in a separate build directory; eins beer, as the Germans say. Neither of us should need to fiddle with PATH or RUNPATH (LD_LIBRARY_PATH).

[srorso]

Hi Folks:

I have made a new friend: strace -e trace=open,mmap .libs/hercules.

The process for dynamically loaded libraries is a bit more complex than for shared/load at program start libraries, with logic buried in ltdl.c. It will take some time to document, but it's worth it.

And one of the side effects is that if you have done make install and run the installed Hercules from the install target directory while in the build directory (no "./" in front of Hercules, you will still get crypto from the build directory tree, even though the installed crypto is in /usr/local/lib/hercv4 or some such. Whether that's stale or uplevel is phase of the moon dependent.

Best Regards, Steve Orso (aka Bishop of Ussher)

[jphartmann]

Steve, you haven't got . (dot) in you $PATH, have you?

For me, even when the build directory is my present working directory, I must say ./hercules to start it.

[srorso]

Hi John,

No dot in $PATH. In the build directory with no make install, ./ is required. After make install, ./ exec's build version, not installed version.

I edited the post; not sure if I got it before git mailed.

Best regards, Steve Orso

On Mar 23, 2017, at 11:40 AM, John P. Hartmann notifications@github.com wrote:

Steve, you haven't got . (dot) in you $PATH, have you?

For me, even when the build directory is my present working directory, I must say ./hercules to start it.

[wrljet]

I believe a '.' in your path is generally not done because it can lead to confusion. (but I've often added it)

Bill

[ivan-w]

Not having '.' in the path is a security measure.

Think of root going to /tmp (where everyone can write) where someone created an executable file called (for example) 'ls'...

Not good !

--Ivan

[jphartmann]

Aha!

Running dasdcat in the top build directory creates this:

[/usr/data/src/hercules/x86_64/hyperion] objdump -x .libs/lt-dasdcat | grep RPATH
34:  RPATH                /usr/data/src/hercules/x86_64/hyperion/.libs:/usr/data/src/hercules/x86_64/hyperion/decNumber/.libs:/home/john/lib