Hercules intermittently fails to load the dyncrypt.so library at startup.

[srorso]

This issue affects UNIX-like and macOS systems.

When either of the following is true, Hercules will not load dyncrypt.so at startup:

• Hercules is executed from the build directory and the build directory is not the current directory.

• Hercules is executed from an install directory that is different from the installation directory specified at build time. This can occur when the installation directory is moved or copied.

An ldmod command specifying a correct relative or absolute path to dyncrypt.so will work.

The issue occurs because the dynamically loaded module table entry for dyncrypt.so includes a subdirectory and a "/" (crypto/dyncrypt) in the file name. When a file name passed to dlopen() includes a "/", dlopen assumes a path and searches only that path for the file. RPATH and RUNPATH in the calling module, the LD_LIBRARY_PATH variable, and other locations are not searched.

This behavior is required by The Open Group Base Specifications Issue 6 and 7, IEEE Std. 1003.1 2004, 2008, and 2016; see:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html

This issue has been reproduced on Hyperion built with CMake and with GNU autotools.

Function hld_load() in hdl.c looks in four places for a module to be pre-loaded. As soon as the module is found, searching stops. Because the preload entry for dyncrypt includes a slash, RPATH, RUNPATH, LD_LIBRARY_PATH, /usr/lib, and /lib are not searched.

1. Search the current working directory for \
2. Search the current working directory for \.so (Note 1)
3. Search the install prefix specified at build time for \ (Notes 2, 3)
4. Search the install prefix specified at build time for \.so (Notes 1, 2. 3)

Note 1: .so is the appropriate suffix for UNIX-like and macOS systems.

Note 2: The install prefix is suffiixed with 'lib'; an install prefix of /home/srorso/herc causes /home/srorso/herc/lib to be searched.

Note 3: The install prefix directory is searched even when Hercules is run from the build directory. This creates the possibility that Hercules, when run from the build directory, would use a back-level dyncrypt module from a previously populated install directory.

Device driver and other modules are not affected by this issue because their names do not include a "/", with the effect that dlopen() will search RUNPATH/RPATH.

A possible solution is to a) remove the subdirectory from the preload table entry for dyncrypt in hdl.c and b) include crypto in the RPATH/RUNPATH for both build and install.

A simpler and I feel better long term solution would be to just a) remove the subdirectory from the preload table entry for dyncrypt and b) build and install the crypto library in the Hercules library directory. But this introduces a source-build compatibility issue between the GNU-autotools build and the CMake build, and I dread putting in something like #ifdef CMAKEBUILD.

Issues loading crypto have bedeviled me for quite a while. I think I now understand it.

Best Regards, Steve Orso

[ghost]

hdl.c is coded to search for crypto/dyncrypt

mi tests were successful when

1) moved by hand dyncrypt.so to crypto/dyncrypt.so

2) modified hdl.c to search for dyncrypt rather then crypto/dyncrypt as spinhawk does

2) would be my first choice, it will not clobber the lib directory

unfortunately the autotools shit creates a completely different install layout it puts everything in lib/hercules-v4 even if make diagnostic tells differently

so You might all forget about consistency

as far as testing with the autotools attempted a couple of times both with the 1Stop and by hand with the most unpleasant result

it generates STATIC libraries/modules and dyncrypt obviously does not load

I am really disappointed

[srorso]

Hi Enrico,

I share your disappointment over the the state of the GNU-autotools build on macOS. I suspect it has been broken for quite some time. I also suspect that fixing it would mean updating the versions of the autotools components in the autoconf directory in Hyperion, for a start.

I tried that once; it was not pretty. I suspect it would have been harder than converting to CMake and had much less benefit. (See your comments in #216.)

Thanks for documenting the impact of this issue on macOS; I did my testing on Debian 9.3.

A separate update to this issue will outline a possible approach to the matter.

Best Regards, Steve Orso

[srorso]

The workaround to this issue is to install Hercules to a working install directory that is not /usr/local/lib or its equivalent and running Hercules from that directory. But it would be nice to address the root cause of this issue.

I am thinking of the following approach:

• The configure step is changed to create a config.h macro MODRELDIR, which provides the library directory relative to the executable directory. For UNIX-like and macOS systems, this would be ../lib. For Windows, it would be a null string.

• The configure step is changed to put the dyncrypt.so module in the same build directory as the executable and other loadable modules, not in the crypto subdirectory.

• hdl.c is modified as follows:

1. The value of MODRELDIR is appended to the executable's directory (available in sysblk).
2. The subdirectory is removed from the preload table entry for dyncrypt.
3. All searches for loadable modules include the module suffix determined by configure. If a system does not use a suffix, it can set LTDL_SHLIB_EXT to a null string.

• The searches in routine hdl_dlopen() in hdl.c are modified as follows:

1. The first load attempt is for <execdir>/<MODRELDIR>/<modname><LTDL_SHLIB_EXT>. This corresponds to the file placement in an installed system.
2. If MODRELDIR is not a null string, a second load attempt is made seeking <execdir>/<modname><LTDL_SHLIB_EXT>. This corresponds to file placement when running from the build directory.
3. A final attemt is made just using <modname><LTDL_SHLIB_EXT>. This lets the OS loader follow its default search order, including any RPATH/LD_LIBRARY_PATH that might be available on UNIX-like an macOS systems.
4. As now, the search ends when the module is found.

Benefits of this approach:

• The first search attempt is most likely to be successful when Hercules is run from an installation directory. Given current coding of hdl.c, the first search, without a .so/.dll suffix, is highly likely to fail.
• Execution from the build directory when when build directory is not the current directory finds all modules including dyncrypt.so. If the first search fails, as it will when Hercules is run from the build directory, this search is likely to succeed.
• Both of the above searches are direct searches; there is no searching of multiple directories in RPATH/RUNPATH, LD_LIBRARY_PATH, or any other directories included in the OS implementation of dlopen().
• The third search is a fail-safe in that it includes RPATH etc in its search for the module.

Disadvantages

• There are backward compatibility issues associated with the change to the preload table, building Hercules using GNU-autotools, and running from the build directory. GNU-autotools builds dyncrpyt.so in the crypto directory. This could be addressed by coding hdl.c such that it uses the old preload table and the old search order if MODRELDIR is not defined, although I would like a better solution.

Comments and suggestions are welcomed, but for the moment I regard getting CMake documented, running on Windows, and running on macOS to be more urgent endeavors.

[jphartmann]

I've always been in favour of pushing the contents of the crypto stuff up into its parent director, linking it statically into the library, and getting rid of the entire rigmarole.

[ghost]

IMO You are complicating things … I already told and tested that

CMAKE install things properly in hdl.c just get rid of the crypto/ directory specification 1(ONE) line change if You want to keep the ifdef delete 4(FOUR) lines to behave as spinhawk does

retested for the umpteenth time .. installed, renamed the install tree and everything worked

anyway installing hercules in the system WELL KNOWN PATHS will probably bring more problem than advantages

on Freebsd /usr/local is used to install all the NON basic SYSTEM packages ( Please John correct me if I am wrong ) I would never dream of installing Hercules there :-)

cheers

PS to confirm all what was said I am looking at the way clang does but it is almost the same as the way my mods do

I would not worry about backward compatibility

I will be able to test my proposed setup later in the evening on debian, ubuntu, FreeBSD

hat off to the inventor/developer of vagrant

On 14 Mar 2018, at 14:43, Stephen Orso notifications@github.com wrote:

The workaround to this issue is to install Hercules to a working install directory that is not /usr/local/lib or its equivalent and running Hercules from that directory. But it would be nice to address the root cause of this issue.

I am thinking of the following approach:

The configure step is changed to create a config.h macro MODRELDIR, which provides the library directory relative to the executable directory. For UNIX-like and macOS systems, this would be ../lib. For Windows, it would be a null string.

The configure step is changed to put the dyncrypt.so module in the same build directory as the executable and other loadable modules, not in the crypto subdirectory.

hdl.c is modified as follows:

The value of MODRELDIR is appended to the executable's directory (available in sysblk). The subdirectory is removed from the preload table entry for dyncrypt. All searches for loadable modules include the module suffix determined by configure. If a system does not use a suffix, it can set LTDL_SHLIB_EXT to a null string. The searches in routine hdl_dlopen() in hdl.c are modified as follows: The first load attempt is for //. This corresponds to the file placement in an installed system. If MODRELDIR is not a null string, a second load attempt is made seeking /. This corresponds to file placement when running from the build directory. A final attemt is made just using . This lets the OS loader follow its default search order, including any RPATH/LD_LIBRARY_PATH that might be available on UNIX-like an macOS systems. As now, the search ends when the module is found. Benefits of this approach:

The first search attempt is most likely to be successful when Hercules is run from an installation directory. Given current coding of hdl.c, the first search, without a .so/.dll suffix, is highly likely to fail. Execution from the build directory when when build directory is not the current directory finds all modules including dyncrypt.so. If the first search fails, as it will when Hercules is run from the build directory, this search is likely to succeed. Both of the above searches are direct searches; there is no searching of multiple directories in RPATH/RUNPATH, LD_LIBRARY_PATH, or any other directories included in the OS implementation of dlopen(). The third search is a fail-safe in that it includes RPATH etc in its search for the module. Disadvantages

There are backward compatibility issues associated with the change to the preload table, building Hercules using GNU-autotools, and running from the build directory. GNU-autotools builds dyncrpyt.so in the crypto directory. This could be addressed by coding hdl.c such that it uses the old preload table and the old search order if MODRELDIR is not defined, although I would like a better solution. Comments and suggestions are welcomed, but for the moment I regard getting CMake documented, running on Windows, and running on macOS to be more urgent endeavors.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/256#issuecomment-373025025, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUXcKPB9jlw7nn2qvUJ4sJPwWQ3ecqaks5teR6UgaJpZM4So2ZZ.

[jphartmann]

Right Enrico. I installed bash by pkg install and it wound up in:

root@fb111:~ # which bash /usr/local/bin/bash

[srorso]

Hi Enrico,

IMO You are complicating things …

Quite possibly...it would not be the first time. That's why I post here: to get the benefit of other's experience and look at diverse approaches.

My only worry is how a changed hdl.c will work when Hercules is built using GNU-autotools, which builds dyncrypt in a subdirectory. I will want to test on Windows too, but I do not think there will be a problem.

And having hdl.c work the same on all platforms (Windows, UNIX-like, macOS) would be nice. I tire of looking at #ifdef _MSVC_.

Hi John,

Are you proposing any/all of the following:

• Move the sources for dyncrypt into the parent sources directory and just compile them into herc.so
• Build a static library .a or .lib and statically linking it to herc.so
• Leave the sources where they are but compile them into herc.so
• Change configure.ac so it does the same thing as CMake

I'm OK with any of the above, although creation of a static library creates issues when using the Ninja build tool on FreeBSD. And I can do the configure.ac changes too, but I hate configure.ac. :-)

Best Regards, Steve Orso

[jphartmann]

Move the sources for dyncrypt into the parent sources directory and just compile them into herc.so

[ivan-w]

There is a reason dyncrypt is being built as an independent module (and it's not a technical reason). It is a legal reason so that hercules can be built without any dependency on any cryptographic library as required by US export regulations - which may at this point consider cryptographic algorithms to be classified as weapons in the same category as nuclear weapons or nuclear weapon components.

[jphartmann]

In contrast to Ivan I have an export licence for cyptrographic equipment and a vivid memory of what that entails. However since anyone can download the crypto source for hercules it makes no sene to keep the source separate from the rest of the code.

[ivan-w]

I'm not saying it shouldn't be done - I was just giving historical insight as to why dyncrypt existed as a separate component (it's possible at the time that dyncrypt also relied on an external crypto library such as openssl). Since IANAL, I'll let you all decide of the legalese implied by the move. (Although I very much guess it's indeed a moot point nowadays).

[jphartmann]

So why don't you shut up, Ivan. You offer no insight and no solutions.

[ivan-w]

What the heck was that for ?

[ghost]

since the crypto source is distributed with hercules I do not see any reason why it could not be build in a different way

if the legal reasons were that strong then the crypto stuff should not be in the hercules source tree

On 14 Mar 2018, at 19:07, Ivan Warren notifications@github.com wrote:

There is a reason dyncrypt is being built as an independent module (and it's not a technical reason). It is a legal reason so that hercules can be built without any dependency on any cryptographic library as required by US export regulations - which may at this point consider cryptographic algorithms to be classified as weapons in the same category as nuclear weapons or nuclear weapon components.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/256#issuecomment-373120980, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUXcGdiz3ZaXXytMGPD6UPKc20sZQO2ks5teVxwgaJpZM4So2ZZ.

[ivan-w]

When dyncrypt was created, it relied on an external crypto library and didn't contain any crypto code in itself (just provided an interface). It could still be important for those wanting to build and export the binary form without crypto... but hey.. ask John - he knows best

[jphartmann]

Absolutely, Enrico.

[srorso]

Hi folks,

Note that I am not a lawyer, I am summarizing available public information regarding US Export regulations, and I am not providing legal advice.

Here's the current status of US controls on the export of software that implements or includes cryptographic capabilities. Back in the 1990's the situation was much as Ivan describes, but things have been relaxed considerably over the last 15 years.

US Export Control regulations classify stuff by an Export Control Classification Number (ECCN) and then controls exports by ECCN. Encryption software has classification 5D002. There are subclassifications, but those are not relevant for publicly available open source software. See the US ECCN Index here:

https://www.bis.doc.gov/index.php/forms-documents/regulations-docs/13-commerce-control-list-index/file

Perform searches on "Encryption", "5D002", and crypto for a quick summary of what is and is not classified as 5D002.

15 CFR 742.15(b) "Publicly available encryption source code" is the relevant regulation. Find it at:

https://www.ecfr.gov/cgi-bin/text-idx?SID=00a8f54989eaf101a84eff3db59ac6e9&mc=true&node=se15.2.742_115&rgn=div88

That regulation excludes publicly available source code from (US) Export Administration Regulations provided that notification has been provided per 15 CFR 742.15(b)(2) to the (US) Bureau of Industry and Security, a part of the US Department of Commerce, and to the ENC Encryption Request Coordinator.

Export of machine readable codes generated by publicly available encryption source code are also exempt from US Export Control; see the note to 15 CFR 734.3(b)(3). The note appears at the end of 734.3(b)(3)(vi). Find 734.3 at:

https://www.ecfr.gov/cgi-bin/text-idx?rgn=div8&node=15:2.1.3.4.22.0.1.3

Much of the above is a summarization of things discussed at: http://www.magicsplat.com/blog/ear/. There is useful information on that page as well.

[ivan-w]

That looks like (IANAL either) enough information to ensure on a best informed basis to merge the "dyncrypt" module - and have it directly linked inside the hercules engine. Thanks for doing the research ! (We still might want to include some extra stuff in the license file... but without an actual lawyer, it may be just as useless - or even more damaging - than putting some ill formed text).

[srorso]

After further review.....

While moving dyncrypt functionality to be compiled and linked directly into the herc shared library makes sense, there is enough involved in validating the result that it should be undertaken as a separate development activity and not included in the development of the CMake build. Such a move also would either break the current GNU-autotools build or require fairly substantial work adapting it to the moved components. (Substantial = more than I am comfortable doing.) So from my perspective: moving dyncrypt is a good idea best put on the list of things to do.

Current documentation also creates an issue for changing what is searched and in what order. The command prompt help says little, and this is as it should be. HTML documentation at https://hercules-390.github.io/html/hercinst.html#operating echoes the Spinhawk web page:

dyndir
is the directory from which dynamic modules are to be loaded. The default depends on the host platform on which Hercules is being run. This option overrides the default.

The PDF documentation offers much more detail (http://hercdoc.glanzmann.org/V400/HerculesUserReference.pdf) says this about -p dyndir--modpath=dyndir on page 616:

This is the directory from which dynamic modules are to be loaded.The general search order is the following:

1. -p from startup argument.
2. HERCULES_LIB environment variable.
3. MODULESDIR compile time definition
4. The path where Hercules was found and if this is not resolved, then "hercules" is used as pathname.

Substantial changes to the hdl.c dynamic module search order, for me at least, joins moving dyncrypt. A separate test cycle is not really required, but changing the documentation and/or changing functionality that people unknown to us depend on is not something undertaken lightly. It also requires more historical understanding of Hercules' use than I have.

To address the original issue, the following will be done:

• The CMake build will be changed to build dyncrypt as a separate dynamically loaded module in the same build directory as all other dynamically loaded modules. Installation will be to the same directory as the other dynamically loaded modules.

• The file hdl.c will be changed as follows when dyncrypt is not built in the crypto subdirectory. The absence of preprocessor macro HDL_USE_LIBTOOL will be used to determine that dyncypt is not built in the crypto subdirectory. At the moment, only the GNU-autotools build defines this macro. Neither the CMake build nor the legacy Windows build define this macro.

  • hdl.c will search for dyncrypt rather than crypto/dyncrypt.
  • The two suffixless searches for dynamically loaded modules will be removed. (if a target system does not use suffixes on shared libraries, this can be detected at configure time and defining LTDL_SHLIB_EXT to a null string.

The GNU-autotools build will not be changed.

[jphartmann]

Point taken, Steve.

However, on UNIX at least, dyndir specifies a single path. Thus, magic and mirrors are required to load libdyncrypt.so while also being able to load the other device managers.

[ghost]

Did You look at the code ???

it picks up the working directory

strncpy(path,argv[0],sizeof(path)-1);

it might be the right thing to do if somebody was testing but the the docs should be amended

if it wanted to pickup the path of the executable the code sequence should have been

/ APPLE /

include

include

include

include

include

int main() { pid_t pid;

char path[1024];

int RC;
char *cp;

pid = getpid();
RC  = proc_pidpath (pid, path, sizeof(path));
if ( RC <= 0 )
{
    exit(-1);
}
else
{
    cp = strrchr(path,'/');
    *cp=0;
    printf("path '%s'\n",path ) ;
    exit(0);
}
exit(-1);

}

/ FreeBSD /

include

include

int main() { char path[1024]; int size;

size = readlink( "/proc/curproc/file", path, sizeof(path)-1);

path[size] = '\0';

printf("***** '%s'\n", path );

return(0);

}

/ debian /

include

include

int main() { char path[1024]; int size;

size = readlink( "/proc/self/exe", path, sizeof(path)-1);

path[size] = '\0';

printf("***** '%s'\n", path );

return(0);

}

the same sequence I am using on object rexx to find the rexx image in the directory of the excutable

tested on APPLE FreeBSD DEBIAN

On 18 Mar 2018, at 15:04, Stephen Orso notifications@github.com wrote:

This is the directory from which dynamic modules are to be loaded.The general search order is the following:

-p from startup argument. HERCULES_LIB environment variable. MODULESDIR compile time definition The path where Hercules was found and if this is not resolved, then "hercules" is used as pathname.

[ghost]

OOPS … my bad

argv[0] contains the command used to invoke Hercules

it will contain a good executable path only if the FULL path UP to / is used to invoke hercules murky situation if only a partial path is used

On 18 Mar 2018, at 15:04, Stephen Orso notifications@github.com wrote:

This is the directory from which dynamic modules are to be loaded.The general search order is the following:

-p from startup argument. HERCULES_LIB environment variable. MODULESDIR compile time definition The path where Hercules was found and if this is not resolved, then "hercules" is used as pathname.

[jphartmann]

On UNIX in general, argv[0] is simply the first pointer in the array of pointers that is the second argument execve(). It could be anything. The first argument specifies the executable, but that is not available to the program.

int execve(const char filename, char const argv[], char *const envp[]);

[ghost]

I just run a silly test and … at least for something invoked from the command line

include

int main(int argc, char *argv[]) { printf("argv[0] '%s'\n",argv[0]); return(0); }

returned the command used

[enrico@enrico-mbp zTests]$cc zargv0.c [enrico@enrico-mbp zTests]$./a.out argv[0] './a.out' [enrico@enrico-mbp zTests]$../ztests/a.out argv[0] '../ztests/a.out’

the sample posted a while ago returned

[enrico@enrico-mbp zTests]$cc zpidpath.c [enrico@enrico-mbp zTests]$./a.out path '/Users/enrico/zTests'

cheers enrico

On 18 Mar 2018, at 17:26, John P. Hartmann notifications@github.com wrote:

On UNIX in general, argv[0] is simply the first pointer in the array of pointers that is the second argument execve(). It could be anything. The first argument specifies the executable, but that is not available to the program.

int execve(const char filename, char const argv[], char *const envp[]);

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/256#issuecomment-374012888, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUXcNXEgUyu95LRJbB8fDVXmAC5S0_4ks5tfoq0gaJpZM4So2ZZ.

[jphartmann]

Right, Enrico, but that depends on the shell setting up argv[0]. Hercules should not assume that it is invoked from a shell.

This is an essential difference between Windows and UNIX. No way it can be band-aided.

[ghost]

that’ s why I feel that setting the MODULEPATH/whatever_name_You_want_to_call_it is just a waste of time

IMO the percentage of wannabe developers willing to test a module located somewhere else is about -(minus)99% faster anyway to build everything with a different set of dirs

I believe that the best way is to have everything relative to the path of the executable so that the whole thing can be carried around on a USB stick

same thing I am doing with My version of open object rexx

On 18 Mar 2018, at 17:46, John P. Hartmann notifications@github.com wrote:

Right, Enrico, but that depends on the shell setting up argv[0]. Hercules should not assume that it is invoked from a shell.

This is an essential difference between Windows and UNIX. No way it can be band-aided.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/256#issuecomment-374014876, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUXcJGWriH9y3qjTcdM5G42v_oWjKrUks5tfo9jgaJpZM4So2ZZ.

[jphartmann]

Enrico, on UNIX, you simply cannot in general find out the path to the executable.

As an extreme example, i can dlopen a module and then fork() to it. I don't have to use exec().

[ghost]

then the APPLE/Darwin developers are really smart :-)

does FreeBSD have anything like that ?

I thought that the procfs filesystem might have been the solution

e.

On 18 Mar 2018, at 17:57, John P. Hartmann notifications@github.com wrote:

Enrico, on UNIX, you simply cannot in general find out the path to the executable.

[srorso]

Hi John & Enrico,

Many thanks for this discussion; it has been most helpful.

I have suspected that the current coding in hdl.c and configure.ac is a collection of patches in search of a problem, and this discussion and testing confirms that a problem has been found.

I agree now that MODULESDIR is not needed and has the side effect of preventing a binary distribution as an .deb, .pkg, or .rpm because it is set to the install prefix at configure time. CMake will no longer set MODULESDIR. Configure.ac sets MODULESDIR, but does so as a -D flag to the compiler, not in config.h. The current Windows NMake build does not set MODULESDIR but can reliably extract the executable path using GetModuleFileName.

As for the HERCULES_LIB environment variable, well, that might be useful. But a testing scenario could also include rmmod for the current version of the module to be tested followed by ldmod with a full path or modpath. And I believe these could be put in a file referenced by -f.

Hercules assumes that argv[0] points to a useful executable name in impl.c, but the result is used only in console.c (for the Hercules logo file), hsccmd.c (same reason), and hdl.c (as discussed above).

And this raises another question: if the point of all of this is to enable testing, why is HERCULES_LIB / MODULESDIR / argv[0] path search after a default search? The default search goes through the OS standard search order (RPATH etc). Shouldn't the specific search be done first, and given the uselessness of MODULESDIR and the uncertainty of argv[0], shouldn't the specific search be skipped if HERCULES_LIB is not set?

This returns to a history question for me: testing capabilities by themselves do not require the current baroque design, either to setting the fall-back search path or the four searches. If we did not have a document mentioning HERCULES_LIB, I would implement Enrico's very clean solution: one search. Period.

At this point, the plan of attack above seems good, although I should document the plan for setting the search path used if first search fails. HERCULES_LIB, documented as it is, forces two searches.

Best Regards, Steve Orso

[ivan-w]

I agree now that MODULESDIR is not needed and has the side effect of preventing a binary distribution as an .deb, .pkg, or .rpm because it is set to the install prefix at configure time.

Odd, since hercules has been distributed by various distributions (debian redhat for example) using fixed path long after this mechanism was put in place. So creating a .deb, .rpm or whatever is not an issue when the search path for moules is fixed in the executable.

It is also a technique that is being used by countless projects that have been working for ages (think of all the projects that have "plugins" and search for those plugins at specific locations - locations that are specified at compile time).

[ghost]

probably it worked because it was installed to /usr/local

On 18 Mar 2018, at 20:55, Ivan Warren notifications@github.com wrote:

I agree now that MODULESDIR is not needed and has the side effect of preventing a binary distribution as an .deb, .pkg, or .rpm because it is set to the install prefix at configure time.

Odd, since hercules has been distributed by various distributions (debian redhat for example) using fixed path long after this mechanism was put in place. So creating a .deb, .rpm or whatever is not an issue when the search path for moules is fixed in the executable.

It is also a technique that is being used by countless projects that have been working for ages (think of all the projects that have "plugins" and search for those plugins at specific locations - locations that are specified at compile time).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/256#issuecomment-374039236, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUXcNGZQz97m4LwuxcwSrtCRgeNI-p0ks5tfrvEgaJpZM4So2ZZ.

[ivan-w]

I don't think the debian package builder (for example) installs anything in /usr/local !

[ivan-w]

/usr/lib/hercules/libhercd.so /usr/lib/hercules/hdt3705.la /usr/lib/hercules/libherct.so /usr/bin /usr/bin/dasdconv /usr/bin/hetmap /usr/bin/dasdcat That's an example of "dpkg -L hercules" from a debian 9 stretech system. (and looked - NOTHING in /usr/local)

[ghost]

I wanted just point out the logic … if it applies to /usr/local it applies also to /usr

I wonder why they put it in /usr, it is against the logic of not messing around with /usr

even all the packages of the GNU of things are installed by default in /usr/local and I thought that it would have been wiser to apply the same to Hercules

and keeping separate the Libraries and the Modules is reasonable for an installation in the WELL known paths /usr /usr/local and for suse IIRC also /opt /opt/local if hercules is installed in a dedicated path no reason to keep them separate

On 18 Mar 2018, at 21:23, Ivan Warren notifications@github.com wrote:

/usr/lib/hercules/libhercd.so /usr/lib/hercules/hdt3705.la /usr/lib/hercules/libherct.so /usr/bin /usr/bin/dasdconv /usr/bin/hetmap /usr/bin/dasdcat That's an example of "dpkg -L hercules" from a debian 9 stretech system. (and looked - NOTHING in /usr/local)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/256#issuecomment-374041968, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUXcAhPMGX5ikeR59Oii7fKxyVWpZUTks5tfsJNgaJpZM4So2ZZ.

[srorso]

Hi Ivan,

Do any of the distros you mention include Hyperion? Or are these Spinhawk packages?

Best Regards, Steve Orso

[srorso]

Hi Ivan,

If the Debian packager set the installation prefix to match how the .deb was set up to install the package, there is no issue. Dyncrypt will be findable.

Other loadable modules are not affected by this issue because they are loaded by filename without a subdirectory prefix, and both Spinhawk and Hyperion include an attempt to load whatever is in the preload table (e.g., hdteq) without the MODULESDIR prefix in hdl_dlopen. Without a prefix or subdirectory in the name passed to dlopen, the OS Loader will use RPATH/RUNPATH, which are presumably set correctly by the package, LD_LIBRARY_PATH, and, at least for some systems, /usr/lib and /lib. So this issue only affects dyncrypt, which includes a subdirectory prefix. It's also worth noting that the crypto subdirectory is not included in the RUNPATH set up by the GNU-autotools build process.

In most circumstances, dyncrypt loads just fine: a) run from (non-relocated) install directory, b) run from build directory while that directory is also the current directory.

In most circumstances where dyncrypt does not load, there are no overt symptoms. If one does not use crypto-based instructions, there are no symptoms. Using 370 mode without S37x extensions, no symptoms. One does not get the message security assist activation message, but that is easy to miss.

If you use RUNTEST when Hyperion is built using CMake and a multi-configuration build tool (Visual Studio or Xcode), RUNTEST will report crypto failures because the build tree directory structure differs from that created when a single-configuration build tool is used (GNU Makefiles, Makefiles JOM, Ninja, BSD Makefiles, Windows NMake).

In other words, the stars really have to align for this issue to be visible.

The question "how is this working now?" is excellent, because if that question cannot be answered, then the issue is not fully understood. If I had a nickle for each time I asked myself that about this issue, I could retire (wait a minute....oh...well, I could return to aviation as a hobby).

Best Regards, Steve Orso

[jphartmann]

As I understand it, make install moves dyncrypt to the install library along with the other device managers; the issue is more a matter of testing before installation, or as Ivan and I prefer, running directly out of the build directory.

[srorso]

Hi John,

You are correct, make install links all shared libraries and loadable modules into /lib. No crypto subdirectory is used. This leads to a requirement for MODULESDIR and creates problems when MODULESDIR does not reflect the current install directory.

Here is what happens in hdl_dlopen() when dyncrypt is loaded when Hyperion is run from its install directory. Assume for this example that /usr/local/ is the install prefix, with bin and lib used as is common. The following reflects either GNU-autotools including 1Stop or the currently committed CMake configure process. MODULESDIR is set to /usr/local/lib/Hercules-V4, wher e dyncrypt.so was placed by make install. (Note: CMake does not include the Hercules-V4 suffix on the install prefix.)

1. Search for crypto/dyncrypt Fails: no .so.
2. Search for crypto/dyncrypt.so Fails, crypto is not a subdirectory of the current working directory. Per Open group, RPATH/RUNPATH and other directories are not searched.
3. Search for /usr/local/lib/Hercules-V4/crypto (basename, no suffix) Fails: no .so.
4. Search for /usr/local/lib/Hercules-V4/crypto.so (basename, with suffix) Succeeds.

When running from the build directory as the current working directory (CWD), search nr. 2 succeeds.

When MODULESDIR no longer reflects the installation library directory, as would happen when the install directory has been relocated, all four searches fail. (I really like Enrico's goal of a relocatable installation tree. CMake does this. It enables a generic Linux installable, although I am sure CMake does a bunch of other stuff to enable binary Linux portability.)

When running from the build directory when the CWD is not the build directory, search nr. 4 will succeed if there is a previously-installed Hyperion in /usr/local/lib/Hercules-V4, or all four searches will fail if not.

When building using macOS Xcode, a multi-configuration generator, and running from the build directory as the CWD, all four searches fail because crypto is not a subdirectory of the build directory. Here's the build tree for a multi-configuration generator:

<build_dir>\                     # build directory for Hyperion
<build_dir>\Release              # contains almost all executables, libraries, and modules.
<build_dir>\crypto               # build directory for crypto source directory contents
<build_dir>\crypto\Release       # contains dyncrypt.so

The failures of runtest when using a multi-configuration generator on a macOS made me dig into this.

The issue does not happen on Windows with a multi-configuration generator because the NMake build does not set MODULESDIR and searches for dyncrypt, not crypto/dyncrypt . The Windows loadable module search is like dlopen() in that a \ stops searches of "extra" directories. Without a \, the executable's location is searched, and search 2 succeeds. The Windows NMake puts dyncrypt.dll in the same build directory and the same install directory as hercules.exe; CMake was coded the same way.

Preliminary testing of coding in hdl.c and CMake scripts shows that this issue can be addressed. I need to do a full cycle of functional and regression testing.

Best Regards, Steve Orso

[ghost]

unless I am blatantly wrong … there is no relation between the build tree structure and the install tree structure

for all the .so modules I imagine a coding like….

if the install prefix is one of the WELL KNOWN PATHS - /usr /usr/local /opt /opt/local ( the opt things are for SUSE IIRC ) then install to lib/Hercules or whatever else install to lib

the testing from the build tree should be unaffected

the CMAKE invocation of the install_name_tool does not really relink anything it just patches the installed things with the proper library

e.

I am rebuilding my environment to make it more portable and I will be able to test in a couple of days

On 19 Mar 2018, at 16:11, Stephen Orso notifications@github.com wrote:

Hi John,

You are correct, make install links all shared libraries and loadable modules into /lib. No crypto subdirectory is used. This leads to a requirement for MODULESDIR and creates problems when MODULESDIR does not reflect the current install directory.

Here is what happens in hdl_dlopen() when dyncrypt is loaded when Hyperion is run from its install directory. Assume for this example that /usr/local/ is the install prefix, with bin and lib used as is common. The following reflects either GNU-autotools including 1Stop or the currently committed CMake configure process. MODULESDIR is set to /usr/local/lib/Hercules-V4, wher e dyncrypt.so was placed by make install. (Note: CMake does not include the Hercules-V4 suffix on the install prefix.)

Search for crypto/dyncrypt Fails: no .so. Search for crypto/dyncrypt.so Fails, crypto is not a subdirectory of the current working directory. Per Open group, RPATH/RUNPATH and other directories are not searched. Search for /usr/local/lib/Hercules-V4/crypto (basename, no suffix) Fails: no .so. Search for /usr/local/lib/Hercules-V4/crypto.so (basename, with suffix) Succeeds. When running from the build directory as the current working directory (CWD), search nr. 2 succeeds.

When MODULESDIR no longer reflects the installation library directory, as would happen when the install directory has been relocated, all four searches fail. (I really like Enrico's goal of a relocatable installation tree. CMake does this. It enables a generic Linux installable, although I am sure CMake does a bunch of other stuff to enable binary Linux portability.)

When running from the build directory when the CWD is not the build directory, search nr. 4 will succeed if there is a previously-installed Hyperion in /usr/local/lib/Hercules-V4, or all four searches will fail if not.

When building using a multi-configuration generator (Windows or macOS) and running from the build directory as the CWD, all four searches fail because crypto is not a subdirectory of the build directory. Here's the build tree for a multi-configuration generator:

\ # build directory for Hyperion \Release # contains almost all executables, libraries, and modules. \crypto # build directory for crypto source directory contents \crypto\Release # contains dyncrypt.so The failures of runtest when using a multi-configuration generator on a macOS made me dig into this. The issue does not happen on Windows with a multi-configuration generator because the NMake build does not set MODULESDIR and searches for dyncrypt, not crypto/dyncrypt . The Windows loadable module search is like dlopen() in that a \ stops searches of "extra" directories. Without a \, the executable's location is searched, and search 2 succeeds. The Windows NMake puts dyncrypt.dll in the same build directory and the same install directory as hercules.exe; CMake was coded the same way. Preliminary testing of coding in hdl.c and CMake scripts shows that this issue can be addressed. I need to do a full cycle of functional and regression testing. Best Regards, Steve Orso — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread .

[srorso]

Hi Enrico,

there is no relation between the build tree structure and the install tree structure

This matches my understanding.

What makes things so difficult for hdl_dlopen() is that the two trees are not related, are usually not the same, and hdl_dlopen() cannot determine which tree structure Hercules is being executed from.

I considered for a time adding the crypto build path to the default build RPATH. I think that would have worked, but it would not decrease complexity.

the CMAKE invocation of the install_name_tool does not really relink anything it just patches the installed things with the proper library

Also matches my understanding. CMake just edits in-place the RPATH/RUNPATH section in the executable / shared library / loadable module. CMake installs are much faster than libtool installs.

I am not sure what happens if the install RPATH is longer than the build RPATH, although for Hyperion such an issue is very unlikely.

Best Regards, Steve Orso

[ghost]

testing now my idea hope to be back soon e.

On 19 Mar 2018, at 16:37, Stephen Orso notifications@github.com wrote:

Hi Enrico,

there is no relation between the build tree structure and the install tree structure

This matches my understanding.

What makes things so difficult for hdl_dlopen() is that the two trees are not related, are usually not the same, and hdl_dlopen() cannot determine which tree structure Hercules is being executed from.

I considered for a time adding the crypto build path to the default build RPATH. I think that would have worked, but it would not decrease complexity.

the CMAKE invocation of the install_name_tool does not really relink anything it just patches the installed things with the proper library

Also matches my understanding. CMake just edits in-place the RPATH/RUNPATH section in the executable / shared library / loadable module. CMake installs are much faster than libtool installs.

I am not sure what happens if the install RPATH is longer than the build RPATH, although for Hyperion such an issue is very unlikely.

Best Regards, Steve Orso

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hercules-390/hyperion/issues/256#issuecomment-374256547, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUXcIWI15sKQChk__rtKAnd1YKtNHQvks5tf9C8gaJpZM4So2ZZ.

[srorso]

Hi Enrico,

It might make sense to just use

/lib/Hercules or some such all of the time, whether the libraries/modules are to be installed in a system location or some other.

If the "Hercules" suffix is always present and included in the install RPATH, the installation directory tree can be moved anywhere, regardless of whether it's a system location. Saves testing for system locations on an assortment of distros or depending on ${CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES}. The install tree would look the same no matter where one finds it.

I am doing something similar in CMakeLists.txt at line 551, but the more I look at it the less I like it.

Just a suggestion, and I defer to your experience on this...

Best Regards, Steve Orso

[ghost]

I tested successfully my proposed environment ... I had to move the build instructions to the main CMakeLists when You use the add_directory the binaries are built in a subdirectory of the CMAKE_BINARY_DIR and without the move make test fails for the crypto things

I already did the work for the moduledir distinction

and ... no it is not necessary at least on apple to include the "Hercules" suffix in the install RPATH but I will check anyway

and right now the installation directory can be moved everywhere i prefer to not use the modulesdir when installing in a dedicated directory

• I just hate an unnecessary nesting of directory IMO the same should applied to the share things if You install to - lets say /hercules it is not necessary to confirm the hercules token it is like using belt and suspenders when just one is enough

but in order not to confuse things it would be nice to install the HTML stuff in a directory of its own

the HDL treatment of the MODULESDIR is still pretty murky it works but the install tree is not relocatable with the FULL path

I thought that the MODULESDIR to get a relocatable tree would be something like bin/Hercules, but I was wrong, for some weird reason it just wants just Hercules when installing to a common directory

here is the zipped diffs related to commit 50504f15076184d76a10268d0803564088965a57 the one before the softfloat fixes

MODULE_2_patch.txt.zip

[Peter-J-Jansen]

Hi Enrico, Steve, John and Ivan, I am busy rigging up a better macOS system. As soon as that is ready I will be testing the results of your joint efforts, for which I want to thank you all very much ! This is fun ! Cheers, Peter J.