Building OpenSSL 3.0 on z/OS (IBM mainframe) fails. Linker cannot handle duplicate file names in source tree

[opensslonzos-github]

When building on z/OS, the fact that there are multiple files named engine.c (info.c, names.c, provider.c, and driver.c ) causes the linker/binder to get confused. EX: openssl-3.0.11/apps/engine.c openssl-3.0.11/apps/lib/engine.c

The linker/binder errors look like this when linking the executable "openssl": IEW2459W 9206 INCLUDED MEMBER get_cipher_from_engine FAILED TO RESOLVE REFERENCE.

IEW2497W 9229 THE SYMBOL get_cipher_from_engine WAS EXPECTED TO BE RESOLVED BY INCLUDING MEMBER libapps-lib-engine.o FROM THE LIBRARY DEFINED BY DDNAME /0000038

In previous versions, OpenSSL has done an excellent job of preventing this problem by having every .c file have a unique name across all directories. I see that in 3.0.X, OpenSSL builds the .o/.obj files with unique names which helps, but does not go far enough for the z/OS toolset.

I was able to get around this by renaming files and editing the build.info files before the configure step as follows: mv apps/engine.c apps/apps_engine.c mv apps/info.c apps/apps_info.c mv apps/lib/names.c apps/lib/apps_names.c mv test/testutil/provider.c test/testutil/test_provider.c mv test/testutil/driver.c test/testutil/test_driver.c mv apps/build.info apps/build.info.orig mv apps/lib/build.info apps/lib/build.info.orig mv test/build.info test/build.info.orig cat apps/build.info.orig | sed "s@OPENSSLSRC engine.c@OPENSSLSRC apps_engine.c@" | sed "s@info.c@apps_info.c@" > apps/build.info cat apps/lib/build.info.orig | sed "s@ names.c@ apps_names.c@" > apps/lib/build.info cat test/build.info.orig | sed "s@\/provider.c@\/test_provider.c@" | sed "s@driver.c@test_driver.c@"> test/build.info

[jamuir]

Is this specific to 3.0? Does 3.2 fail similarly?

[opensslonzos-github]

I have not tried, but I can see the same issue of multiple files with the same name in them in both 3.1.2 and 3.2.0 Ex: openssl-3.2.0\apps\engine.c openssl-3.2.0\apps\lib\engine.c

openssl-3.1.2\apps\engine.c openssl-3.1.2\apps\lib\engine.c

[levitte]

It's a bit surprising that the source file names have an impact on the linker.

A wild guess is that z/OS maintains some sort of "module name" derived from the source file name, and that the linker works with those rather than the object file names, is that about correct? Is there a way to affect that other than a file rename fest? (I know other platforms that resolve similar things through C preprocessing pragmas, for example)

[opensslonzos-github]

I agree that is is both annoying and surprising that this is the case. I was suggesting the fix above as it already seems common (and has historical precedence) in the OpenSSL source tree to have globally unique names. Ex: openssl-3.2.0\ssl\statem\statemclnt.c
openssl-3.2.0\ssl\quic\quic* (all have quic_ prepended) rather than openssl-3.2.0\ssl\statem\clnt.c
openssl-3.2.0\ssl\quic....

And it was this way in 1.1.1xyz versions of the library.

I believe there are only 5 files that need to be renamed.

[jamuir]

@opensslonzos-github : what version of the linker do you see this with? have you also reported this to z/OS?

[opensslonzos-github]

This is with "z/OS V2 R1 BINDER".
This is what IBM calls a linker. I have not mentioned it to IBM. Getting mainframe tooling changed is a very long process that can take years.

[levitte]

We already have a check somewhere to see that we don't have multiple object files with the same name in static libraries. This shouldn't be too terrible to extend to source file names that goes into any end product (libraries as well as executable). But oh boy...

[jamuir]

This is with "z/OS V2 R1 BINDER".

I see the most recent version is 3.1.0. I wonder if the problem is still present there.

https://www.ibm.com/docs/en/zos/3.1.0

I have not mentioned it to IBM.

That's unfortunate. It should be reported.

[nhorman]

FWIW, this is the documentation for the error referenced above: https://www.ibm.com/docs/en/zos/3.1.0?topic=iew2999-iew2459w It makes reference to the fact that the error can occur because the 'label' for the symbol being linked isn't present and that the user response should be to "Ensure that the member name and library are correct and that the member contains a section or label which matches the member name."

That seems pretty vague to me (or perhaps I just don't understand the terminology), but it kind of lends some support to @levitte comment above about the compiler storing some information based on the source name.

While it would be pretty easy to rectify the situation by renaming the files to be globally unique, I'm not sure thats going to be reasonable in perpituity. After all, directories are meant to manage that, no?

[opensslonzos-github]

I believe the OpenSSL project ran into this same issue long ago and that is why there are only a few "stragglers" now. Most of the files within the OpenSSL directory structure are already uniquely named with their parent directory name prepended to their name.
I do not know this as a fact, but it would appear so. Thank you. I have also noticed that after getting 3.0.11 to build on z/OS, many tests fail. I hope to have something useful to contribute to the project in the near future.