Build failure with nss when nss headers/libraries are in a nonstandard location

[ryandesign]

Issue description

I'm a MacPorts developer. Another MacPorts developer recently attempted to add optional nss support to our zmq port but it fails to build. I believe the problem is a deficiency in the zmq cmake build system.

Environment

• libzmq version (commit hash if unreleased): 4.3.4
• OS: macOS 12.3.1

Minimal test code / Steps to reproduce the issue

On macOS, Install MacPorts, then run:

sudo port install cmake libsodium nss
curl -OL https://github.com/zeromq/libzmq/archive/refs/tags/v4.3.4.tar.gz
tar xzf v4.3.4.tar.gz
mkdir build
cd build
cmake -DWITH_NSS=ON ../libzmq-4.3.4
make -j$(sysctl -n hw.activecpu)

What's the actual result? (include assertion message & call stack if applicable)

libzmq-4.3.4/src/ctx.cpp:56:10: fatal error: 'nss.h' file not found
#include <nss.h>
         ^~~~~~~

In MacPorts, nss and its dependency nspr install their headers in a subdirectory.

If this is fixed by adding the nss include path:

% cmake -DCMAKE_CXX_FLAGS=-I/opt/local/include/nss -DWITH_NSS=ON ../libzmq-4.3.4
% make -j$(sysctl -n hw.activecpu)

Then the error is:

In file included from libzmq-4.3.4/src/ctx.cpp:56:
In file included from /opt/local/include/nss/nss.h:34:
/opt/local/include/nss/seccomon.h:17:10: fatal error: 'prtypes.h' file not found
#include "prtypes.h"
         ^~~~~~~~~~~

-I flags for both the nss and nspr include paths are already in the pkg-config files:

% pkg-config nss --cflags
-I/opt/local/include/nss -I/opt/local/include/nspr

If this is fixed by using pkg-config:

% cmake -DCMAKE_CXX_FLAGS="$(pkg-config nss --cflags)" -DWITH_NSS=ON ../libzmq-4.3.4
% make -j$(sysctl -n hw.activecpu)

Then the error is:

ld: library not found for -lssl3
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Here again the problem is that, in MacPorts, the nss ssl3 library is installed in a subdirectory, which is reflected in the pkg-config files:

% pkg-config nss --libs
-L/opt/local/lib/nss -L/opt/local/lib/nspr -L/opt/local/lib -lssl3 -lsmime3 -lnssutil3 -lnss3 -lplds4 -lplc4 -lnspr4
% pkg-config nss --libs-only-l
-lssl3 -lsmime3 -lnssutil3 -lnss3 -lplds4 -lplc4 -lnspr4
% pkg-config nss --libs-only-L
-L/opt/local/lib/nss -L/opt/local/lib/nspr -L/opt/local/lib

I'm not familiar enough with how cmake integrates with pkg-config to know how to fix your build system to get this information from the pkg-config files.

What's the expected result?

successful build