ZeroMQ does not build for QNX 6.3 with CMake

[sherwood-nyab]

Issue description

I am evaluating ZeroMQ for use with our embedded product, but we are constrained to running on QNX 6.3 (ancient) and building with GCC 3.3.5. I am cross-compiling the library on Windows using CMake within Qt Creator/MinGW. CMake generation succeeds, but compilation fails.

For example, ZeroMQ expects malloc() to be declared in but QNX declares it in . This could be fixed with something like:

#ifdef __QNXNTO__
#include <malloc.h>
#endif

A similar conditional would correct the problem that QNX has <sys/poll.h> instead of .

Environment

• libzmq version: 4.3.1
• OS: Windows 7 x64 SP1
• QNX Momentics IDE 6.3.0 + Service Pack 2 provides the toolchain - GCC 3.3.5
• CMake 3.9.4
• Qt Creator 3.6.1 - kit: Desktop Qt 5.6.0 MinGW 32bit

Minimal test code / Steps to reproduce the issue

1. It's complicated since it requires custom CMake files that include setup for cross-compiling for QNX.

What's the actual result? (include assertion message & call stack if applicable)

15:01:46: Running steps for project 3rdPartyResearch... 15:01:46: Starting: "C:\Projects\Qt\5.6.0\MingW\Tools\mingw492_32\bin\mingw32-make.exe" [ 0%] Building C object libzmq/CMakeFiles/objects.dir/src/tweetnacl.c.o [ 1%] Building CXX object libzmq/CMakeFiles/objects.dir/src/precompiled.cpp.o [ 1%] Building CXX object libzmq/CMakeFiles/objects.dir/src/address.cpp.o [ 1%] Building CXX object libzmq/CMakeFiles/objects.dir/src/client.cpp.o In file included from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/socket_base.hpp:39, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/client.hpp:33, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/client.cpp:32: C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp: In constructor zmq::blob_t::blob_t(unsigned int): C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp:79: error: malloc undeclared (first use this function) C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp:79: error: (Each undeclared identifier is reported only once for each function it appears in.) C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp: In constructor zmq::blob_t::blob_t(const unsigned char, unsigned int): C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp:89: error: malloc undeclared (first use this function) C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp: In member function void zmq::blob_t::set_deep_copy(const zmq::blob_t&): C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp:129: error: malloc undeclared (first use this function) C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp: In member function void zmq::blob_t::set(const unsigned char, unsigned int): C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp:140: error: malloc undeclared (first use this function) C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp: In member function void zmq::blob_t::clear(): C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp:151: error: free undeclared (first use this function) C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp: In destructor zmq::blob_t::~blob_t(): C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/blob.hpp:160: error: free undeclared (first use this function) In file included from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poller.hpp:52, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/socket_base.hpp:41, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/client.hpp:33, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/client.cpp:32: C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poll.hpp:43:18: poll.h: No such file or directory In file included from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poller.hpp:52, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/socket_base.hpp:41, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/client.hpp:33, from C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/client.cpp:32: C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poll.hpp: At global scope: C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poll.hpp:96: error: pollfd was not declared in this scope C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poll.hpp:96: error: template argument 1 is invalid C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poll.hpp:96: error: template argument 2 is invalid C:/Data/Workspaces/3rdPartyResearch/source/zeromq-4.3.1/src/poll.hpp:96: error: ISO C++ forbids declaration of pollset_t with no type cc: C:/QNX630/host/win32/x86/usr/lib/gcc-lib/i386-pc-nto-qnx6.3.0/3.3.5/cc1plus caught signal 1 libzmq\CMakeFiles\objects.dir\build.make:133: recipe for target 'libzmq/CMakeFiles/objects.dir/src/client.cpp.o' failed mingw32-make[2]: [libzmq/CMakeFiles/objects.dir/src/client.cpp.o] Error 1 CMakeFiles\Makefile2:128: recipe for target 'libzmq/CMakeFiles/objects.dir/all' failed mingw32-make[1]: [libzmq/CMakeFiles/objects.dir/all] Error 2 makefile:128: recipe for target 'all' failed mingw32-make: [all] Error 2 15:01:51: The process "C:\Projects\Qt\5.6.0\MingW\Tools\mingw492_32\bin\mingw32-make.exe" exited with code 2. Error while building/deploying project 3rdPartyResearch (kit: Desktop Qt 5.6.0 MinGW 32bit) When executing step "Make" 15:01:51: Elapsed time: 00:04.

What's the expected result?

Code compiles without errors for QNX 6.3.

[bluca]

Could you please send a PR with the fixes?

[bluca]

Iirc somebody in the past sent fixes for qnx - have you tried using autotools instead of cmake?

[sherwood-nyab]

I have been trying to fix it using CMake tricks without modifying any library files but have not been successful, which is why I am considering patching the code. So, if "PR" means "pull request" then no, I have not even created a branch. I wanted first to open a conversation about this.

We use CMake for our builds so that is what I have been using. Is it possible to get the autotools to cross-compile? But you gave me an idea: it might be worthwhile to try to build it on a QNX machine. However, our Jenkins builds are done on Linux boxes so a native build is a deviation.

[bluca]

yes, autotools can cross compile (and it's in fact one of its main purposes) - it's possible to update cmake too of course, it sounds like it needs something like if target = qnx then CPPFLAGS+=-isystem sys

[sherwood-nyab]

(Sorry, I unintentionally closed this issue, so I reopened it.)

I have successfully built libzmq(.a, .so, .so.5, .so.5.2.2) by modifying about a dozen files with

#if defined(__QNXNTO__) && (_NTO_VERSION == 630)
  // @sherwood-nyab:
  // QNX 6.3.0 declares malloc() here, not in <stdlib.h>.
  // TODO Update the _NTO_VERSION conditional if newer QNX versions also require <malloc.h>.
#include <malloc.h>
#endif

and

#if defined(__QNXNTO__) && (_NTO_VERSION == 630)
  // @sherwood-nyab:
  // QNX 6.3.0 puts poll.h here.
  // TODO Update the _NTO_VERSION conditional if newer QNX versions also have <sys/poll.h>.
#include <sys/poll.h>
#else
#include <poll.h>
#endif

and in one case

#if defined(__QNXNTO__) && (_NTO_VERSION == 630)
  // @sherwood-nyab:
  // QNX build does not see this file. It was explicitly included by
  // some other source files - was it forgotten here?
#include <arpa/inet.h>
#endif

I made these changes to a fork of libzmq-master, sherwood-nyab/libzmq. Unfortunately, due to concerns from our company management I am currently unable to create a pull request so that I can share it with you.

Anyway, I regard this as a trial run. Maybe there is a better way to do this. In the source code I see where ZMQ_HAVE_WINDOWS is used for conditional compilation. It occurs to me that something like ZMQ_HAVE_QNX630 might be a better way, but I don't know enough about the project to make that happen.

Now I must build the unit tests. This line in unity.c fails to compile

for (int i = 1; i < argc; i++)

with error 'for' loop initial declaration used outside C99 mode I think that this is for a newer C standard not supported by GCC 3.3.5.

[bluca]

I made these changes to a fork of libzmq-master, sherwood-nyab/libzmq. Unfortunately, due to concerns from our company management I am currently unable to create a pull request so that I can share it with you.

I can try to provide a fix for autoconf/cmake later today then

Anyway, I regard this as a trial run. Maybe there is a better way to do this. In the source code I see where ZMQ_HAVE_WINDOWS is used for conditional compilation. It occurs to me that something like ZMQ_HAVE_QNX630 might be a better way, but I don't know enough about the project to make that happen.

As mentioned the best way would be to add -isystem sys to the cppflags in cmake/autoconf, so that the code doesn't need to change, I can work on that and the C99 thing

[sherwood-nyab]

Please, I don't want to waste your time. If I cannot get the unit tests to pass on QNX 6.3 then this library may not be a candidate for our product, so compatibility with this ancient compiler and OS would be a non-issue.

(I am aware of the 2 closed 2010 issues that mention QNX (#32 and #34), but they provide no details about the version of QNX or GCC.)

BTW, I tried '-isystem sys' (also with the full path to sys) but it had no effect. I examined the compiler command line during the build and everything looked correct, but it still could not find <sys/poll.h>. I finally gave up and started to modify the code just so that I could make some progress.

But that would not cause to be included in the source files that call malloc(). I have already tried '-include malloc.h' but that broke CMake (in the CMake check_function_exists() macro). Once I removed the -include the problem went away.

As for C99, I am going to look for a compiler option. The way I figure it, if the compiler knows about a "C99 mode" then it must know about other modes. Hopefully it supports a newer mode that compiles the test code. (And with any luck it will fix the other problems.)

[bluca]

@sigiesec any chance unity.c could be fixed upstream to work without C99? it seems a simple change

[sherwood-nyab]

I thought that the problem might be our use of the '-pedantic' option, but removing it had no effect.

[bluca]

unity.c is embedded in the repo, so you can just modify it yourself to make it compile for now

[sherwood-nyab]

FWIW, here is the compiler command line for unity.c (the weird formatting is due to special characters in the output). One thing to note is the use of function UnityParseOptions on line 3.

[ 34%] Building C object libzmq/tests/CMakeFiles/unity.dir//external/unity/unity.c.o C:/Data/Workspaces/3rdPartyResearch/source/libzmq-master/external/unity/unity.c: In function UnityParseOptions': C:/Data/Workspaces/3rdPartyResearch/source/libzmq-master/external/unity/unity.c:1408: error:for' loop initial declaration used outside C99 mode cc: looking for gcc_ntox86_cpp in C:/QNX630/host/win32/x86/etc/qcc/gcc/3.3.5/gcc_ntox86_cpp++.conf cc: looking for gcc_ntox86_cpp in C:/QNX630/host/win32/x86/etc/qcc/gcc/3.3.5/gcc_ntox86_cpp.conf C:/QNX630/host/win32/x86/usr/lib/gcc-lib/i386-pc-nto-qnx6.3.0/3.3.5/cc1 -DUNITY_EXCLUDE_FLOAT -DUNITY_USE_COMMAND_LINE_ARGS -DZMQ_CUSTOM_PLATFORM_HPP -D_REENTRANT -D_THREAD_SAFE -IC:\Data\Workspaces\3rdPartyResearch\source\libzmq-master\tests..\external\unity -IC:\Data\Workspaces\3rdPartyResearch\source\libzmq-master..\include -IC:\Data\Workspaces\3rdPartyResearch\source\build-cmake-Desktop_Qt_5_6_0_MinGW_32bit-Default\libzmq -Wall -D_NTO_VERSION=630 -O3 -quiet -nostdinc -DLANGUAGE_C -D_LANGUAGE_C -DQNX -DQNXNTO -DGNUC=3 -DGNUC_MINOR=3 -DGNUC_PATCHLEVEL=5 -Dunix -Dunix -DELF -DX86 -Di386 -DLITTLEENDIAN__ -Wpointer-arith -Wwrite-strings -idirafter C:/QNX630/target/qnx6/usr/include C:\Data\Workspaces\3rdPartyResearch\source\libzmq-master\external\unity\unity.c -dumpbase C:\Data\Workspaces\3rdPartyResearch\source\libzmq-master\external\unity\unity.c -o C:\Users\sherwood\AppData\Local\Temp\2_qcc_40464\unity.s

[bluca]

Can you check if it builds after you change that for (int i...) to fix it for c90? It's a very simple change and it can be sent upstream

[bluca]

Also, please try with autotools - I think the compiler does support c99, but cmake does not set -std=c99

[sherwood-nyab]

unity.c compiles after changing it to

int i;
for (i = 1; i < argc; i++)

and I have successfully built 100% of the 'tests' and 'unittests', but only after including in a bunch of cpp files.

Sorry, I have very little experience with autotools and have no idea how to get it to cross-compile for QNX, and right now I have my hands full just getting the CMake build to work.

However, I realize that my first attempt at this is ugly. After I have convinced myself that this is a viable library for our product I expect that I can clean things up by modifying the library CMake files. As you suggested, the autotools might already build for QNX, so fixing the CMake build without modifying any source code is my preference.

[bluca]

ok, in the meanwhile I opened a PR to unity's upstream. If you end up needing it, we can backport it to this repo.

[sherwood-nyab]

@bluca Can we close this? I now know that I can get libzmq to build for QNX 6.3.0. Next up is testing. I am running the tests now and I see that many test cases are failing, so I am thinking that I should create a separate issue for that.

[bluca]

You had to do some changes to the code/makefiles right? If so this issue is not really solved yet

[sherwood-nyab]

Yes, I made changes to code. I am not sure what to do next. 16 test suites (executables) are failing one or more test case (I have not collected all the details). If all the tests had passed then I was going to start over and see if I could get the library to build by first modifying the CMake files, and then modifying the code as a last resort. Now I am not sure since I have no idea about how much effort it will be to get all the tests to pass (or at least those for capabilities that we might want to use for our embedded product).

[bluca]

What exactly is failing?

[sherwood-nyab]

Here is the stdout and stderr from my test-runner script. stdout.txt stderr.txt

[bluca]

Those are all related to the wildcard IPC bind. Not all unixes support it, GNU/Hurd doesn't for example - you can just ignore them. Once the makefiles are updated to detect QNX, they can be skipped like for Hurd.

[bluca]

So for your application, just avoid using "ipc://*" and you'll be fine.

[sherwood-nyab]

Great news! I will proceed with my plan to update the CMake build files, including skipping the wildcard IPC bind tests.

[sherwood-nyab]

I see that these are all "address already in use" errors for zmq_bind() or bind(), and it is not clear to me that all of them are for wildcard IPC. For TCP sockets on QNX I have had issues with bind() failing with that error during unit tests, even after multiple delayed retries. I remember trying different socket options, but in some cases I could only fix it by rebooting the machine!

[bluca]

as far as I could see they were all in ipc tests

[sherwood-nyab]

@bluca So how do I skip the wildcard IPC bind tests for QNX like for GNU/Hurd? I looked through configure.ac but did not find anything that seems related.

[bluca]

It's in Makefile.am, there should be an "if GNU ... XFAIL += ..."

[sherwood-nyab]

Thank you, that helped. I am making progress. 79 out of 86 test executables PASS for QNXNTO after excluding the build of 3 executables, and excluding some test cases via conditional compilation. The latest test output is attached. output.txt Most of the remaining errors are "address already in use" for the bind, a problem that I have seen before for QNX.

From tests/CMakeLists.txt, exclude some test executables:

if(NOT QNXNTO) list(APPEND tests test_ipc_wildcard test_pair_ipc test_reqrep_ipc ) endif() list(APPEND tests test_rebind_ipc test_proxy test_proxy_hwm test_proxy_single_socket test_proxy_terminate test_getsockopt_memset test_filter_ipc test_stream_exceeds_buffer test_router_mandatory_hwm test_use_fd test_zmq_poll_fd )

... etc.

From tests/test_reconnect_iv1.cpp, exclude a test case (one of several):

if !defined(ZMQ_HAVE_WINDOWS) && !defined(ZMQ_HAVE_GNU) && !defined(ZMQ_HAVE_QNXNTO)

void test_reconnect_ivl_ipc (void) { char my_endpoint[256]; size_t len = sizeof (my_endpoint);

void *sb = test_context_socket (ZMQ_PAIR);
TEST_ASSERT_SUCCESS_ERRNO (zmq_bind (sb, "ipc://*"));

... etc.

[bluca]

Most of the remaining errors are "address already in use" for the bind, a problem that I have seen before for QNX.

Those are all IPC wildcard binds:

C:/Users/sherwood/Data/Workspaces/3rdPartyResearch/source/libzmq-master/tests/test_filter_ipc.cpp:55:test_no_filters:FAIL: zmq_bind (sb, endpoint) failed, errno = 248 (Address already in use)
C:/Users/sherwood/Data/Workspaces/3rdPartyResearch/source/libzmq-master/tests/test_rebind_ipc.cpp:55:test_rebind_ipc:FAIL: zmq_bind (sb0, my_endpoint) failed, errno = 248 (Address already in use)
C:/Users/sherwood/Data/Workspaces/3rdPartyResearch/source/libzmq-master/tests/test_term_endpoint.cpp:55:test_unbind_via_last_endpoint:FAIL: zmq_unbind (pull, my_endpoint) failed, errno = 22 (Invalid argument)
C:/Users/sherwood/Data/Workspaces/3rdPartyResearch/source/libzmq-master/tests/test_use_fd.cpp:55:test_req_rep_ipc:FAIL: bind (s_pre, addr_, addr_len_) failed, errno = 248 (Address already in use)
C:/Users/sherwood/Data/Workspaces/3rdPartyResearch/source/libzmq-master/tests/test_use_fd.cpp:55:test_pair_ipc:FAIL: bind (s_pre, addr_, addr_len_) failed, errno = 248 (Address already in use)

You can skip those like the others.

I suspect the root cause is that QNX, like GNU/Hurd, does not support getsockname on AF_UNIX sockets. https://github.com/zeromq/libzmq/blob/master/src/ipc_listener.cpp#L202

[sherwood-nyab]

Okay, but there was no conditional compilation in those files (for lines 1, 2, 4, 5; I missed the one for line 3) to exclude those tests cases for other platforms, so I did not want to exclude them only for QNX without more information. Maybe these are newer tests that have not been run on GNU/Hurd?

Based on this new information I will exclude them for QNX.

[sherwood-nyab]

I am down to these failures: failures.txt test_fork appears to be terminating due to a SIGALARM, but I don't know why, nor to I know the source of the "Alarm clock" message. test_spec_push_pull appears to end abnormally during the 'test_spec_pushpull_inproc_push_multipart_atomic_drop_block' test case. test_system is failing due to a system limit, but QNX has ksh instead of bash and I have not found a way to increase the number of file handles.

[bluca]

test_fork appears to be terminating due to a SIGALARM, but I don't know why, nor to I know the source of the "Alarm clock" message.

It just means it times out - the tests have a 60 seconds timer to avoid deadlocks holding a build back. Does QNX have POSIX compliant fork semantics? Any known issues compared to Linux?

test_spec_push_pull appears to end abnormally during the 'test_spec_pushpull_inproc_push_multipart_atomic_drop_block' test case.

Log doesn't say anything useful, have you tried running manually through gdb?

test_system is failing due to a system limit, but QNX has ksh instead of bash and I have not found a way to increase the number of file handles.

Solaris has the same problem with a low default, there's an ifdef at the beginning of the test that lowers the amountof file handles required, just edit it to also change it for QNX

[sherwood-nyab]

I see no evidence that test_fork actually creates a new process on the system, which is why it "times out" with a SIGALARM. This concerns me since our product is multi-processing. Is this the only test that verifies communication between processes?

[sherwood-nyab]

The QNX 6.3 docs say that fork() conforms to POSIX Std 1003.1, 2004 Edition, which includes some draft standards. This is from the example in the library reference:

if( ( pid = fork() ) == -1 ) {
   perror( "fork" );
   return EXIT_FAILURE;
}
if( pid == 0 ) {
  /* This is the child process. */
  [snip]
} else {
  /* This is the parent process. */
  [snip]
}

[sherwood-nyab]

It turns out that ksh does have ulimit, so test_system now passes.

[bluca]

does src/platform.hpp have #define HAVE_FORK 1 ?

[sherwood-nyab]

The platform.hpp in the build tree has #define HAVE_FORK

[bluca]

Ok so it's not a detection problem - are you able to check where it's actually stuck with gdb or a debugger? The test is pretty small https://github.com/zeromq/libzmq/blob/master/tests/test_fork.cpp

[sherwood-nyab]

Please hold while I learn to use gdb. The first thing I need to figure out is how to get a CMake build that puts debugging symbols into the executable. I know that I need to add the -g compiler option, but right now it appears to be building for a Release which does not include that.

[sigiesec]

You need to specify -DCMAKE_BUILD_TYPE=Debug when calling cmake.

[sherwood-nyab]

Okay, the executable now has debugging symbols, but I am cross-compiling on a Windows host so (a) all the paths to the source files are Windows paths, and (b) in any case my target QNX machine does not have the source files. It's never easy!

Update: I put the source on the QNX target machine and I figured out how to tell gdb to look there, so now I am ready to do some debugging!

[sherwood-nyab]

test_spec_pushpull is getting hit with a SIGPIPE. Here is the stack backtrace for all threads: test_spec_pushpull-backtrace.txt It happens in thread 1 when it calls msleep(SETTLE_TIME) on line 338. Since that is not the first call to msleep in that function there seems to be something special about this call.

[sherwood-nyab]

test_fork does not check for fork() returning -1, so in that case it will enter the parent process block. A fork error is the only thing that explains what is going on: I never see the child process on the system, and the parent process blocks on the zmq_recv(pull, ...) call until a SIGALARM terminates the process. (gdb reports that the "program exited normally", which I find odd. It clearly exited abnormally since it never returned from zmq_recv.)

    int pid = fork ();
    if (pid == 0) {
        //  Child process
        [snip]
    } else {
        //  Parent process
        [snip]
    }

[bluca]

did you manage to check with gdb if it's returning an error? or simply edit the test code to print it

[sherwood-nyab]

gdb says pid is -1 and errno is 89 (ENOSYS, unknown system call).

[sigiesec]

http://www.qnx.com/developers/docs/6.5.0SP1.update/com.qnx.doc.neutrino_lib_ref/f/fork.html#Caveats

[bluca]

Right - libzmq requires the use of threads. Disable that test on QNX, and in your application make sure you do the fork before initialising the context

[sherwood-nyab]

Oh! The 6.3 docs state that as well, didn't see it. So I will remove test_fork for QNXNTO.

However, shouldn't the test do something like assert (pid >= 0)? Or is it working as intended?

[bluca]

Yes, the test should definitely be improved

[sherwood-nyab]

Success! I now have 84 test executables returning a zero exit code (although many do that but don't output any message indicating pass/fail or anything else). The final change was to ignore SIGPIPE just like MVS to prevent test_spec_pushpull_tcp_push_multipart_atomic_drop_block() from terminating abnormally.