z/OS changes for https://github.com/libuv/libuv/pull/3060/

richardlau commented 3 years ago

This task is to log required changes to build/test https://github.com/libuv/libuv/pull/3060/

[ ] Download and build https://github.com/ibmruntimes/zoslib (tbd do this as part of the ansible set up or in the libuv test job)
[ ] Set CXX environment variable. Currently we set CC to /bin/xlc https://github.com/nodejs/build/blob/c6d76d0bdf38b0f5997c002fa72bc21a7ee02ca6/ansible/roles/jenkins-worker/templates/zos_start.j2#L18. @zsw007 has informed me that IBM have been using xlclang/xlclang++ which I believe we have on our z/OS CI hosts.

richardlau commented 3 years ago

I've made a temporary clone of https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake as https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake-richardlau/ to test out changes. For now I'm limiting the changes to the Jenkins job.

Cloning issue

The first issue I've run into is cloning https://github.com/ibmruntimes/zoslib. Attempting to clone with the https URL results in an SSL verification error: e.g. https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake-richardlau/5/BUILDTYPE=Debug,nodes=zos-s390x/console

+ /u/iojs/git-wrapper clone https://github.com/ibmruntimes/zoslib --depth 1
Cloning into 'zoslib'...
fatal: unable to access 'https://github.com/ibmruntimes/zoslib/': SSL certificate problem: unable to get local issuer certificate

there's a note about git on z/OS not supporting https URLs in https://github.com/nodejs/build/blob/ae1841834dfd8d941f356b38f187c07e39131642/ansible/roles/jenkins-worker/tasks/main.yml#L328-L330 but I'm wondering if we can supply certificates like we used to do on AIX 6.1: https://github.com/nodejs/build/blob/4671676f9885e8e58b9e676025a40a37d0f9cfbd/ansible/aix61-standalone/resources/S20jenkins#L19-L20 for now I've passed -c http.sslVerify=false to the clone command to temporarily skip SSL verification.

FWIW https://github.com/ibmruntimes/zoslib says to do:

$ git clone git@github.com:ibmruntimes/zoslib.git zoslib

but this doesn't work on our CI presumably because of GitHub permissions for the whatever account the iojs user's ssh key maps to: e.g. https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake-richardlau/6/BUILDTYPE=Debug,nodes=zos-s390x/console

+ /u/iojs/git-wrapper clone git@github.com:ibmruntimes/zoslib --depth 1
Cloning into 'zoslib'...
FOTS1370 Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Compiler issues

If I set CC/CXX environment variables to xlc/xlc++ compilation fails: https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake-richardlau/7/BUILDTYPE=Debug,nodes=zos-s390x/console

FSUM3210 xlc++: Command option -Wno-parentheses contains an incorrect subargument.
FSUM8226 make: Error code 40 
FSUM8226 make: Error code 255 
FSUM8226 make: Error code 255

@zsw007 let me know via email that the IBM z/OS team use xlclang/xlclang++ to build. This appears to be available on our CI hosts but is failing: https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake-richardlau/8/BUILDTYPE=Debug,nodes=zos-s390x/console

+ cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
-- The CXX compiler identification is unknown
-- The C compiler identification is zOS 2.4.0
-- The ASM compiler identification is zOS
-- Found assembler: /bin/xlc
-- Check for working CXX compiler: /bin/xlclang++
-- Check for working CXX compiler: /bin/xlclang++ -- broken
CMake Error at /u/iojs/local/share/cmake-3.5/Modules/CMakeTestCXXCompiler.cmake:54 (message):
  The C++ compiler "/bin/xlclang++" is not able to compile a simple test
  program.

  It fails with the following output:

   Change Dir: /u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Debug/nodes/zos-s390x/zoslib/out/CMakeFiles/CMakeTmp

  Run Build Command:"/bin/make" "cmTC_c2359/fast"

  make -f CMakeFiles/cmTC_c2359.dir/build.make
  CMakeFiles/cmTC_c2359.dir/build

  Building CXX object CMakeFiles/cmTC_c2359.dir/testCXXCompiler.cxx.o

  /bin/xlclang++ -o CMakeFiles/cmTC_c2359.dir/testCXXCompiler.cxx.o -c
  /u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Debug/nodes/zos-s390x/zoslib/out/CMakeFiles/CMakeTmp/testCXXCompiler.cxx

  FSUM3224 xlclang++: Fatal error in /usr/lpp/cbclib/xlclang/exe/clcdrvr:
  signal 9 received.

  FSUM8226 make: Error code 251

  FSUM8226 make: Error code 255

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:9 (project)

-- Configuring incomplete, errors occurred!
See also "/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Debug/nodes/zos-s390x/zoslib/out/CMakeFiles/CMakeOutput.log".
See also "/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Debug/nodes/zos-s390x/zoslib/out/CMakeFiles/CMakeError.log".

It looks like xlclang/xlclang++ can display usage information but not seem to compile:

/u/iojs/temp: >xlclang++
FSUM7261 xterm-256color: Unknown terminal type, using dumb.

  z/OS V2.4.1 XL C/C++

  NAME
         xlclang, xlclang++ - invoke the IBM C/C++ compiler.
...

/u/iojs/temp: >xlclang runAsciiBash.c
FSUM3224 xlclang: Fatal error in /usr/lpp/cbclib/xlclang/exe/clcdrvr: signal 9 received.
/u/iojs/temp: >

zsw007 commented 3 years ago

@richardlau

For the Cloning issue, we are able to clone https URL by supplying GIT_SSL_CAINFO and OPENSSL_CONF, so a similar solution to AIX 6.1 may work for z/OS as well.

For the problem with xlclang/xlclang++, we suspect that the error may be due to not being able to find the compiler dataset. The dataset should be defined in the xlclang.cfg file, could you try taking a look there? Running xlclang -v or xlclang++ -v should output the path to xlclang.cfg.

richardlau commented 3 years ago

@zsw007 this is the current cfg file:

UNIX1:/u/iojs/temp: >xlclang -v
FSUM0000I  Utility(xlclang)                  Level(D191018.Z2R4)
exec: export(export,CLC_CONFIG=/bin/../usr/lpp/cbclib/xlclang/etc/xlclang.cfg:clang,NULL)
exec: /bin/pg(/bin/pg,/bin/../usr/lpp/cbclib/xlclang/exe/default_msg/xlclangd.help,NULL)
exec: export(export,STEPLIB=CBC.SCLCCMP,NULL)
exec: export(export,_C89_ACCEPTABLE_RC=4,NULL)
exec: export(export,_C89_PVERSION=0x42030001,NULL)
exec: export(export,_C89_SYSDEFSD_PATH=,NULL)
exec: export(export,_C89_LXSYSIX=CEE.SCEELIB(CELHS003,CELHS001),NULL)
exec: export(export,_C89_LXSYSLIB=CEE.SCEEBND2:SYS1.CSSLIB,NULL)
FSUM7261 xterm-256color: Unknown terminal type, using dumb.

  z/OS V2.4.1 XL C/C++
...

UNIX1:/u/iojs/temp: >cat /usr/lpp/cbclib/xlclang/etc/xlclang.cfg
*
* FUNCTION: z/OS V2.4.1 XL C/C++ Compiler Configuration file
*
* Licensed Materials - Property of IBM
* 5650-ZOS Copyright IBM Corp. 2019.
* US Government Users Restricted Rights - Use, duplication or
* disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
*

* Clang C compiler
clang:        use               = DEFLT

* Clang C++ compiler
clang++:      use               = DEFLT
              options           = -D_XOPEN_SOURCE=600,-D__static_assert=static_assert,-Wno-parentheses,-Wno-unused-value

* common definitions
DEFLT:        cppcomp           = /usr/lpp/cbclib/xlclang/exe/clcdrvr
              ccomp             = /usr/lpp/cbclib/xlclang/exe/clcdrvr
              as                = /bin/c89
              ld_c              = /bin/c89
              ld_cpp            = /bin/cxx
              xlC               = /usr/lpp/cbclib/xlclang/bin/xlclang
              xlCcopt           = -D_XOPEN_SOURCE
              sysobj            = cee.sceeobj:cee.sceecpp
              syslib_x          = cee.sceebnd2:sys1.csslib
              exportlist_c_x    = cee.sceelib(celhs003,celhs001)
              exportlist_cpp_x  = cee.sceelib(celhs003,celhs001,celhscpp)
              exportlist_c_64   = cee.sceelib(celqs003)
              exportlist_cpp_64 = cee.sceelib(celqs003,celqscpp,cxxrt64)
              cinc              = -isystem/usr/include/le
              cppinc            = -isystem/usr/include/c++
              options           = -D_UNIX03_WITHDRAWN,-L/usr/lpp/cbclib/lib
              libraries         = -libmcmp
              steplib           = cbc.sclccmp
UNIX1:/u/iojs/temp: >

richardlau commented 3 years ago

For git and https URLs, pointing GIT_SSL_CAINFO to a local copy of https://github.com/nodejs/build/blob/master/ansible/aix61-standalone/resources/ca-bundle.crt works and fixes the SSL verification error. That copy of ca-bundle.crt dates from 2012, so we probably want a more up to date set of certificates. The comments in that file indicate that it was generated from Mozilla certificate data for/by the curl project. The latest version looks to be obtainable from https://curl.se/docs/caextract.html.

richardlau commented 3 years ago

@zsw007 / @IgorTodorovskiIBM It looks like the changes marist have made for us now enable zoslib and libuv to compile with xlclang/xlclang++ but the tests are failing, e.g. https://ci.nodejs.org/job/libuv-test-commit-zos-cmake-richardlau/27/. The output from the tests is garbled -- probably an ASCII/EBCDIC conversion issue. Is there a known issue when running with ctest?

The current job config looks like this:

#!/u/iojs/runAsciiBash
set -e
set -x

export PATH=/u/iojs/local/bin:$PATH

export RESOLVER_CONFIG=/u/iojs/resolver
export _C89_CCMODE=1
export _CXX_CCMODE=1
export _CC_CCMODE=1
# use xlclang/xlclang++
export CC=xlclang
export CXX=xlclang++
if [ "$verbose" = "true" ]; then
  export VERBOSE_OPTION=-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
  export VERBOSE_TEST_OPTION=--verbose
fi

# build zoslib
rm -rf zoslib
 /u/iojs/git-wrapper clone -c http.sslVerify=false  https://github.com/ibmruntimes/zoslib --depth 1
cd zoslib
mkdir out
cd out
cmake .. -DCMAKE_BUILD_TYPE=$BUILDTYPE $VERBOSE_OPTION -DCMAKE_INSTALL_PREFIX="$WORKSPACE/zoslib"
cmake --build . --target install

mkdir -p $WORKSPACE/out/cmake 
cd $WORKSPACE/out/cmake
cmake ../.. -DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=$BUILDTYPE -DZOSLIB_DIR="$WORKSPACE/zoslib/install" $VERBOSE_OPTION ../..
cmake --build .
ctest --output-on-failure $VERBOSE_TEST_OPTION

zsw007 commented 3 years ago

We have been running the tests with ./build/uv_run_tests_a instead of ctest so far. I took a look at ctest and we are seeing the same problem internally as well. Is using ctest the preferred way to run the tests?

richardlau commented 3 years ago

I can change the job but it was what we were doing before.

richardlau commented 3 years ago

@zsw007 Are these failures expected?

https://ci.nodejs.org/job/libuv-test-commit-zos-cmake-richardlau/32/BUILDTYPE=Release,nodes=zos-s390x/console

not ok 113 - get_currentexe
# exit code 3
# Output from process `get_currentexe`:
# Assertion failed in /u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/test-get-currentexe.c on line 96: `r == 0` (-121 == 0)
#  1: 0x1AE5E2CC abort
#  2: 0x1AE14D66 run_test_get_currentexe+0x838 [S0W1:/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/test-get-currentexe.c:96]
#  3: 0x1AE10AB2 run_test_part+0x122 [S0W1:/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/runner.c:376]
#  4: 0x1B0ADADE main+0x10e [S0W1:/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/run-tests.c:82]
#  5: 0x1B0ADEFA CELQINIT+0x1aca [CELQINIT:]
# CEE5207E The signal SIGABRT was received.
...
not ok 198 - process_title
# exit code 3
# Output from process `process_title`:
# Assertion failed in /u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/test-process-title.c on line 40: strcmp(buffer, title) == 0
#  1: 0x1AE9E724 abort
#  2: 0x1AE14D66 run_test_process_title+0x2b8 [S0W1:/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/test-process-title.c:40]
#  3: 0x1AE10AB2 run_test_part+0x122 [S0W1:/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/runner.c:376]
#  4: 0x1B0ADADE main+0x10e [S0W1:/u/iojs/build/workspace/libuv-test-commit-zos-cmake-richardlau/BUILDTYPE/Release/nodes/zos-s390x/test/run-tests.c:82]
#  5: 0x1B0ADEFA CELQINIT+0x1aca [CELQINIT:]
# CEE5207E The signal SIGABRT was received.

zsw007 commented 3 years ago

@zsw007 Are these failures expected?

https://ci.nodejs.org/job/libuv-test-commit-zos-cmake-richardlau/32/BUILDTYPE=Release,nodes=zos-s390x/console

Yes, we were seeing these two failures internally without our changes as well. We have changes prepared to fix these two failures, but we wanted to avoid making the first PR too large so we didn't include them there. Would it be preferred to include the fix for the failures in the first PR, or submit a separate PR?

richardlau commented 3 years ago

@zsw007 Are these failures expected? https://ci.nodejs.org/job/libuv-test-commit-zos-cmake-richardlau/32/BUILDTYPE=Release,nodes=zos-s390x/console

Yes, we were seeing these two failures internally without our changes as well. We have changes prepared to fix these two failures, but we wanted to avoid making the first PR too large so we didn't include them there. Would it be preferred to include the fix for the failures in the first PR, or submit a separate PR?

How much extra are the fixes? I'm asking because I'm going to have to do some coordination to land both the PR but also switch the Jenkins job configuration at the same time as unfortunately building with xlclang/xlclang++ is broken on the current libuv without the PR that introduces zoslib (src/unix/os390.c:32:10: fatal error: 'csrsic.h' file not found, https://ci.nodejs.org/job/libuv-test-commit-zos-cmake-richardlau/33/BUILDTYPE=Debug,nodes=zos-s390x/console). Currently https://ci.nodejs.org/job/libuv-test-commit-zos-cmake/ (the z/OS libuv job without any changes) is generally green -- switching the job configuration is going to make the z/OS libuv builds fail.

My first preference would be if we could keep the builds passing but a short period of it failing may be acceptable, i.e. we land the current PR as-is with an issue/another PR raised to address the failures.

zsw007 commented 3 years ago

The fix would provide separate z/OS implementation of uv_setup_args(), uv_set_process_title(), uv_get_process_title(), and uv__process_title_cleanup() similar to the approach used with AIX. I can go ahead and include the fix in https://github.com/libuv/libuv/pull/3060 to keep the build passing.

richardlau commented 3 years ago

We have a clean CI in https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake-richardlau/71/ so we just need to coordinate updating the https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake job with landing the PR.

richardlau commented 3 years ago

Changes have been applied and https://github.com/libuv/libuv/pull/3060 merged. Today's z/OS libuv build is green: https://ci.nodejs.org/view/libuv/job/libuv-test-commit-zos-cmake/683/

nodejs / build

z/OS changes for https://github.com/libuv/libuv/pull/3060/ #2491

Cloning issue

Compiler issues