Unstable appveyor & circleci builds

wayneparrott commented 2 years ago

This is is a tracking issue for issues observed with Windows builds using Appveyor.

1) After many weeks of stable appveyor builds they recently began failing across all version of node due to a python environment error. The test suite is failing due to an error when compiling the rclnodejs test message. Here's a snippet of the error msgs during the build:

SET PATH=%PYTHON3%;%PYTHON3%\bin;%PYTHON3%\Scripts;C:\Program Files\CMake\bin;%PATH%
npm test
> rclnodejs@0.21.0 test C:\proj\rclnodejs
> node ./scripts/compile_tests.js && node --expose-gc ./scripts/run_test.js && npm run dtslint
Starting >>> cpp_nodes
Starting >>> rclnodejs_test_msgs
--- stderr: cpp_nodes
CMake Error in CMakeLists.txt:
  Imported target
  "example_interfaces::example_interfaces__rosidl_generator_c" includes
  non-existent path
    "C:\Python38\lib\site-packages\numpy\core\include"
  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
  * The path was deleted, renamed, or moved to another location.
  * An install or uninstall procedure did not complete successfully.
  * The installation package was faulty and references files it does not
  provide.
CMake Error in CMakeLists.txt:
  Imported target
  "example_interfaces::example_interfaces__rosidl_generator_c" includes
  non-existent path
    "C:\Python38\lib\site-packages\numpy\core\include"
  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
  * The path was deleted, renamed, or moved to another location.
  * An install or uninstall procedure did not complete successfully.
  * The installation package was faulty and references files it does not
  provide.

The error "C:\Python38\lib\site-packages\numpy\core\include" is a bad path. In this instance python 3.10 is setup on the PATH and the AMENT_PYTHON_EXECUTABLE. The python 3.10 environment has worked consistently as we prepared for the rclnodejs 0.21.0 release. This error appeared approx 2 days after the release. A compare of recent appveyor.yml commits does not reveal any change that can be associated with this failure.

2) appveyor - node16 build failing to install int64-napi Following is an error observed on both appveyor and my Windows 11 dev environment:

SET PATH=%PYTHON2%;%PYTHON2%\bin;%PYTHON2%\Scripts;%PATH%
node --version
v16.13.0
npm --version
8.1.0
python --version
Python 2.7.18
npm install
npm ERR! code 1
npm ERR! path C:\proj\rclnodejs\node_modules\int64-napi
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c node-gyp rebuild
npm ERR! gyp: C:\Users\appveyor\AppData\Local\node-gyp\Cache\16.13.0\common.gypi not found (cwd: C:\proj\rclnodejs\node_modules\int64-napi) while reading includes of binding.gyp while trying to load binding.gyp
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
npm ERR! gyp ERR! stack     at ChildProcess.onCpExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:353:16)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:390:28)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
npm ERR! gyp ERR! System Windows_NT 10.0.17763
npm ERR! gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd C:\proj\rclnodejs\node_modules\int64-napi
npm ERR! gyp ERR! node -v v16.13.0
npm ERR! gyp ERR! node-gyp -v v8.2.0
npm ERR! gyp ERR! not ok
npm ERR! A complete log of this run can be found in:

This is the key line:

npm ERR! gyp: C:\Users\appveyor\AppData\Local\node-gyp\Cache\16.13.0\common.gypi not found (cwd: C:\proj\rclnodejs\node_modules\int64-napi) while reading includes of binding.gyp while trying to load binding.gyp

The node-gyp\Cache\16.13.0 folder is missing the common.gypi file. On my dev environment I resolved this issue by uninstalling and then reinstalling node16 using nvm - the common.gypi file was then present. Early in the 0.21 dev cycle I observed this issue on appveyor-windows builds. Then it magically stopped appearing and builds have been successful until now when this environment-configuration error reappeared.

wayneparrott commented 2 years ago

CONCLUSION: there must be a recent change in the latest successful rolling distro that is causing colcon to look for python38 rather than use env vars to local the version of python to use. Thus windows (appveyor) and macOs (circleci) are breaking because the default environments provide python310 & python39 respectively. Linux builds (travis) are succeeding only because ubuntu 20.04 includes python 3.8 as part of its default toolchain.

wayneparrott commented 2 years ago

MacOS builds are failing for a similar reason as Windows build. The macos build is expecting python38 yet the ci environment has python 3.9 installed by default.

#!/bin/bash --login
source ~/ros2_install/ros2-osx/local_setup.bash && export OPENSSL_ROOT_DIR="/usr/local/opt/openssl" && node scripts/compile_tests.js && node --expose-gc ./node_modules/.bin/istanbul cover ./scripts/run_test.js --report lcovonly && cat ./coverage/lcov.info | ./node_modules/coveralls/bin/coveralls.js && rm -rf ./coverage
[rti_connext_dds_cmake_module][warning] No RTI Connext DDS installation specified.. RTI Connext DDS will not be available at runtime,unless you already configured DYLD_LIBRARY_PATH manually.
COMPILE TEST ENV:  {
  NVM_INC: '/Users/distiller/.nvm/versions/node/v16.13.2/include/node',
  CIRCLE_WORKFLOW_JOB_ID: 'cb3b8365-699a-4a25-88e0-68ab252dbf84',
  CIRCLE_BUILD_NUM: '1182',
  CIRCLE_PR_REPONAME: 'rclnodejs',
  CIRCLE_NODE_TOTAL: '1',
  CIRCLE_ARTIFACTS: '/tmp/circleci-artifacts',
  NVM_CD_FLAGS: '',
  GEM_HOME: '/Users/distiller/.gem/ruby/2.7.2',
  SHELL: '/bin/bash',
  CI_PULL_REQUESTS: 'https://github.com/RobotWebTools/rclnodejs/pull/835',
  TMPDIR: '/var/folders/6y/gy9gggt14379c_k39vwb50lc0000gn/T/',
  SSH_CLIENT: '18.210.28.174 44224 2222',
  CIRCLE_WORKFLOW_UPSTREAM_JOB_IDS: '',
  CIRCLE_INTERNAL_SCRATCH: '/var/folders/6y/gy9gggt14379c_k39vwb50lc0000gn/T/circleci-474677802',
  ROS_PYTHON_VERSION: '3',
  ROS_VERSION: '2',
  COLCON_PREFIX_PATH: '/Users/distiller/ros2_install/ros2-osx',
  CIRCLE_PROJECT_USERNAME: 'RobotWebTools',
  AMENT_PREFIX_PATH: '/Users/distiller/ros2_install/ros2-osx',
  SSH_TTY: '/dev/ttys000',
  CIRCLE_INTERNAL_TASK_DATA: '/tmp/.circleci-task-data-61e046a75a7949028de8acbd-0-build',
  NVM_DIR: '/Users/distiller/.nvm',
  USER: 'distiller',
  CIRCLE_NODE_INDEX: '0',
  CIRCLE_COMPARE_URL: '',
  CIRCLE_BRANCH: 'pull/835',
  SSH_AUTH_SOCK: '/var/folders/6y/gy9gggt14379c_k39vwb50lc0000gn/T/circleci-474677802/ssh_auth_sock',
  CIRCLE_PULL_REQUEST: 'https://github.com/RobotWebTools/rclnodejs/pull/835',
  CIRCLE_JOB: 'build_and_test',
  CIRCLE_PULL_REQUESTS: 'https://github.com/RobotWebTools/rclnodejs/pull/835',
  CI_PULL_REQUEST: 'https://github.com/RobotWebTools/rclnodejs/pull/835',
  CIRCLE_WORKING_DIRECTORY: '~/RobotWebTools/rclnodejs',
  CIRCLE_PR_USERNAME: 'wayneparrott',
  PATH: '/Users/distiller/ros2_install/ros2-osx/bin:/Users/distiller/.nvm/versions/node/v16.13.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/distiller/.gem/ruby/2.7.2/bin:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0/bin:/Users/distiller/.rubies/ruby-2.7.2/bin',
  MAIL: '/var/mail/distiller',
  CIRCLE_WORKFLOW_WORKSPACE_ID: 'f1f69a3a-5949-44a5-a7f4-839a57c721a5',
  CIRCLE_USERNAME: 'wayneparrott',
  CIRCLE_PREVIOUS_BUILD_NUM: '',
  PWD: '/Users/distiller/RobotWebTools/rclnodejs',
  CIRCLE_STAGE: 'build_and_test',
  CIRCLE_PR_NUMBER: '835',
  RUBY_ENGINE: 'ruby',
  CIRCLE_BUILD_URL: 'https://circleci.com/gh/RobotWebTools/rclnodejs/1182',
  CIRCLE_WORKFLOW_ID: 'f1f69a3a-5949-44a5-a7f4-839a57c721a5',
  CIRCLECI: 'true',
  OPENSSL_ROOT_DIR: '/usr/local/opt/openssl',
  SHLVL: '2',
  HOME: '/Users/distiller',
  ROS_DISTRO: 'rolling',
  GEM_ROOT: '/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0',
  CIRCLE_SHA1: 'b26ebf377579e2a352f2aa7edaee941e9b1e1417',
  CI: 'true',
  DYLD_LIBRARY_PATH: '/Users/distiller/ros2_install/ros2-osx/opt/yaml_cpp_vendor/lib:/Users/distiller/ros2_install/ros2-osx/opt/rviz_ogre_vendor/lib:/Users/distiller/ros2_install/ros2-osx/lib',
  CIRCLE_REPOSITORY_URL: 'git@github.com:RobotWebTools/rclnodejs.git',
  CIRCLE_INTERNAL_CONFIG: '/tmp/61e046a75a7949028de8acbd-0-build/circleci-runner-config.json',
  BASH_ENV: '/tmp/.bash_env-61e046a75a7949028de8acbd-0-build',
  PYTHONPATH: '/Users/distiller/ros2_install/ros2-osx/lib/python3.8/site-packages',
  LOGNAME: 'distiller',
  CIRCLE_SHELL_ENV: '/tmp/.bash_env-61e046a75a7949028de8acbd-0-build',
  GEM_PATH: '/Users/distiller/.gem/ruby/2.7.2:/Users/distiller/.rubies/ruby-2.7.2/lib/ruby/gems/2.7.0',
  SSH_CONNECTION: '18.210.28.174 44224 162.252.208.120 2222',
  LC_CTYPE: 'en_US.UTF-8',
  PKG_CONFIG_PATH: '/Users/distiller/ros2_install/ros2-osx/lib/pkgconfig',
  NVM_BIN: '/Users/distiller/.nvm/versions/node/v16.13.2/bin',
  RUBY_ROOT: '/Users/distiller/.rubies/ruby-2.7.2',
  CMAKE_PREFIX_PATH: '/Users/distiller/ros2_install/ros2-osx',
  ROS_LOCALHOST_ONLY: '0',
  RUBYOPT: '',
  CIRCLE_TEST_REPORTS: '/tmp/circleci-test-results',
  CIRCLE_PROJECT_REPONAME: 'rclnodejs',
  RUBY_VERSION: '2.7.2',
  _: '/Users/distiller/.nvm/versions/node/v16.13.2/bin/node',
  __CF_USER_TEXT_ENCODING: '0x1F5:0x0:0x0'
}
Starting >>> cpp_nodes

Starting >>> rclnodejs_test_msgs

--- stderr: cpp_nodes
CMake Error in CMakeLists.txt:
  Imported target
  "example_interfaces::example_interfaces__rosidl_generator_c" includes
  non-existent path

    "/usr/local/Cellar/python@3.8/3.8.2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

minggangw commented 2 years ago

The python path issue has a looong history, sometimes, it seems that it was hardcoded. It's not the fault of rclnodejs or the CIs, usually, the problem will be gone in some following release (rolling release).

minggangw commented 2 years ago

I think we can close this issue as we migrated to actions.

RobotWebTools / rclnodejs

Unstable appveyor & circleci builds #834