search

som-shahlab / ehr_ml

Code for doing machine learning with various EHRs
MIT License
21 stars 3 forks source link

Unable to install ehr_ml #1

Closed guolin1 closed 3 years ago

guolin1 commented 3 years ago
Hi,

I ran into "Build failed" while attempting to install ehr_ml following your guide on 'https://som-shahlab.github.io/ehr_ml/setup.html'.

ERROR: Command errored out with exit status 1: /root/.cache/pypoetry/virtualenvs/ehr-ml-YgABPiuD-py3.8/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/ehr_ml/setup.py'"'"'; file='"'"'/usr/ehr_ml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

at /opt/conda/lib/python3.8/site-packages/poetry/utils/env.py:1074 in _run 1070│ output = subprocess.checkoutput( 1071│ cmd, stderr=subprocess.STDOUT, **kwargs 1072│ ) 1073│ except CalledProcessError as e: → 1074│ raise EnvCommandError(e, input=input) 1075│ 1076│ return decode(output) 1077│ 1078│ def execute(self, bin, *args, **kwargs):

Below are the steps to reproduce (see end for conda installed packages):

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster

Output:

Creating virtualenv ehr-ml-81KIsM5H-py3.8 in /root/.cache/pypoetry/virtualenvs Installing dependencies from lock file

Package operations: 58 installs, 0 updates, 0 removals

• Installing certifi (2020.11.8) • Installing chardet (3.0.4) • Installing idna (2.10) • Installing markupsafe (1.1.1) • Installing pyparsing (2.4.7) • Installing pytz (2020.4) • Installing six (1.15.0) • Installing urllib3 (1.26.2) • Installing alabaster (0.7.12) • Installing babel (2.9.0) • Installing docutils (0.16) • Installing imagesize (1.2.0) • Installing jinja2 (2.11.2) • Installing lazy-object-proxy (1.4.3) • Installing mccabe (0.6.1) • Installing numpy (1.19.1) • Installing packaging (20.4) • Installing pycodestyle (2.6.0) • Installing pyflakes (2.2.0) • Installing pygments (2.7.2) • Installing requests (2.25.0) • Installing snowballstemmer (2.0.0) • Installing sphinxcontrib-applehelp (1.0.2) • Installing sphinxcontrib-devhelp (1.0.2) • Installing sphinxcontrib-htmlhelp (1.0.3) • Installing sphinxcontrib-jsmath (1.0.1) • Installing sphinxcontrib-qthelp (1.0.3) • Installing sphinxcontrib-serializinghtml (1.1.4) • Installing wrapt (1.12.1) • Installing appdirs (1.4.4) • Installing astroid (2.4.2) • Installing attrs (20.3.0) • Installing click (7.1.2) • Installing flake8 (3.8.4) • Installing joblib (0.17.0) • Installing more-itertools (8.6.0) • Installing mypy-extensions (0.4.3) • Installing pathspec (0.8.1) • Installing pluggy (0.13.1) • Installing py (1.9.0) • Installing pyyaml (5.3.1) • Installing regex (2020.11.11) • Installing scipy (1.5.4) • Installing sphinx (3.3.1) • Installing threadpoolctl (2.1.0) • Installing toml (0.10.2) • Installing typed-ast (1.4.1) • Installing typing-extensions (3.7.4.3) • Installing unidecode (1.1.1) • Installing wcwidth (0.2.5) • Installing black (19.10b0) • Installing flake8-future-import (0.4.6) • Installing isort (5.6.4) • Installing mypy (0.782) • Installing pytest (5.4.3) • Installing scikit-learn (0.23.2) • Installing sphinx-autoapi (1.5.1) • Installing sphinx-rtd-theme (0.5.0)

Installing the current project: ehr_ml (0.1.0) EnvCommandError

Command ['/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin/pip', 'install', '-e', '/home/lawrence/ehr_ml', '--no-deps'] errored with the following return code 1, and output: Obtaining file:///home/lawrence/ehr_ml Installing collected packages: ehr-ml Running setup.py develop for ehr-ml ERROR: Command errored out with exit status 1: command: /root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/lawrence/ehr_ml/setup.py'"'"'; file='"'"'/home/lawrence/ehr_ml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps cwd: /home/lawrence/ehr_ml/ Complete output (180 lines): running develop running egg_info creating ehr_ml.egg-info writing ehr_ml.egg-info/PKG-INFO writing dependency_links to ehr_ml.egg-info/dependency_links.txt writing entry points to ehr_ml.egg-info/entry_points.txt writing requirements to ehr_ml.egg-info/requires.txt writing top-level names to ehr_ml.egg-info/top_level.txt writing manifest file 'ehr_ml.egg-info/SOURCES.txt' reading manifest file 'ehr_ml.egg-info/SOURCES.txt' writing manifest file 'ehr_ml.egg-info/SOURCES.txt' running build_ext Extracting Bazel installation... Starting local Bazel server and connecting to it... Loading: Loading: 0 packages loaded INFO: SHA256 (https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip) = 3e6b0691fc57db8217d535393dcc2cf7c1d39fc87e9adb6e7d7bab1483915110 DEBUG: Rule 'rules_foreign_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "3e6b0691fc57db8217d535393dcc2cf7c1d39fc87e9adb6e7d7bab1483915110" DEBUG: Repository rules_foreign_cc instantiated at: no stack (--record_rule_instantiation_callstack not enabled) Repository rule http_archive defined at: /root/.cache/bazel/_bazel_root/b8e4070296da67f000f3840fb7b9b7a3/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in Loading: 0 packages loaded Analyzing: target //:extension.so (1 packages loaded) Analyzing: target //:extension.so (1 packages loaded, 0 targets configured) Analyzing: target //:extension.so (16 packages loaded, 35 targets configured) Analyzing: target //:extension.so (16 packages loaded, 35 targets configured) Analyzing: target //:extension.so (39 packages loaded, 577 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (41 packages loaded, 732 targets configured) Analyzing: target //:extension.so (42 packages loaded, 794 targets configured) Analyzing: target //:extension.so (42 packages loaded, 794 targets configured) Analyzing: target //:extension.so (42 packages loaded, 794 targets configured) Analyzing: target //:extension.so (42 packages loaded, 794 targets configured) INFO: Analyzed target //:extension.so (47 packages loaded, 5926 targets configured). INFO: Found 1 target... [0 / 89] [Prepa] BazelWorkspaceStatusAction stable-status.txt ... (2 actions, 0 running) ERROR: /home/lawrence/ehr_ml/native/BUILD:43:15: error executing shell command: '/bin/bash -c #!/usr/bin/env bash function cleanup_function() { local ecode=$? if [ $ecode -eq 0 ]; then cleanup_on_success else cleanup_on_failure fi } set -e function cleanup_on_success() { printf...' failed (Exit 1) bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)

  Use --sandbox_debug to see verbose messages from the sandbox

  rules_foreign_cc: Build failed!
  rules_foreign_cc: Keeping temp build directory  and dependencies directory  for debug.
  rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify '--sandbox_debug' Bazel command line flag.

  rules_foreign_cc: Printing build logs:

  _____ BEGIN BUILD LOGS _____
  Bazel external C/C++ Rules #0.0.8. Building library 'jansson'
  Environment:______________
  BUILD_SCRIPT=bazel-out/k8-opt/bin/jansson/logs/CMake_script.sh
  EXT_BUILD_ROOT=/root/.cache/bazel/_bazel_root/b8e4070296da67f000f3840fb7b9b7a3/sandbox/processwrapper-sandbox/22/execroot/__main__
  BUILD_LOG=bazel-out/k8-opt/bin/jansson/logs/CMake.log
  PWD=/root/.cache/bazel/_bazel_root/b8e4070296da67f000f3840fb7b9b7a3/sandbox/processwrapper-sandbox/22/execroot/__main__
  TMPDIR=/tmp
  EXT_BUILD_DEPS=/tmp/tmp.X01nbwud8N
  BUILD_TMPDIR=/tmp/tmp.yrEjmmmZjG
  SHLVL=2
  INSTALLDIR=/root/.cache/bazel/_bazel_root/b8e4070296da67f000f3840fb7b9b7a3/sandbox/processwrapper-sandbox/22/execroot/__main__/bazel-out/k8-opt/bin/jansson
  PATH=/root/.cache/bazel/_bazel_root/b8e4070296da67f000f3840fb7b9b7a3/sandbox/processwrapper-sandbox/22/execroot/__main__:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin:/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  _=/usr/bin/env
  __________________________
  CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
  -- Configuring incomplete, errors occurred!
  See also "/tmp/tmp.yrEjmmmZjG/CMakeFiles/CMakeOutput.log".

  _____ END BUILD LOGS _____
  Printing build script:

  _____ BEGIN BUILD SCRIPT _____
  #!/usr/bin/env bash
  function children_to_path() {
  if [ -d $EXT_BUILD_DEPS/bin ]; then
  local tools=$(find $EXT_BUILD_DEPS/bin -maxdepth 1 -mindepth 1)
  for tool in $tools;
  do
  if  [[ -d "$tool" ]] || [[ -L "$tool" ]]; then
  export PATH=$PATH:$tool
  fi
  done
  fi
  }
  function replace_in_files() {
  if [ -d "$1" ]; then
  find -L $1 -type f   \( -name "*.pc" -or -name "*.la" -or -name "*-config" -or -name "*.cmake" \)   -exec sed -i 's@'"$2"'@'"$3"'@g' {} ';'
  fi
  }
  printf ""
  printf "Bazel external C/C++ Rules #0.0.8. Building library 'jansson'\n"
  printf ""
  set -e

  export EXT_BUILD_ROOT=$(pwd)
  export BUILD_TMPDIR=$(mktemp -d)
  export EXT_BUILD_DEPS=$(mktemp -d)
  export INSTALLDIR=$EXT_BUILD_ROOT/bazel-out/k8-opt/bin/jansson
  export PATH="$EXT_BUILD_ROOT:$PATH"
  mkdir -p $INSTALLDIR
  printf "Environment:______________\n"
  env
  printf "__________________________\n"
  children_to_path $EXT_BUILD_DEPS/bin
  export PATH="$EXT_BUILD_DEPS/bin:$PATH"
  cd $BUILD_TMPDIR
  export INSTALL_PREFIX="jansson"
  CC="/opt/conda/bin/x86_64-conda_cos6-linux-gnu-cc" CXX="/opt/conda/bin/x86_64-conda_cos6-linux-gnu-cc" CFLAGS="-U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=\"redacted\" -D__TIMESTAMP__=\"redacted\" -D__TIME__=\"redacted\" -march=native" CXXFLAGS="-U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -std=c++0x -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=\"redacted\" -D__TIMESTAMP__=\"redacted\" -D__TIME__=\"redacted\" -march=native -std=c++17" ASMFLAGS="-U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=\"redacted\" -D__TIMESTAMP__=\"redacted\" -D__TIME__=\"redacted\" -march=native" cmake -DCMAKE_AR="/usr/bin/ar" -DCMAKE_SHARED_LINKER_FLAGS="-shared -fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/opt/conda/bin -pass-exit-codes -lstdc++ -lm -Wl,--gc-sections" -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/opt/conda/bin -pass-exit-codes -lstdc++ -lm -Wl,--gc-sections" -DCMAKE_POSITION_INDEPENDENT_CODE="on" -DCMAKE_PREFIX_PATH="$EXT_BUILD_DEPS" -DCMAKE_INSTALL_PREFIX="$INSTALL_PREFIX" -DCMAKE_BUILD_TYPE="Release" -DCMAKE_RANLIB=""  $EXT_BUILD_ROOT/external/source_jansson
  make
  make install
  cp -L -r --no-target-directory "$BUILD_TMPDIR/$INSTALL_PREFIX" "$INSTALLDIR"
  replace_in_files $INSTALLDIR $BUILD_TMPDIR \${EXT_BUILD_DEPS}
  replace_in_files $INSTALLDIR $EXT_BUILD_DEPS \${EXT_BUILD_DEPS}
  mkdir -p $EXT_BUILD_ROOT/bazel-out/k8-opt/bin/copy_jansson/jansson
  cp -L -r --no-target-directory "$INSTALLDIR" "$EXT_BUILD_ROOT/bazel-out/k8-opt/bin/copy_jansson/jansson"
  touch $EXT_BUILD_ROOT/bazel-out/k8-opt/bin/empty_jansson.txt
  cd $EXT_BUILD_ROOT
  _____ END BUILD SCRIPT _____
  rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/jansson/logs/CMake_script.sh
  rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/jansson/logs/CMake.log

  Target //:extension.so failed to build
  Use --verbose_failures to see the command lines of failed build steps.
  INFO: Elapsed time: 141.109s, Critical Path: 1.42s
  INFO: 31 processes: 20 internal, 11 processwrapper-sandbox.
  FAILED: Build did NOT complete successfully
  FAILED: Build did NOT complete successfully
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/home/lawrence/ehr_ml/setup.py", line 44, in <module>
      setup(**setup_kwargs)
    File "/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/opt/conda/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/conda/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/conda/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
      self.install_for_development()
    File "/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/lib/python3.8/site-packages/setuptools/command/develop.py", line 136, in install_for_development
      self.run_command('build_ext')
    File "/opt/conda/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
      _build_ext.run(self)
    File "/opt/conda/lib/python3.8/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/home/lawrence/ehr_ml/build.py", line 72, in build_extensions
      subprocess.run(
    File "/opt/conda/lib/python3.8/subprocess.py", line 512, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['bazel', 'build', '-c', 'opt', 'extension.so']' returned non-zero exit status 1.
  ----------------------------------------

ERROR: Command errored out with exit status 1: /root/.cache/pypoetry/virtualenvs/ehr-ml-81KIsM5H-py3.8/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/lawrence/ehr_ml/setup.py'"'"'; file='"'"'/home/lawrence/ehr_ml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

at /opt/conda/lib/python3.8/site-packages/poetry/utils/env.py:1074 in _run 1070│ output = subprocess.checkoutput( 1071│ cmd, stderr=subprocess.STDOUT, **kwargs 1072│ ) 1073│ except CalledProcessError as e: → 1074│ raise EnvCommandError(e, input=input) 1075│ 1076│ return decode(output) 1077│ 1078│ def execute(self, bin, *args, **kwargs):

conda list

Name Version Build Channel _libgcc_mutex 0.1 main brotlipy 0.7.0 py38h7b6447c_1000 anaconda ca-certificates 2020.10.14 0 anaconda certifi 2020.6.20 py38_0 anaconda cffi 1.14.3 py38he30daa8_0 anaconda chardet 3.0.4 py38_1003 anaconda conda 4.9.0 py38_0 anaconda conda-package-handling 1.7.2 py38h03888b9_0 anaconda cryptography 3.1.1 py38h1ba5d50_0 anaconda idna 2.10 py_0 anaconda ld_impl_linux-64 2.33.1 h53a641e_7 libedit 3.1.20181209 hc058e9b_0 libffi 3.3 he6710b0_2 anaconda libgcc-ng 9.1.0 hdf63c60_0 libstdcxx-ng 9.1.0 hdf63c60_0 ncurses 6.2 he6710b0_0 openssl 1.1.1h h7b6447c_0 anaconda pip 20.2.4 py38_0 anaconda pycosat 0.6.3 py38h7b6447c_1 anaconda pycparser 2.20 py_2 anaconda pyopenssl 19.1.0 py_1 anaconda pysocks 1.7.1 py38_0 anaconda python 3.8.3 hcff3b4d_0 anaconda readline 8.0 h7b6447c_0 anaconda requests 2.24.0 py_0 anaconda ruamel_yaml 0.15.87 py38h7b6447c_0 anaconda setuptools 50.3.0 py38hb0f4dca_1 anaconda six 1.15.0 py_0 anaconda sqlite 3.31.1 h7b6447c_0 tk 8.6.8 hbc83047_0 tqdm 4.42.1 py_0 urllib3 1.25.11 py_0 anaconda wheel 0.35.1 py_0 anaconda xz 5.2.5 h7b6447c_0 anaconda yaml 0.1.7 had09818_2 zlib 1.2.11 h7b6447c_3

EthanSteinberg commented 3 years ago

Wow, thanks for the detailed bug report! I'll try reproducing this tomorrow once I get access to a machine with Docker. Looks like CMake failed to detect both the compiler and make executable.

guolin1 commented 3 years ago

Awesome. Thank you!

EthanSteinberg commented 3 years ago

Hi @guolin1, I figured out what's going on. Bazel, the build system, has a bug that prevents it from working correctly with conda installed build tools.

I would recommend including your C++ compiler directly in your docker as a workaround for now. I tried that locally and it seems to work on my side.

I'll submit a pull request upstream to bazel later this week with the corresponding fix on their side (it's a subtle bug in their compiler detection code, it forgets to check the AR environmental variable).

Sorry for the trouble!

EthanSteinberg commented 3 years ago

https://github.com/bazelbuild/bazel/pull/12541 appears to fix this issue.