Pip install in new environment fails

nilsvu commented 1 year ago

Installing charm4py in a new environment fails on my macOS 12.5 machine with Python 3.9.13:

$ python3 -m pip install charm4py
Collecting charm4py
  Using cached charm4py-1.0.tar.gz (3.4 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [37 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      Traceback (most recent call last):
        File "/private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-80ig3b34/charm4py_33d4cf448d404cbcb787ec860eaf1109/setup.py", line 33, in <module>
          charm4py_version = subprocess.check_output(['git', 'describe']).rstrip().decode().split('-')[0]
        File "/usr/local/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
        File "/usr/local/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['git', 'describe']' returned non-zero exit status 128.

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-80ig3b34/charm4py_33d4cf448d404cbcb787ec860eaf1109/charm4py/__init__.py", line 7, in <module>
          import greenlet
      ModuleNotFoundError: No module named 'greenlet'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-80ig3b34/charm4py_33d4cf448d404cbcb787ec860eaf1109/setup.py", line 42, in <module>
          from charm4py import _version
        File "/private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-80ig3b34/charm4py_33d4cf448d404cbcb787ec860eaf1109/charm4py/__init__.py", line 10, in <module>
          exit(-1)
        File "/usr/local/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/_sitebuiltins.py", line 26, in __call__
          raise SystemExit(code)
      SystemExit: -1

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-80ig3b34/charm4py_33d4cf448d404cbcb787ec860eaf1109/setup.py", line 45, in <module>
          raise DistutilsSetupError('Could not determine Charm4py version')
      distutils.errors.DistutilsSetupError: Could not determine Charm4py version
      Charm4py requires the greenlet package. It can be installed via pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Any ideas?

matthiasdiener commented 1 year ago

This appears to happen when a non-git charm4py is being installed before the greenlet package is installed. As a workaround, can you try installing python3 -m pip install greenlet before installing charmp4y?

nilsvu commented 1 year ago

After installing greenlet:

$ pip install greenlet
Collecting greenlet
  Downloading greenlet-1.1.2-cp39-cp39-macosx_10_14_x86_64.whl (92 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.8/92.8 kB 1.2 MB/s eta 0:00:00
Installing collected packages: greenlet
Successfully installed greenlet-1.1.2

$ pip install charm4py
Collecting charm4py
  Using cached charm4py-1.0.tar.gz (3.4 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-q3zl3jal/charm4py_20d90b467baf4b1eb1db167c0b055fc2/setup.py", line 260, in <module>
          from Cython.Build import cythonize
      ModuleNotFoundError: No module named 'Cython'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Then after installing cython:

$ pip install cython
Collecting cython
  Downloading Cython-0.29.32-py2.py3-none-any.whl (986 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 986.3/986.3 kB 1.7 MB/s eta 0:00:00
Installing collected packages: cython
Successfully installed cython-0.29.32

$ pip install charm4py
Collecting charm4py
  Using cached charm4py-1.0.tar.gz (3.4 MB)
  Preparing metadata (setup.py) ... done
Collecting numpy>=1.10.0
  Downloading numpy-1.23.2-cp39-cp39-macosx_10_9_x86_64.whl (18.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.1/18.1 MB 2.4 MB/s eta 0:00:00
Requirement already satisfied: greenlet in ./env/lib/python3.9/site-packages (from charm4py) (1.1.2)
Building wheels for collected packages: charm4py
  Building wheel for charm4py (setup.py) ... error
  error: subprocess-exited-with-error

  × Running setup.py install for charm4py did not run successfully.
  │ exit code: 1
  ╰─> [188 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      Compiling charm4py/charmlib/charmlib_cython.pyx because it depends on /Users/nlf/Projects/pyspectre/env/lib/python3.9/site-packages/numpy/__init__.pxd.
      [1/1] Cythonizing charm4py/charmlib/charmlib_cython.pyx
      /Users/nlf/Projects/pyspectre/env/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-tjvv0x1l/charm4py_dc69ade22a524497b2a1a609ff032306/charm4py/charmlib/charmlib_cython.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      running install
      /Users/nlf/Projects/pyspectre/env/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-12-x86_64-cpython-39
      creating build/lib.macosx-12-x86_64-cpython-39/charm4py
      creating build/lib.macosx-12-x86_64-cpython-39/charm4py/.libs
      creating /private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-tjvv0x1l/charm4py_dc69ade22a524497b2a1a609ff032306/charm4py/.libs
      creating build/lib.macosx-12-x86_64-cpython-39/charmrun
      Uncompressing charm.tar.gz...
      Selected Compiler:
      Selected Options:  tcp
      Creating dir: netlrts-darwin-x86_64-tcp
      Creating dir: netlrts-darwin-x86_64-tcp/tmp
      Soft-linking over bin
      Soft-linking over lib
      Soft-linking over include
      Soft-linking over tmp
      Copying src/scripts/Makefile to netlrts-darwin-x86_64-tcp/tmp
      Generating netlrts-darwin-x86_64-tcp/tmp/conv-mach-pre.sh
      Generating netlrts-darwin-x86_64-tcp/tmp/conv-mach-opt.h, conv-mach-opt.sh, conv-mach-opt.mak
      Performing 'make -j 6 basics OPTS=-optimize -production QUIET= CONFIG_OPTS=--disable-controlpoint --disable-tracing --disable-tracing-commthread --disable-charmdebug --disable-replay --disable-error-checking --disable-stats --enable-charmpy' in netlrts-darwin-x86_64-tcp/tmp
      pwd
      /private/var/folders/jb/8vbfr0cn737bzhdszgdjc2rc0000gp/T/pip-install-tjvv0x1l/charm4py_dc69ade22a524497b2a1a609ff032306/charm_src/charm/netlrts-darwin-x86_64-tcp/tmp
      find . -type l -exec rm {} \;
      rm -rf QuickThreads
      rm -rf hwloc
      rm -rf libs
      rm -rf ../bin ; mkdir ../bin
      if case netlrts-darwin-x86_64 in *win64*) true;; *-win-*) true;; *) false;; esac; then \
                  cp -f ../../src/arch/win/unix2nt* ../bin; \
                  cp -f ../../src/arch/win/createlink.exe ../bin; \
          fi
      rm -rf ../lib ; mkdir ../lib
      rm -rf ../benchmarks
      rm -rf ../examples
      rm -rf ../tests
      ../../src/scripts/gatherflat ../../src/scripts          .
      ./gatherflat ../../src/conv-core                        .
      ./gatherflat ../../src/conv-ldb                         .
      ./gatherflat ../../src/conv-ccs                         .
      ./gatherflat ../../src/conv-perf                        .
      ./gatherflat ../../src/ck-core                          .
      ./gatherflat ../../src/ck-perf                          .
      ./gatherflat ../../src/ck-pics                          .
      ./gatherflat ../../src/ck-ldb                           .
      ./gatherflat ../../src/ck-cp                            .
      ./gatherflat ../../src/langs/simplemsg                  .
      ./gatherflat ../../src/langs/pvmc                       .
      ./gatherflat ../../src/langs/bluegene                   .
      ./gatherflat ../../src/langs/f90charm                   .
      ./gatherflat ../../src/xlatcpm                          .
      ./gathertree ../../src/QuickThreads                     QuickThreads
      ./gathertree ../../src/../contrib/hwloc                 hwloc
      ./gathertree ../../src/libs                             libs
      ./gathertree ../../src/arch/util                        .
      ./gathertree ../../src/arch/util/proc_management        proc_management
      ./gathertree ../../src/util                             .
      ./gathertree ../../src/langs                            langs
      ./gathertree ../../src/langs/jade                       langs/jade
      ./gathertree ../../src/arch/common                      .
      ./gathertree ../../src/arch/netlrts                 .
      test -f ../../src/arch/netlrts/gdir_link && cat ../../src/arch/netlrts/gdir_link > .gdir.new && ./gathertree ../../src/arch/`cat .gdir.new`      . || true
      ./gatherflat ../../src/arch/netlrts-darwin-x86_64                 .
      ./gathertree ../../src/../benchmarks                    ../benchmarks
      ./gathertree ../../src/../examples                      ../examples
      ./gathertree ../../src/../tests                         ../tests
      rm -f ../bin/dep.pl ; cp dep.pl ../bin/
      chmod +x charmc
      ./system_ln  ../tmp/charmc ../bin/
      ./system_ln  ../../src/scripts/testrun ../bin/
      ./system_ln  ../tmp/tree.txt ../bin/
      ./system_ln  ../tmp/fuzzytree.txt ../bin/
      rm -rf ../include ; mkdir ../include
      ./system_ln  ../tmp/conv-*.*h ../include
      ./system_ln  ../tmp/cc-*.*h ../include
      ./system_ln  ../tmp/conv-mach-opt.sh ../include
      ./system_ln  ../tmp/conv-mach-opt.mak ../include
      if [ -x ./special.sh ] ; then SRCBASE=../../src ./special.sh ; fi
      if [ ! -f conv-common.h ] ; then ( touch conv-common.h ) ; fi
      touch dirs+sources
      /Library/Developer/CommandLineTools/usr/bin/make -C topomanager -f Makefile_charm headers
      /bin/cp TopoManager.h ../../include
      /bin/cp BGQTorus.h ../../include
      /bin/cp XTTorus.h ../../include
      echo "//empty" > topomanager_config.h
      /bin/cp topomanager_config.h ../../include
      /Library/Developer/CommandLineTools/usr/bin/make charmxi
      autoreconf
      autoheader
      configure.ac:9: warning: The macro `AC_CONFIG_HEADER' is obsolete.
      configure.ac:9: You should run autoupdate.
      ./lib/autoconf/status.m4:719: AC_CONFIG_HEADER is expanded from...
      configure.ac:9: the top level
      configure.ac:2994: warning: The macro `AC_TRY_COMPILE' is obsolete.
      configure.ac:2994: You should run autoupdate.
      ./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from...
      lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
      lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
      ./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
      hwloc/config/hwloc_check_attributes.m4:174: _HWLOC_CHECK_ATTRIBUTES is expanded from...
      hwloc/config/hwloc.m4:25: HWLOC_SETUP_CORE is expanded from...
      configure.ac:2994: the top level
      configure.ac:2994: warning: The macro `AC_TRY_COMPILE' is obsolete.
      configure.ac:2994: You should run autoupdate.
      ./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from...
      lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
      lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
      ./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
      hwloc/config/hwloc_check_attributes.m4:93: _HWLOC_CHECK_SPECIFIC_ATTRIBUTE is expanded from...
      hwloc/config/hwloc_check_attributes.m4:174: _HWLOC_CHECK_ATTRIBUTES is expanded from...
      hwloc/config/hwloc.m4:25: HWLOC_SETUP_CORE is expanded from...
      configure.ac:2994: the top level
      configure.ac:2994: warning: The macro `AC_HELP_STRING' is obsolete.
      configure.ac:2994: You should run autoupdate.
      ./lib/autoconf/general.m4:204: AC_HELP_STRING is expanded from...
      hwloc/config/hwloc_check_visibility.m4:57: _HWLOC_CHECK_VISIBILITY is expanded from...
      hwloc/config/hwloc.m4:25: HWLOC_SETUP_CORE is expanded from...
      configure.ac:2994: the top level
      configure.ac:2994: warning: The macro `AC_HAVE_HEADERS' is obsolete.
      configure.ac:2994: You should run autoupdate.
      ./lib/autoconf/oldnames.m4:35: AC_HAVE_HEADERS is expanded from...
      hwloc/config/hwloc.m4:25: HWLOC_SETUP_CORE is expanded from...
      configure.ac:2994: the top level
      configure.ac:2994: warning: The macro `AC_TRY_LINK' is obsolete.
      configure.ac:2994: You should run autoupdate.
      ./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
      hwloc/config/hwloc.m4:25: HWLOC_SETUP_CORE is expanded from...
      configure.ac:2994: the top level
      configure.ac:2994: warning: The macro `AC_HELP_STRING' is obsolete.
      configure.ac:2994: You should run autoupdate.
      ./lib/autoconf/general.m4:204: AC_HELP_STRING is expanded from...
      hwloc/config/hwloc.m4:25: HWLOC_SETUP_CORE is expanded from...
      configure.ac:2994: the top level
      automake: warning: am_libhwloc_embedded_la_rpath was already defined in condition TRUE, which includes condition !HWLOC_BUILD_STANDALONE ...
      configure.ac:2972: ... 'am_libhwloc_embedded_la_rpath' previously defined here
      ./configure --disable-controlpoint --disable-tracing --disable-tracing-commthread --disable-charmdebug --disable-replay --disable-error-checking --disable-stats --enable-charmpy
      Error checking is disabled
      AMPI error checking is enabled
      Statistics collection is disabled
      Charm tracing is disabled
      Charm tracing communication thread is disabled
      Task Queue is disabled
      Drone mode is disabled
      CharmDebug is disabled
      Charm record/replay is disabled
      CCS is enabled
      Charm control point is disabled
      Charm LB user data is disabled
      Lockless queue for PE local and node queue is disabled
      Charm shrink expand is disabled
      charm4py support is enabled
      Setting load balancing timer type as 'double'
      checking machine name... netlrts-darwin-x86_64-tcp
      checking "cp command as"... cp -p
      checking "C++ compiler as"... "clang++  "
      checking "whether C++ compiler works"... "ok"
      checking "C++ linker as"... "clang++  "
      checking "whether linker works"... "ok"
      checking "Native C++ compiler as"... "clang++"
      checking "Sequential C++ compiler as"... "clang++"
      checking "whether compiler accept -fno-stack-protector"... "ok"
      checking "whether C++ compiler supports C++11 without flags"... "no"
      checking "whether C++ compiler supports C++11 with '-std=c++11'"... "no"
      checking "whether C++ compiler supports C++11 with '--c++11'"... "no"
      checking "whether C++ compiler supports C++11 with '-h std=c++11'"... "no"
      Charm++ requires C++11 support, but doesn't know the flag to enable it

      For Intel's compiler please see
      https://github.com/UIUC-PPL/charm/issues/1560
      about making a suitable version of gcc/g++/libstdc++ available

      For Blue Gene/Q please use the Clang compiler
      *** Please find detailed output in tmp/charmconfig.out ***
      make: *** [headers] Error 2
      -------------------------------------------------
      Charm++ NOT BUILT. Either cd into netlrts-darwin-x86_64-tcp/tmp and try
      to resolve the problems yourself, visit
      http://charm.cs.illinois.edu/
      for more information. Otherwise, email the developers at charm@cs.illinois.edu
      error: An error occured while building charm library
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> charm4py

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

matthiasdiener commented 1 year ago

Can you put the contents of tmp/charmconfig.out into a gist or so?

CABA1998 commented 4 months ago

I have the same problem.

charmplusplus / charm4py

Pip install in new environment fails #234