search

spinsphotonics / fdtdz

Fast, scalable, accessible photonic simulation
MIT License
110 stars 13 forks source link

Unable to build. Any ideas? #9

Closed flaport closed 1 year ago

flaport commented 1 year ago

Hi, Thanks for this exciting new library! I'd love to give it a try!

However, I was trying to build the library but i get the following errors:

~/Programming/fdtdz$  pip install .
Processing /home/flaport/Programming/fdtdz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting jax (from fdtdz==0.1.dev20+g67a1fe3.d20230505)
  Downloading jax-0.4.8.tar.gz (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 8.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting jaxlib (from fdtdz==0.1.dev20+g67a1fe3.d20230505)
  Downloading jaxlib-0.4.7-cp310-cp310-manylinux2014_x86_64.whl (66.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.6/190.6 kB 38.5 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.21 in /home/flaport/.anaconda/lib/python3.10/site-packages (from jax->fdtdz==0.1.dev20+g67a1fe3.d20230505) (1.24.3)
Collecting opt-einsum (from jax->fdtdz==0.1.dev20+g67a1fe3.d20230505)
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.5/65.5 kB 28.2 MB/s eta 0:00:00
Requirement already satisfied: scipy>=1.7 in /home/flaport/.anaconda/lib/python3.10/site-packages (from jax->fdtdz==0.1.dev20+g67a1fe3.d20230505) (1.10.1)
Building wheels for collected packages: fdtdz, jax
  Building wheel for fdtdz (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for fdtdz (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [980 lines of output]
      running bdist_wheel
      running build
      running build_py
      running egg_info
      writing src/fdtdz.egg-info/PKG-INFO
      writing dependency_links to src/fdtdz.egg-info/dependency_links.txt
      writing requirements to src/fdtdz.egg-info/requires.txt
      writing top-level names to src/fdtdz.egg-info/top_level.txt
      reading manifest template 'MANIFEST.in'
      warning: no directories found matching 'lib'
      warning: no previously-included files found matching '.*'
      no previously-included directories found matching '.github'
      adding license file 'LICENSE'
      writing manifest file 'src/fdtdz.egg-info/SOURCES.txt'
      /tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'fdtdz_jax.ptx' is absent from the `packages` configuration.
      !!

              ********************************************************************************
              ############################
              # Package would be ignored #
              ############################
              Python recognizes 'fdtdz_jax.ptx' as an importable package[^1],
              but it is absent from setuptools' `packages` configuration.

              This leads to an ambiguous overall configuration. If you want to distribute this
              package, please make sure that 'fdtdz_jax.ptx' is explicitly added
              to the `packages` configuration field.

              Alternatively, you can also rely on setuptools' discovery methods
              (for example by using `find_namespace_packages(...)`/`find_namespace:`
              instead of `find_packages(...)`/`find:`).

              You can read more about "package discovery" on setuptools documentation page:

              - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

              If you don't want 'fdtdz_jax.ptx' to be distributed and are
              already explicitly excluding 'fdtdz_jax.ptx' via
              `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
              you can try to use `exclude_package_data`, or `include-package-data=False` in
              combination with a more fine grained `package-data` configuration.

              You can read more about "package data files" on setuptools documentation page:

              - https://setuptools.pypa.io/en/latest/userguide/datafiles.html

              [^1]: For Python, any directory (with suitable naming) can be imported,
                    even if it does not contain any `.py` files.
                    On the other hand, currently there is no concept of package data
                    directory, all directories are treated like packages.
              ********************************************************************************

      !!
        check.warn(importable)
      running build_ext
      CMake Error at /tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCompilerId.cmake:751 (message):
        Compiling the CUDA compiler identification source file
        "CMakeCUDACompilerId.cu" failed.

        Compiler: /opt/cuda/bin/nvcc

        Build flags:

        Id flags: --keep;--keep-dir;tmp -v

        The output was:

        4

        #$ _NVVM_BRANCH_=nvvm

        #$ _SPACE_=

        #$ _CUDART_=cudart

        #$ _HERE_=/opt/cuda/bin

        #$ _THERE_=/opt/cuda/bin

        #$ _TARGET_SIZE_=

        #$ _TARGET_DIR_=

        #$ _TARGET_DIR_=targets/x86_64-linux

        #$ TOP=/opt/cuda/bin/..

        #$ NVVMIR_LIBRARY_DIR=/opt/cuda/bin/../nvvm/libdevice

        #$ LD_LIBRARY_PATH=/opt/cuda/bin/../lib:

        #$
        PATH=/opt/cuda/bin/../nvvm/bin:/opt/cuda/bin:/tmp/pip-build-env-92yt10qq/overlay/bin:/tmp/pip-build-env-92yt10qq/normal/bin:/home/flaport/.anaconda/bin:/home/flaport/.anaconda/condabin:/home/flaport/.local/bin:/home/flaport/.cargo/bin:/home/flaport/.local/bin:/home/flaport/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/home/flaport/.dotnet/tools:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl

        #$ INCLUDES="-I/opt/cuda/bin/../targets/x86_64-linux/include"

        #$ LIBRARIES= "-L/opt/cuda/bin/../targets/x86_64-linux/lib/stubs"
        "-L/opt/cuda/bin/../targets/x86_64-linux/lib"

        #$ CUDAFE_FLAGS=

        #$ PTXAS_FLAGS=

        #$ rm tmp/a_dlink.reg.c

        #$ gcc -D__CUDA_ARCH_LIST__=520 -E -x c++ -D__CUDACC__ -D__NVCC__
        "-I/opt/cuda/bin/../targets/x86_64-linux/include" -D__CUDACC_VER_MAJOR__=12
        -D__CUDACC_VER_MINOR__=1 -D__CUDACC_VER_BUILD__=105
        -D__CUDA_API_VER_MAJOR__=12 -D__CUDA_API_VER_MINOR__=1
        -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64
        "CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.cpp4.ii"

        #$ cudafe++ --c++17 --gnu_version=130101 --display_error_number
        --orig_src_file_name "CMakeCUDACompilerId.cu" --orig_src_path_name
        "/home/flaport/Programming/fdtdz/CMakeFiles/3.26.3/CompilerIdCUDA/CMakeCUDACompilerId.cu"
        --allow_managed --m64 --parse_templates --gen_c_file_name
        "tmp/CMakeCUDACompilerId.cudafe1.cpp" --stub_file_name
        "CMakeCUDACompilerId.cudafe1.stub.c" --gen_module_id_file
        --module_id_file_name "tmp/CMakeCUDACompilerId.module_id"
        "tmp/CMakeCUDACompilerId.cpp4.ii"

        /usr/include/stdlib.h(141): error: identifier "_Float32" is undefined

          extern _Float32 strtof32 (const char *__restrict __nptr,
                 ^

        /usr/include/stdlib.h(147): error: identifier "_Float64" is undefined

          extern _Float64 strtof64 (const char *__restrict __nptr,
                 ^

        /usr/include/stdlib.h(153): error: identifier "_Float128" is undefined

          extern _Float128 strtof128 (const char *__restrict __nptr,
                 ^

        /usr/include/stdlib.h(159): error: identifier "_Float32x" is undefined

          extern _Float32x strtof32x (const char *__restrict __nptr,
                 ^

        /usr/include/stdlib.h(165): error: identifier "_Float64x" is undefined

          extern _Float64x strtof64x (const char *__restrict __nptr,
                 ^

        /usr/include/stdlib.h(234): error: identifier "_Float32" is undefined

                   _Float32 __f)
                   ^

      [BASICALLY THE SAME ERROR OVER AND OVER...]

      Call Stack (most recent call first):
        /tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
        /tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
        /tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCUDACompiler.cmake:307 (CMAKE_DETERMINE_COMPILER_ID)
        CMakeLists.txt:4 (project)

      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/home/flaport/.anaconda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/flaport/.anaconda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/flaport/.anaconda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 416, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 401, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 93, in <module>
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 343, in run
          self.run_command("build")
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 64, in build_extensions
        File "/home/flaport/.anaconda/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/home/flaport/Programming/fdtdz', '-DCMAKE_INSTALL_PREFIX=/home/flaport/Programming/fdtdz/build/lib.linux-x86_64-cpython-310', '-DPython_EXECUTABLE=/home/flaport/.anaconda/bin/python', '-DPython_LIBRARIES=/home/flaport/.anaconda/lib/libpython3.10.a', '-DPython_INCLUDE_DIRS=/home/flaport/.anaconda/include/python3.10', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=/tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/pybind11/share/cmake/pybind11']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for fdtdz
  Building wheel for jax (pyproject.toml) ... done
  Created wheel for jax: filename=jax-0.4.8-py3-none-any.whl size=1439678 sha256=07d95387bb962e45d4aa1c2b2da204875256b5fe959141449db72fcbc5e25475
  Stored in directory: /tmp/pip-ephem-wheel-cache-j9rsmqb3/wheels/09/6f/35/a8fac8b61de8e0d9eb07988481528898561923e260b1fa7d2f
Successfully built jax
Failed to build fdtdz
ERROR: Could not build wheels for fdtdz, which is required to install pyproject.toml-based projects

~/Programming/fdtdz$  nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0

~/Programming/fdtdz$  cmake --version
cmake version 3.26.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Any idea on how to solve this? Thanks

jlu-spins commented 1 year ago

The line 

   CMake Error at /tmp/pip-build-env-92yt10qq/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCompilerId.cmake:751 (message):
        Compiling the CUDA compiler identification source file
        "CMakeCUDACompilerId.cu" failed.

makes me think this is CUDA related.

I have actually been using CUDA 11.8 to build things, would you be able to try that? Please let me know whether it works or not.

I use the following to install it (on Ubuntu 22.04):

wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.run
flaport commented 1 year ago

Unfortunately that did not seem to solve it. I'll try a few more things and see if I can make it work. I'll report my findings afterwards

jlu-spins commented 1 year ago

Thanks. Any additional information on your system, OS, etc... may be helpful (and of course the demo works in the colab notebook)

On Fri, May 5, 2023 at 3:53 PM Floris Laporte @.***> wrote:

Unfortunately that did not seem to solve it. I'll try a few more things and see if I can make it work. I'll report my findings afterwards

— Reply to this email directly, view it on GitHub https://github.com/spinsphotonics/fdtdz/issues/9#issuecomment-1536868894, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3VGPQMUYJ3SS6PWTV4QQ7TXEWAFTANCNFSM6AAAAAAXWUJDJE . You are receiving this because you commented.Message ID: @.***>

danielpeace commented 1 year ago

Not sure if its the same error, but in my case (on Ubuntu 20.04) I had to manually install JAX with CUDA first.

From: https://github.com/google/jax#installation

pip install --upgrade pip

# CUDA 12 installation
# Note: wheels only available on linux.
pip install --upgrade "jax[cuda12_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

# CUDA 11 installation
# Note: wheels only available on linux.
pip install --upgrade "jax[cuda11_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
jlu-spins commented 1 year ago

@danielpeace Thanks for mentioning this. Yes, I think this could be the issue as well. In my experience with installing JAX, the sticking point is often correctly installing the CUDNN package. Since fdtd-z is explicitly designed to run on the GPU, this is a must.

@flaport Please let me know if installing JAX with CUDNN solved this for you!

flaport commented 1 year ago

Installing jax with cuda support indeed solved the issue! Thanks @danielpeace :)