search

ECP-WarpX / WarpX

WarpX is an advanced electromagnetic & electrostatic Particle-In-Cell code.
https://ecp-warpx.github.io
Other
291 stars 185 forks source link

Compilation failure in QED with AMReX_PARTICLES_PRECISION=SINGLE #2553

Open PhilMiller opened 2 years ago

PhilMiller commented 2 years ago

When I configure with -DWarpX_QED=ON -DAMReX_PARTICLES_PRECISION=SINGLE, I encounter a compilation error as follows:

[3/318] Building CUDA object CMakeFiles/WarpX.dir/Source/Diagnostics/Diagnostics.cpp.o
FAILED: CMakeFiles/WarpX.dir/Source/Diagnostics/Diagnostics.cpp.o 
/usr/local/bin/ccache /usr/local/cuda-11.3/bin/nvcc  -DPICSAR_GIT_VERSION=\"a78be127f66a\" -DWARPX_DIM_XZ -DWARPX_GIT_VERSION=\"21.11-33-g8f0ccdbc8336-dirty\" -DWARPX_QED -DWARPX_USE_GPUCLOCK -DWARPX_USE_OPENPMD -I/home/ubuntu/repos/WarpX/Source -I_deps/fetchedamrex-src/Tools/C_scripts -I_deps/fetchedopenpmd-src/include -I_deps/fetchedopenpmd-build/include -I_deps/fetchedpicsar-src/multi_physics/QED/include -isystem=_deps/fetchedamrex-src/Src/Base -isystem=_deps/fetchedamrex-src/Src/Base/Parser -isystem=_deps/fetchedamrex-src/Src/Boundary -isystem=_deps/fetchedamrex-src/Src/AmrCore -isystem=_deps/fetchedamrex-src/Src/EB -isystem=_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -isystem=_deps/fetchedamrex-src/Src/LinearSolvers/Projections -isystem=_deps/fetchedamrex-src/Src/Particle -isystem=_deps/fetchedamrex-build -isystem=/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -isystem=/usr/lib/x86_64-linux-gnu/openmpi/include -isystem=_deps/fetchedopenpmd-src/share/openPMD/thirdParty/variant/include -m64 --expt-relaxed-constexpr --expt-extended-lambda -Wno-deprecated-gpu-targets -gencode=arch=compute_70,code=sm_70 -maxrregcount=255 -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored --use_fast_math -Xcudafe --display_error_number --Wext-lambda-captures-this --forward-unknown-to-host-compiler --generate-line-info --source-in-ptx --display-error-number --diag-error 20092 -O3 -DNDEBUG -Xcompiler=-fPIC   -Xcompiler -pthread -std=c++14 -x cu -c /home/ubuntu/repos/WarpX/Source/Diagnostics/Diagnostics.cpp -o CMakeFiles/WarpX.dir/Source/Diagnostics/Diagnostics.cpp.o && /usr/local/cuda-11.3/bin/nvcc  -DPICSAR_GIT_VERSION=\"a78be127f66a\" -DWARPX_DIM_XZ -DWARPX_GIT_VERSION=\"21.11-33-g8f0ccdbc8336-dirty\" -DWARPX_QED -DWARPX_USE_GPUCLOCK -DWARPX_USE_OPENPMD -I/home/ubuntu/repos/WarpX/Source -I_deps/fetchedamrex-src/Tools/C_scripts -I_deps/fetchedopenpmd-src/include -I_deps/fetchedopenpmd-build/include -I_deps/fetchedpicsar-src/multi_physics/QED/include -isystem=_deps/fetchedamrex-src/Src/Base -isystem=_deps/fetchedamrex-src/Src/Base/Parser -isystem=_deps/fetchedamrex-src/Src/Boundary -isystem=_deps/fetchedamrex-src/Src/AmrCore -isystem=_deps/fetchedamrex-src/Src/EB -isystem=_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -isystem=_deps/fetchedamrex-src/Src/LinearSolvers/Projections -isystem=_deps/fetchedamrex-src/Src/Particle -isystem=_deps/fetchedamrex-build -isystem=/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -isystem=/usr/lib/x86_64-linux-gnu/openmpi/include -isystem=_deps/fetchedopenpmd-src/share/openPMD/thirdParty/variant/include -m64 --expt-relaxed-constexpr --expt-extended-lambda -Wno-deprecated-gpu-targets -gencode=arch=compute_70,code=sm_70 -maxrregcount=255 -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored --use_fast_math -Xcudafe --display_error_number --Wext-lambda-captures-this --forward-unknown-to-host-compiler --generate-line-info --source-in-ptx --display-error-number --diag-error 20092 -O3 -DNDEBUG -Xcompiler=-fPIC   -Xcompiler -pthread -std=c++14 -x cu -M /home/ubuntu/repos/WarpX/Source/Diagnostics/Diagnostics.cpp -MT CMakeFiles/WarpX.dir/Source/Diagnostics/Diagnostics.cpp.o -o CMakeFiles/WarpX.dir/Source/Diagnostics/Diagnostics.cpp.o.d
/home/ubuntu/repos/WarpX/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.H(178): error: no instance of function template "picsar::multi_physics::phys::breit_wheeler::evolve_optical_depth" matches the argument list
            argument types are: (const double, const amrex::Real, const amrex::Real, amrex::ParticleReal, const BW_dndt_table_view)

_deps/fetchedpicsar-src/multi_physics/QED/include/picsar_qed/physics/breit_wheeler/breit_wheeler_engine_core.hpp(224): error: no operator "*" matches these operands
            operand types are: picsar::multi_physics::math::vec3<amrex::ParticleReal> * double
          detected during instantiation of "__nv_bool picsar::multi_physics::phys::breit_wheeler::generate_breit_wheeler_pairs(RealType, const picsar::multi_physics::math::vec3<RealType> &, RealType, const TableType &, picsar::multi_physics::math::vec3<RealType> &, picsar::multi_physics::math::vec3<RealType> &, RealType) [with RealType=amrex::ParticleReal, TableType=BW_pair_prod_table_view, UnitSystem=picsar::multi_physics::phys::unit_system::SI]" 
/home/ubuntu/repos/WarpX/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.H(266): here

_deps/fetchedpicsar-src/multi_physics/QED/include/picsar_qed/physics/breit_wheeler/breit_wheeler_engine_core.hpp(226): error: no operator "*" matches these operands
            operand types are: picsar::multi_physics::math::vec3<amrex::ParticleReal> * double
          detected during instantiation of "__nv_bool picsar::multi_physics::phys::breit_wheeler::generate_breit_wheeler_pairs(RealType, const picsar::multi_physics::math::vec3<RealType> &, RealType, const TableType &, picsar::multi_physics::math::vec3<RealType> &, picsar::multi_physics::math::vec3<RealType> &, RealType) [with RealType=amrex::ParticleReal, TableType=BW_pair_prod_table_view, UnitSystem=picsar::multi_physics::phys::unit_system::SI]" 
/home/ubuntu/repos/WarpX/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.H(266): here

_deps/fetchedpicsar-src/multi_physics/QED/include/picsar_qed/physics/quantum_sync/quantum_sync_engine_core.hpp(229): error: no operator "*" matches these operands
            operand types are: picsar::multi_physics::math::vec3<amrex::ParticleReal> * const double
          detected during instantiation of "__nv_bool picsar::multi_physics::phys::quantum_sync::generate_photon_update_momentum(RealType, picsar::multi_physics::math::vec3<RealType> &, RealType, const TableType &, picsar::multi_physics::math::vec3<RealType> &, RealType) [with RealType=amrex::ParticleReal, TableType=QS_phot_em_table_view, UnitSystem=picsar::multi_physics::phys::unit_system::SI]" 
/home/ubuntu/repos/WarpX/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.H(245): here

4 errors detected in the compilation of "/home/ubuntu/repos/WarpX/Source/Diagnostics/Diagnostics.cpp".
PhilMiller commented 2 years ago

Easily worked around for us, since QED=ON is the default, but we don't need it, so we can just turn it off.

ax3l commented 2 years ago

Ah, mixing precision for fields and particles is not yet supported/tested, but we are working on it: (here: DP particles, SP fields #2017).

This definitely needs a few interface updates and/or extra casts. cc-ing @lucafedeli88 and @NeilZaim on it, I think that's likely a quick fix :)

The general "all SP" mode (-DWarpX_PRECISION=SINGLE) should work, though. Are you aware of that option? It wraps both the AMReX particle and general precision switches into one. https://warpx.readthedocs.io/en/latest/install/cmake.html#build-options

NeilZaim commented 2 years ago

Thanks for finding this issue. :+1:

Yes, picsar_qed does not currently support mixed precision, we only have a single Real type for both fields and particle quantities so I'm not surprised that we have compilation issues if we try to use float for particle quantities and double for fields.

I think supporting this would require a bit of work (either adding a second Real type in picsar_qed and modifying the types in the routines where needed, or adding a bunch of casts where needed in WarpX), so in the meantime if you want mixed precision, temporarily disabling QED is indeed the way to go (or if you just want single precision for both fields and particles, doing as @ax3l suggested will work without having to disable QED).