ccober6
commented
1 year ago @trilinos/framework
Open USAMAH1989 opened 1 year ago
ccober6
commented
1 year ago @trilinos/framework
USAMAH1989
commented
1 year ago What does @trilinos/framework https://github.com/orgs/trilinos/teams/framework mean?
On Mon, May 1, 2023 at 7:41 PM Curtis Ober @.***> wrote:
@trilinos/framework https://github.com/orgs/trilinos/teams/framework
— Reply to this email directly, view it on GitHub https://github.com/trilinos/Trilinos/issues/11834#issuecomment-1529932197, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMQ7KA3VGNKZZR5DRIDOIOLXD7RRZANCNFSM6AAAAAAXREEO2E . You are receiving this because you authored the thread.Message ID: @.***>
-- Best Regards, Osama SH. ALAtawneh
cgcgcg
commented
1 year ago @USAMAH1989 Please post your CMake invocation and the corresponding terminal output.
ccober6
commented
1 year ago What does @trilinos/framework https://github.com/orgs/trilinos/teams/framework mean? … On Mon, May 1, 2023 at 7:41 PM Curtis Ober @.> wrote: @trilinos/framework https://github.com/orgs/trilinos/teams/framework — Reply to this email directly, view it on GitHub <#11834 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMQ7KA3VGNKZZR5DRIDOIOLXD7RRZANCNFSM6AAAAAAXREEO2E . You are receiving this because you authored the thread.Message ID: @.> -- Best Regards, Osama SH. ALAtawneh
I was "mentioning" the Trilinos Framework team to look at this. As @cgcgcg mentioned, please post your cmake command and output to this issue to help diagnose your issue and possibly reproduce it.
USAMAH1989
commented
1 year ago Dear sir: Please kindly find the attached files that represent CMAKE invocation and corresponding terminal output for both Trilinos tarball release and Trilinos installed using spack.
Warm Regards, Osama ALAtawneh 2nd May 2023
On Tue, May 2, 2023 at 10:08 AM Curtis Ober @.***> wrote:
What does @trilinos/framework https://github.com/orgs/trilinos/teams/framework https://github.com/orgs/trilinos/teams/framework mean? … <#m-7772687157186961962> On Mon, May 1, 2023 at 7:41 PM Curtis Ober @.> wrote: @trilinos/framework https://github.com/orgs/trilinos/teams/framework https://github.com/orgs/trilinos/teams/framework https://github.com/orgs/trilinos/teams/framework — Reply to this email directly, view it on GitHub <#11834 (comment) https://github.com/trilinos/Trilinos/issues/11834#issuecomment-1529932197>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMQ7KA3VGNKZZR5DRIDOIOLXD7RRZANCNFSM6AAAAAAXREEO2E https://github.com/notifications/unsubscribe-auth/AMQ7KA3VGNKZZR5DRIDOIOLXD7RRZANCNFSM6AAAAAAXREEO2E . You are receiving this because you authored the thread.Message ID: @.> -- Best Regards, Osama SH. ALAtawneh
I was "mentioning" the Trilinos Framework team to look at this. As @cgcgcg https://github.com/cgcgcg mentioned, please post your cmake command and output to this issue to help diagnose your issue and possibly reproduce it.
— Reply to this email directly, view it on GitHub https://github.com/trilinos/Trilinos/issues/11834#issuecomment-1531548920, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMQ7KAZWBPDQ77LJL4VMZU3XEEINBANCNFSM6AAAAAAXREEO2E . You are receiving this because you were mentioned.Message ID: @.***>
-- Best Regards, Osama SH. ALAtawneh
ccober6
commented
1 year ago @USAMAH1989 your files did not get posted to the issue through your email. Can you directly post them?
USAMAH1989
commented
1 year ago Sorry for that. The files are below: Trilinos Compilation Process(Spack)-Responce.TXT Trilinos Compilation Process(Tarball)-Responce.TXT
cgcgcg
commented
1 year ago Please also post the terminal output of the CMake invocation. Also note that your CMake line has errors in it, so you should be seeing a warning about unused variables.
kliegeois
commented
1 year ago The PyTrilinos issue seems to be the same as https://github.com/trilinos/Trilinos/issues/11073, https://github.com/trilinos/Trilinos/issues/11596 and https://github.com/trilinos/Trilinos/issues/11691 ( I am facing the same issue right now too ).
USAMAH1989
commented
1 year ago Dear Sir: Sorry for not posting what you exactly wanted. Here, I am attaching the CMAKE terminals output that were appeared on my screen in addition to the CMakeOutput.log for both Trilinos 13 and Trilinos 14 tarball releases. Another important thing I would also like to add here is that the output terminal for Trilinos 14 stops at a given location not as the normal terminal output for Trilinos13 as shown in the files attached below.
Warm Regards, Osama ALAtawneh Trilinos13-CMakeOutput.log Trilinos14-CMakeOutput.log Trilinos(13.4.1)-CMAKE output terminal.TXT Trilinos(14.0)-CMAKE output terminal.TXT
cgcgcg
commented
1 year ago It looks like standard GNU compilers are used and you get an error because they do not know about Cuda. Could you try to use
CMAKE_CXX_COMPILER=~/Downloads/TrilinosSource14/Trilinos-trilinos-release-14-0-0/packages/kokkos/bin/nvcc_wrapperinstead of
KOKKOS_COMPILE_LAUNCHER
KOKKOS_NVCC_COMPILERAlternatively, you might have to specify the MPI compiler wrappers and then point them to nvcc_wrapper using OMPI_CXX or MPICH_CXX.
USAMAH1989
commented
1 year ago I am going to try that at evening.
From: Christian Glusa @.> Sent: Thursday, May 4, 2023 8:34:29 AM To: trilinos/Trilinos @.> Cc: USAMAH1989 @.>; Mention @.> Subject: Re: [trilinos/Trilinos] Trilinos Source Compilation-Issue (Issue #11834)
It looks like standard GNU compilers are used and you get an error because they do not know about Cuda. Could you try to use
CMAKE_CXX_COMPILER=~/Downloads/TrilinosSource14/Trilinos-trilinos-release-14-0-0/packages/kokkos/bin/nvcc_wrapper
instead of
KOKKOS_COMPILE_LAUNCHER KOKKOS_NVCC_COMPILER
Alternatively, you might have to specify the MPI compiler wrappers and then point them to nvcc_wrapper using OMPI_CXX or MPICH_CXX.
— Reply to this email directly, view it on GitHubhttps://github.com/trilinos/Trilinos/issues/11834#issuecomment-1534117342, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMQ7KA4TW2TCKUDZGYSWCY3XEM5WLANCNFSM6AAAAAAXREEO2E. You are receiving this because you were mentioned.Message ID: @.***>
USAMAH1989
commented
1 year ago Dear Sir: Thank you very much for your guidance. Now the problem was recognized. The issue was due to the disability of the gcc compiler that I used at first to deal with nvcc arguments, actually I did what @cgcgcg recommended to do and that solved the "unrecognized command line option" issue mentioned above. I also used mpi from openmpi as also stated above. At first, I found another issue at the end of building saying that the include/mpi.h was not found, where this solved by setting MPI_BASE_DIR=/.../openmpi/include. Also, I faced the "Argument list too long" issue represented by error 127. and I tried to solve it using -DCMAKE_CXX_USE_RESPONSE_FILES_FOR_OBJECTS=ON, but nvcc produced error : nvcc fatal: no input files specified; use option --help for more information. To overcome the last issue I used ninja build tool within cmake generator using -G Ninja in configuration command line. The error 127 was not observed and I compiled Trilinos 13.4.1 with PyTrilinos and installed successfully with pamgen,seacas,shards,kokkos,kokkos-kernels, all other kokkos packages in trilinos, Teuchos, Amesos, Amessos2. Ifpack, Ifpack2, Zoltan, and Zoltan2 without any issue or observations. Trilinos 14.1.0 and Trilinos 14.0.0 is still suffering from the PyTrilinos issue cleared above in the issue description "unable to find issue" andalso SUperLU version 5, version 4.3 of SuperLU is deprecated I tried to install it, but that was not sucessfull.
Warm Regards, Osama ALAtawneh 8th May 2023
cgcgcg
commented
1 year ago Sounds like we resolved some of the issue. As @kliegeois mentioned, the PyTrilinos problems are already tracked in different issues. What's the problem with SuperLU? Can you post the error message?
EDIT: Is the issue that the Trilinos Spack package wants SuperLU 4.3? I believe Trilinos supports up to 5.2.1
USAMAH1989
commented
1 year ago The error below appeared when I used SuperLU 5.3 and SuperLU 5.2.1:
packages/amesos/src/Amesos_Superlu.cpp(490): error: argument of type "SLU::mem_usage_t " is incompatible with parameter of type "SLU::GlobalLU_t "
cgcgcg
commented
1 year ago Something doesn't work correctly then in the Trilinos CMake. We have a check for a newer SuperLU interface here:
https://github.com/trilinos/Trilinos/blob/master/cmake/TPLs/FindTPLSuperLU.cmake
This should be setting Trilinos_ENABLE_SuperLU5_API. Amesos uses that variable here:
https://github.com/trilinos/Trilinos/blob/b95d382c8faab6dca66e70d2098181f3846723d8/packages/amesos/CMakeLists.txt#L46-L57
and then here
https://github.com/trilinos/Trilinos/blob/b95d382c8faab6dca66e70d2098181f3846723d8/packages/amesos/src/Amesos_Superlu.cpp#L482-L490
Could you grep through CMakeCache.txt in your build directory to see what Trilinos_ENABLE_SuperLU5_API and HAVE_AMESOS_SUPERLU5_API are set to?
USAMAH1989
commented
1 year ago Regarding "Trilinos_ENABLE_SuperLU5_API" this what I exactly found in CMakeCache.txt in the build directory:
//Test Trilinos_ENABLE_SuperLU5_API Trilinos_ENABLE_SuperLU5_API:INTERNAL= Where, nothing after equal sign.
While, I did not find "HAVE_AMESOS_SUPERLU5_API" in the CMakeCache.txt file.
cgcgcg
commented
1 year ago Ok. Can you set Trilinos_ENABLE_SuperLU5_API=ON and see if that fixes the build?
sebrowne
commented
1 year ago If you want to debug why the API test failed, I believe the results of that configure test are in ${YOUR_BUILD_DIR}/CMakeFiles/CMakeError.txt. When I see that test fail (when it shouldn't), it's usually due to something linker-related like improper MKL library specification (which doesn't apply here), or something similar.
USAMAH1989
commented
1 year ago I applied "Trilinos_ENABLE_SuperLU5_API=ON" and "HAVE_AMESOS_SUPERLU5_API=ON" and the following appeared at the end of configuration:
CMake Error at packages/ml/CMakeLists.txt:22 (MESSAGE):
ML CONFIGURATION ERROR: SuperLU_5.0 detected -
only SuperLU version < 5.0 currently supported for this package.
Please either use a supported version of SuperLU or disable SuperLU for
this package with the following configuration option:
ML_ENABLE_SuperLU:BOOL=OFFOk. I guess we should start discussing then if you really want to build that combination of packages. You want a Cuda build, right? Neither Amesos nor ML have any support for Cuda.
USAMAH1989
commented
1 year ago Yes that is right, but in the above configuration I only activated Amessos, Amesos2, SuperLU, and SuperLUDist in addition to ParMETIS as it is required for SuperLU. I did that just to save time and obtain the results quickly, specially when I catched that the issue of SuperLU is mainly related to Amessos.
USAMAH1989
commented
1 year ago However, now I am applying the above, but this time with "-D ML_ENABLE_SuperLU:BOOL=OFF" in configuration line. configuration done successfully as generation. building process has just started.
cgcgcg
commented
1 year ago @USAMAH1989 Could you also try @sebrowne's suggestion from above and determine why the check for SuperLU5 failed? This would be helpful for us.
USAMAH1989
commented
1 year ago building results in : packages/amesos/src/Amesos_Superludist.cpp(514): error: identifier "DOUBLE" is undefined DOUBLE packages/amesos/src/Amesos_Superludist.cpp(523): error: identifier "EXTRA" is undefined EXTRA Ok, I will
USAMAH1989
commented
1 year ago I reviewed that file,CMakeError.log", and I found that some include files were not being found like: superlu_ddefs.h and superlu_enum_consts.h and these were in the include dir of superlu_dist. The version of the superlu_dist that I used was 8.1.2 and the superlu_dist version defined in trilinos is up to 4. When I disabled super_dist to check if that was right or not I disabled it and reconfigure Trilinos again with SuperLU5, Amesos, Amesos2, MPI, and ParaMETIS. And configuration done successfully as the configuration and generation. So, SuperLU_dist version was the cause behind that, where Trilinos was looking for superlu_dist version up to 4 and the the one that I used was 8.1.2
USAMAH1989
commented
1 year ago There is a problem with using superlu, superlu-dist, and Amesos2 through the compilation. when I used superlu-dist 2.5 and superlu 3.0 I observed the error:
Where the issue above observed with other versions like: Superlu5.3 with superlu-dist 6.2 and 6.3 Superlu 5.2.1 with superlu-dist 6.2 and 6.3 Superlu 4.3 with superlu-dist 4.0
USAMAH1989
commented
1 year ago I also tried to use recent versions of superlu and superlu-dist (superlu5.3 and superlu-dist8.1.2) and the following observed:
USAMAH1989
commented
1 year ago The above issues were not observed with Amesos , but only with Amesos2.
jhux2
commented
1 year ago @trilinos/amesos2 can you advise which SuperLU and SuperLU-dist versions are supported?
waveman68
commented
6 months ago I have the same issue with superlu_dist and amesos (not v2). Setting -D Trilinos_ENABLE_SuperLU5_API=ON did not help. I am using superlu_dist 8.2.1.
The cmake command runs through but make build stops at exactly the same spot. For completeness here is my cmake configuration passed in via the command line (I have tried a number of flag combinations with no success).
cmake \
-DTPL_ENABLE_MPI=ON \
-DCMAKE_C_COMPILER=/opt/extlib/openmpi/5.0.2/gcc/13.2.0/bin/mpicc \
-DCMAKE_CXX_COMPILER=/opt/extlib/openmpi/5.0.2/gcc/13.2.0/bin/mpicxx \
-DCMAKE_Fortran_COMPILER=/opt/extlib/openmpi/5.0.2/gcc/13.2.0/bin/mpifort \
\
-D Trilinos_ENABLE_SuperLU5_API=ON \
-D Trilinos_ENABLE_COMPLEX=ON \
-D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \
-D Trilinos_ENABLE_COMPLEX_FLOAT=ON \
-D Trilinos_ENABLE_FLOAT=ON \
-D Trilinos_ENABLE_FLOAT=ON \
-D TPL_BLAS_LIBRARIES="-framework Accelerate" \
-D TPL_LAPACK_LIBRARIES="-framework Accelerate" \
\
-D Trilinos_ENABLE_Amesos2=OFF \
-D Trilinos_ENABLE_Anasazi=ON \
-D Trilinos_ENABLE_AztecOO=ON \
-D Trilinos_ENABLE_Belos=OFF \
-D Trilinos_ENABLE_Epetra=ON \
-D Trilinos_ENABLE_EpetraExt=ON \
-D Trilinos_ENABLE_AztecOO=ON \
-D Trilinos_ENABLE_Ifpack=ON \
-D Trilinos_ENABLE_Ifpack2=OFF \
-D Trilinos_ENABLE_ML=ON \
-D Trilinos_ENABLE_Tpetra=OFF \
-D Tpetra_INST_INT_INT=ON \
-D Tpetra_INST_SYCL=OFF \
-D Trilinos_ENABLE_Xpetra=ON \
-D Xpetra_ENABLE_Epetra=ON \
\
-DTrilinos_ENABLE_BoostLib=ON \
-DTrilinos_ENABLE_HDF5=ON \
-DTPL_ENABLE_ParMETIS=ON \
-DParMETIS_INCLUDE_DIRS="/opt/extlib/parmetis/4.0.3/openmpi/4.1.4/include;/opt/extlib/metis/5.1.0/gcc/13.2.0/include" \
-DParMETIS_LIBRARY_DIRS="/opt/extlib/parmetis/4.0.3/openmpi/4.1.4/lib;/opt/extlib/metis/5.1.0/gcc/13.2.0/lib" \
-DTrilinos_ENABLE_SuperLUDist=ON \
-DTPL_SuperLUDist_LIBRARIES="/opt/extlib/superludist/git/openmpi/5.0.2/gcc/13.2.0/lib/libsuperlu_dist.a;/opt/extlib/superludist/git/openmpi/5.0.2/gcc/13.2.0/lib/libsuperlu_dist_fortran.a" \
-DTPL_SuperLUDist_INCLUDE_DIRS=/opt/extlib/superludist/git/openmpi/5.0.2/gcc/13.2.0/include \
-DBoostLib_LIBRARY_DIRS=/opt/extlib/boost/1.85.0/openmpi/5.0.2/gcc/13.2.0/lib \
-DBoostLib_INCLUDE_DIRS=/opt/extlib/boost/1.85.0/openmpi/5.0.2/gcc/13.2.0/include \
-DCMAKE_INSTALL_PREFIX=/opt/extlib/trilinos/15.1.1/openmpi/5.0.2/gcc/13.2.0 \
../
Dear Developers:
I tried to compile Trilinos 14 from source obtained from formal Trilinos github, when I enabled CUDA and kokkos I was always facing issues related to nvcc_wrapper as shown below. What made me to be so excited is that when I used git clone to download the version 14 source I did not face any issue. But when I used the tarball release of the same version, I face the issues below:
c++: error: unrecognized command-line option ‘-Wext-lambda-captures-this’ c++: error: unrecognized command-line option ‘-arch=sm_61 make[2]: *** [packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/build.make:76: packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o] Error 1
So, could you let me know how could I solve the issue. I am so curios about it. Where, the above issues were also observed with pamgen, seacas. and shards packages. And disabling them did not reveal the issues through compilation.
Also, PyTrilinos was not compiled due to the disability to find some headers in Trilinos 14.0, like : BUILD14/packages/PyTrilinos/src/Teuchos.RCP.i:82: Error: Unable to find 'Teuchos_DLLExportMacro.h' and all of headers mentioned in Teuchos.RCP.i file. So, what is the possible solution for that?
Actually, that made me to think that Trilinos 14 dose not support pyTrilinos. So, I tried Trilinos 13.4.1 and I found the following: 1.) PyTrilinos was compiled successfully with this version. 2.) Unfortunately, The above issues were also observed when I used the git cloned version and .tar.gz version : c++: error: unrecognized command-line option ‘-Wext-lambda-captures-this’ c++: error: unrecognized command-line option ‘-arch=sm_61 make[2]: *** [packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/build.make:76: packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o] Error 1
So, could you help me to recognize the cause of those issues? Another thing I like to add here is that I also utilized spack to install Trilinos 13.4.1. Every thing procedded well, but superlu 5 and 6 is incompatible with Amesos. Trilinos 14 also suffers from the same thing and pyTrilinos produced the issue of "BUILD14/packages/PyTrilinos/src/Teuchos.RCP.i:82: Error: Unable to find 'Teuchos_DLLExportMacro.h' and all of headers mentioned in Teuchos.RCP.i file"
So I am looking to hear from you soon to solve the issues above.
Warm Regards, Osama ALAtawneh