Closed fryeguy52 closed 5 years ago
@bartlettroscoe I synced my branch with upstream and it has a right enabling flag for rdc.
You should be able to do this build on any RHEL7 machine with a GPU that has the SEMS env loaded. (The only machine like that that I can get to in 'ascicgpu15'.)
Thanks @bartlettroscoe. I already tried a build on an x86 machine and get the same linking errors.
@ndellingwood When I am debugging intrepid2, I see the following error:
error: The closure type for a lambda ("lambda [](int)->void"
The lambda somehow misses __device__
. Any idea?
@ndellingwood my bad.... there is typo in my configuration script.
I think I have just determined that NVCC doesn't support shared+RDC. From the NVCC documentation:
6.3. Libraries The device linker has the ability to read the static host library formats (.a on Linux and Mac OS X, .lib on Windows). It ignores any dynamic (.so or .dll) libraries.
Thanks very very very much @etphipp . If we exclude errors from shared libraries, intrepid2, kokkoskernels, kokkos and tacho do not have build errors. @mperego @srajama1 @ndellingwood .
@bartlettroscoe Maybe we can remove the labels of packages ?
@etphipp @kyungjoo-kim : Thank you both for looking at this right away ..
@etphipp said:
I think I have just determined that NVCC doesn't support shared+RDC
Yea, that seems to be the case:
But is that universally true? As shown here, how is it that some Kokkos test executables build and run such fine with shared libs and only 3 executables fail to link as shown here?
If CUDA does not support shared+RDC, what does this mean for SPARC that has to build with shared libs with CUDA on some systems? Can SPARC just never use RDC?
Could the RDC-related object code be split off and put into a static lib and have the final executable link that in as well as the shared libs? Is that possible? Otherwise, we can mixed shared and static libs if we are careful.
But is that universally true? As shown here, how is it that some Kokkos test executables build and run such fine with shared libs and only 3 executables fail to link as shown here?
luck
If CUDA does not support shared+RDC, what does this mean for SPARC that has to build with shared libs with CUDA on some systems? Can SPARC just never use RDC?
If shared libraries are a requirement, RDC is out.
Could the RDC-related object code be split off and put into a static lib and have the final executable link that in as well as the shared libs? Is that possible? Otherwise, we can mixed shared and static libs if we are careful.
I think the possibility of cheating our way out of this is low, but anyone is welcome to try. If RDC is enabled, any code that does anything remotely Kokkos-related needs to be compiled with RDC.
@ibaned said:
If shared libraries are a requirement, RDC is out.
So SPARC is out of luck on using RDC because it requires shared libs when CUDA is enabled? So we just tell SPARC developers "sorry, no RDC for you"?
Before worrying about potential work-arounds, I think we need to better understand the requirements for shared libraries and RDC, i.e., is this a real requirement, or is it just something people would like to have. If it is in fact a real requirement, the best path forward is to get Nvidia to support it. Trying to cobble together shared and static libraries is a recipe for disaster. Furthermore, we likely only see a few linking errors because currently very little use of CUDA in Trilinos (whether through Kokkos or something else), actually uses RDC. If it was used pervasively, I seriously doubt working around the issue would even be possible.
@etphipp said:
Before worrying about potential work-arounds, I think we need to better understand the requirements for shared libraries and RDC, i.e., is this a real requirement, or is it just something people would like to have.
We should talk about this with Micah at the next ATDM PI meeting.
Seems lame to have a system build tool that does not support shared libraries in 2019 (just another example of how bad this is). Perhaps native CUDA/GPU support in Clang will address this if NVIDIA can't?
@bartlettroscoe - FYI - I reached out to NVIDIA this morning to get some information/guidance on this particular issue. Will report back to you all shortly.
@bartlettroscoe said:
Perhaps native CUDA/GPU support in Clang will address this if NVIDIA can't?
My impression is Clang uses parts of the Nvidia toolchain behind the scenes, so I suspect not, but perhaps @crtrott or @nmhamster know.
Replacing our CUDA builds with Clang is something we should pursue though. The first step is to get SEMS to add Clang 7.0 to the SEMS environments.
Seems lame to have a system build tool that does not support shared libraries in 2019
I think it is RDC that is underdeveloped, shared libraries work fine without RDC. I would recommend applications take some time to reflect on whether relying on RDC as a requirement is such a great idea.
My impression is Clang uses parts of the Nvidia toolchain behind the scenes, so I suspect not
That's my understanding too, RDC support in Clang has lagged NVCC.
Regarding the static-RDC builds, I still cannot reproduce any failures. I have run the Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt build (using the scripts as described in the initial comment), several times on both white and ride, and still can't get any failures.
Does the build environment differ in any way in how the nightlies are run versus these atdm scripts? All of the stokhos tests fail with something like
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
which sounds like something is killing the linker. I am guessing either memory is exhausted or cpu overload.
Out of curiosity, are these builds done on the head node or a compute node? It doesn't seem to matter for me, as I have tried both and don't get any failures. However I am wondering if several other builds are going on at the same time overloading the machine.
@ibaned wrote:
I would recommend applications take some time to reflect on whether relying on RDC as a requirement is such a great idea.
SPARC could perhaps go without, but Aria right now really needs virtual method calls on device and would need a big redesign to go without.
In that case we should collectively put real pressure on NVIDIA to complete the RDC implementation.
@vbrunini or @rrdrake might like to comment here about Aria -- if I remember right, usersubs require dlopen
, even if we build static.
@etphipp asked:
Out of curiosity, are these builds done on the head node or a compute node?
Compute nodes.
Then I have no clue. There must be some difference in the environment between the automated tests and my builds because I can't get the builds to fail.
@etphipp said:
I can't get the builds to fail.
Can you copy and paste your exact commands after logging onto 'white' or 'ride' so I can try to reproduce?
On white:
TRILINOS_DIR=$HOME/project/atdm/atdm_trilinos/Trilinos
source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
cmake \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnvAllPtPackages.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Stokhos=ON \
$TRILINOS_DIR
make -j16 -k
Citation for what Roscoe noted: https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#using-separate-compilation-in-cuda
says: 6.3. Libraries The device linker has the ability to read the static host library formats (.a on Linux and Mac OS X, .lib on Windows). It ignores any dynamic (.so or .dll) libraries. The --library and --library-path options can be used to pass libraries to both the device and host linker. The library name is specified without the library file extension when the --library option is used.
@etphipp when RDC support in clang is required you need to link with nvcc.
@hcedwar,
Any plans/hope for support for RDC with shared libs a future version of CUDA?
I have no info on this.
@etphipp
Using ninja -j64
(which is what the automated builds use) I was able to reproduce the Stokhos link failures (see details below). When I backed off to ninja -j16
, Stokhos built and ran fine (see details below).
So it seems the problem is that building with RDC is more sensitive to building with a higher build parallelism level. Is this due to exhausting memory on the system? Is RDC binary code larger?
In any case, I will try running a full build with a lower parallel level (I will try 32) and see what happens. But if backing off the parallel build level fixes these types of problems then that is what we will do (and suffer a longer wall-clock time to build).
@bartlettroscoe was this a static build?
@lucbv asked:
@bartlettroscoe was this a static build?
https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#shared_static
@bartlettroscoe if you deduce the configuration of a build based on its name you will quickly have bad surprises, sorry for checking instead of making assumptions...
@lucbv said
if you deduce the configuration of a build based on its name you will quickly have bad surprises
What do you mean?
Thanks @bartlettroscoe. I agree that it is likely too high of parallelism in the build and OS is killing the linker. That was my suspicion all along.
Btw, this could have been sorted out much sooner had the instructions in the initial comment for this issue been consistent with what the automated builds actually use.
@etphipp said:
Btw, this could have been sorted out much sooner had the instructions in the initial comment for this issue been consistent with what the automated builds actually use.
Agreed. This was basically a copy and paste error.
@lucbv asked:
@bartlettroscoe was this a static build?
Sorry, I misread that the first time. Static libs are the default so if you leave them out of the build name, that is what you get as described in:
Just spending too much time in ATDM where 90% of the builds are static :-)
FYI: I will provide more details here in a bit but it appears that at least on 'ride' that if you reduce the parallel build level that the entire (implicitly) static Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
build completes successfully (passing all Stokhos and Paners tests for example) except for 29 build errors for TrlinosCouplings
(that occur even with build level ninja -j1
). I am now experimenting with the parallel link level (using the CMake Ninja generator "JOB POOL" feature) to see how high we can go and still get successful builds but not increasing our build wallclock time by more than necessary (which will especially will be an issue for a future CUDA PR build).
In the coming days we will:
Submit a PR to adjust the parallel build levels to get everything to build that we can.
Create new ATDM Trilinos GitHub Issues for lingering failures not related to the parallel build level like one for the TrilinosCouplings
build errors for the static Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
build on 'ride' (which is about identical to the current Trilinos CUDA PR build (#2464) and therefore would impact a future RDC CUDA PR build).
Create a new ATDM Trilinos GitHub Issue for problems with the cuda+rdc+shared builds. (And then either close it or mark it as "Stalled" since this is problem with NVIDA nvcc that we can't do anything about.)
Close this ATDM Trilinos GitHub issue citing these other new issues.
Sorry for the the hassle these RDC builds have caused. Hopefully soon we will have clean cuda+rdc+static builds all cleaned up and promoted so that we can maintain them so they will be in place when the APPs need them. (Given that the app ASC IC Sierra is already relying on RDC being turned on Trilinos needs to be supporting RDC build right now independent of what ATDM needs or does not need.)
Some initial experiments with just linking the 2623 *.exe
targets shows that ninja -j16
was able to link all the targets in the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
on ride (minus TrilinosCouplings
which does not build even with ninja -j1
). (See details below.)
Therefore, after I implement the TriBITS option to limit the number of link processes (see https://github.com/TriBITSPub/TriBITS/issues/281) I will be able to run experiments on the full build of objects and libraries to see if we can use 64 *.o
build processes but only 16 *.exe
link processes at the same time and get everything to build. (If not, I will test ramping down.)
It looks like at least some of the TrilinosCouplings link errors are due to multiple definitions of main(). For example, the FENL example link error is:
packages/stk/stk_unit_test_utils/libstk_unit_main.a(UnitTestMain.cpp.o): In function `main':
/ascldap/users/rabartl/Trilinos.base/BUILDS/RIDE/CTEST_S/Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt/SRC_AND_BUILD/Trilinos/packages/stk/stk_unit_test_utils/unit_main_lib/UnitTestMain.cpp:44: multiple definition of `main'
packages/trilinoscouplings/examples/fenl/CMakeFiles/TrilinosCouplings_fenl_ensemble.dir/main_ensemble.cpp.o:/ascldap/users/rabartl/Trilinos.base/BUILDS/RIDE/CTEST_S/Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt/SRC_AND_BUILD/Trilinos/packages/trilinoscouplings/examples/fenl/main_ensemble.cpp:978: first defined here
collect2: error: ld returned 1 exit status
It looks like stk/stk_unit_test_utils/libstk_unit_main.a
is being included in the link when it shouldn't be. This example has no dependency on STK, so it must be happening somewhere higher up in the dependency chain. Is there somewhat in Tribits to find out which target is inserting that dependency?
@etphipp asked:
Is there somewhat in Tribits to find out which target is inserting that dependency?
The easiest thing to do is to just enable the downstream thing you want to know the upstream dependencies for and then watch the cmake STDOUT to see what gets enabled and why. For example, you would configure with:
-D Triilnos_ENABLE_TESTS=ON \
-D Triinos_ENABLE_TrilinosCouplings=ON \
and then watch cmake STDOUT (redirected to a file).
Actually TrilinosCouplings defines a dependency on all of STK as you can see here:
which shows:
SET(LIB_OPTIONAL_DEP_PACKAGES
EpetraExt Isorropia Amesos AztecOO Belos Ifpack ML MueLu NOX Zoltan STK Stokhos)
Better question is what is a main()
function doing inside of a package's library? That makes no sense.
Yeah, I just saw that myself.
The main()
is coming from the Unit_test_utils subpackage, which is probably just meant for unit testing, and so it probably isn't expected anyone would include it as a dependency that didn't need it.
So I guess whoever owns the examples that use STK needs to redo the dependencies to only include the relevant subpackages so that Unit_test_utils isn't included.
Actually there is only one example in TrilinosCouplings that declares it uses STK, and it is this one in examples/scaling:
IF(${PACKAGE_NAME}_ENABLE_Epetra AND ${PACKAGE_NAME}_ENABLE_EpetraExt AND
${PACKAGE_NAME}_ENABLE_Amesos AND ${PACKAGE_NAME}_ENABLE_AztecOO AND
${PACKAGE_NAME}_ENABLE_Intrepid AND ${PACKAGE_NAME}_ENABLE_ML AND
${PACKAGE_NAME}_ENABLE_SEACAS AND ${PACKAGE_NAME}_ENABLE_STK
AND ${PACKAGE_NAME}_ENABLE_STKIO AND ${PACKAGE_NAME}_ENABLE_STKMesh)
TRIBITS_ADD_EXECUTABLE(
Example_Poisson_STK
SOURCES example_Poisson_stk.cpp
)
TRIBITS_COPY_FILES_TO_BINARY_DIR(CopyMeshFilesSTK
SOURCE_FILES unit_cube_10int_hex.exo
unit_cube_5int_tet.exo
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}
DEST_DIR ${CMAKE_CURRENT_BINARY_DIR}
)
ENDIF()
If that logic is actually correct, it only needs to depend on STKIO and STKMesh, and they should be optional test dependencies and not library dependencies.
According to git-blame, @jhux2 created that example in this commit.
Yes we've run into trouble before with that "main library". At one point it seemed like it made sense in sierra's bjam system to put a main in a library so that multiple unit-test executables could reuse it. But in the trilinos system it seems to cause more trouble than it's worth...
@etphipp said:
The main() is coming from the Unit_test_utils subpackage, which is probably just meant for unit testing, and so it probably isn't expected anyone would include it as a dependency that didn't need it.
Then it should be a test-only library, not a regular library. See TESTONLY and TESTONLYLIBS.
If that logic is actually correct, it only needs to depend on STKIO and STKMesh, and they should be optional test dependencies and not library dependencies.
Sounds like a 2-line change in the Dependencies.cmake to fix that. Can someone fix this?
Sounds perfect, I didn't know about that. Is TESTONLY a tribits thing or a cmake thing? In any case, I'll try to add that asap.
@alanw0 asked:
Is TESTONLY a tribits thing or a cmake thing?
It is a TriBITS thing. Since TriBITS installs regular libraries and links these to downstream libs and executables by default, there had to be a way to opt-out of that. The only purpose for a library that is not installed and not linked to production libraries and exectuables is to use them in tests and examples in the build tree and hence "TESTONLY".
If you grep:
$ cd Trilinos/
$ find . -name "CMakeListx.txt" -exec grep -nH TESTONLY {} \;
you should see several examples of this usage.
There is even a tested example in TribitsExampleProject.
Let me know if you have any problems or questions with this.
Thanks Ross @bartlettroscoe , I've got a build going now, and if it looks good I'll put in a pull-request.
@alanw0 said:
I've got a build going now, and if it looks good I'll put in a pull-request.
Are you only adding 'TESTONLY' to some STK libs or are you also fixing the dependencies in TrilinosCouplings? If you are not fixing TrilinosCouplings dependencies on STK, I will go ahead and fix that, test it locally, and put in a separate PR.
CC: Trilinos Product areas leads: @jwillenbring, @rppawlo, @kddevin, @mperego, @srajama1
Other CC: @bartlettroscoe @fryeguy52
Next Action Status
Next: Waiting for PR #4761 to get tested, approved, and merged ...
Description
As shown here, there are several failing tests and build errors in the builds:
all test failures
build error output
These are builds that enable cuda relocatable device code. Most of the errors look something like:
or
Current Status on CDash
Steps to Reproduce
One should be able to reproduce this failure on ride or white as described in:
More specifically, the commands given for ride or white are provided at:
For the "-pt" builds
<build-name>
:Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt
Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-shared-release-debug-pt
and for
<Package>
=Kokkos
,KokkosKernels
,Belos
, etc., the commands to reproduce the build and test failures should be:For the other builds with
<build-name>
:and for
<Package>
=Kokkos
,KokkosKernels
,Belos
, etc., the commands to reproduce the build and test failures on 'white' or 'ride' should be:The build and test failures for the 'waterman' builds with
<build-name>
:Trilinos-atdm-waterman-cuda-9.2-rdc-release-debug
Trilinos-atdm-waterman-cuda-9.2-rdc-shared-release-debug
one uses: