CUDA 10.1 - Githubissues

delayedchoice commented 5 years ago

Hello, just curious if 10.1 is on your radar yet.

Thanks!

jcuda commented 5 years ago

Yes, I've seen the announcement in the NVIDIA newsletter. From quickly skimming over the release notes, there don't seem too many significant changes (which could either mean that I can do the update quickly, or that it's not really urgent - not sure about this yet ;-)). I'll try to allocate some time for a closer look, and do the update ASAP. I'll keep you posted here.

blueberry commented 5 years ago

@jcuda I've just installed CUDA 10.1 on macos (without nvidia hardware) and tried to build the existing JCuda 10.0

Good news: this seems to work, in a sense that the lack of nvidia hardware was not a deal breaker.
Expected news: It seems that there is something that should be updated to 10.1. when I tried make I got 29 warnings and this single error:

                                   ^
/Users/dragan/workspace/java/jcuda/jcuda/JCudaDriverJNI/src/JCudaDriver.cpp:9215:15: error: 
      no matching function for call to 'cuStreamBeginCapture_v2'
        int result = cuStreamBeginCapture(nativeHStream);

All in all, I think once the build works on Windows and Linux, I will probably be able to build it for the Mac too! (#23)

jcuda commented 5 years ago

That's great to hear!

Regarding the error: Comparing 10.0 https://docs.nvidia.com/cuda/archive/10.0/cuda-driver-api/group__CUDA__STREAM.html#group__CUDA__STREAM_1gea22d4496b1c8d02d0607bb05743532f 10.1 https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__STREAM.html#group__CUDA__STREAM_1g767167da0bbf07157dc20b6c258a2143 it indeed seems like there was an API change, and another parameter has been added to cuStreamBeginCapture. But this function is part of the new graph API, which is a bit difficult and not really supported right now anyhow...

blueberry commented 5 years ago

@jcuda Hi Marco. Any plans regarding this? I am available for help...

delayedchoice commented 5 years ago

Hey, Dragan -- just some FYI in case you didn't realize. I ran into this problem because somewhere the neanderthal documentation says to install the latest cuda... I think this issue might be avoided by newbies if that reads to install 10.0 until this issue is resolved.

Thanks again for everything!

jcuda commented 5 years ago

OK, admittedly, I considered this as "low priority", because as far as I see, there are no really significant changes or extensions in 10.1. But I see that compatibility and the desire to have the latest version indeed is an issue here.

I'll try to do the update next week (I already scheduled a 3 day vacation for next week anyhow).

blueberry commented 5 years ago

@jcuda It is not a big step feature-vise, but this time we can include macOS binaries, which would be a significant step compatibility-wise.

jcuda commented 5 years ago

Some delays here. Sorry, the same hassle again as in https://github.com/jcuda/jcuda-main/issues/25#issuecomment-396707465 and commented upon with refreshing clarity in https://forums.geforce.com/default/topic/525866/geforce-experience/driver-update-stuck-at-34-preparing-for-installation-34-/post/6028394/#6028394

I've given it several tries, each time the installer is taking about an hour (!) saying "Preparing for installation", and afterwards partially installs some parts but never seems to fully complete the installation. I'll keep trying, but my time is limited.

blueberry commented 5 years ago

Huh, that sucks. I have never done the installation on windows. On Linux and macOS, I never experienced problems. The only thing that comes to my mind is that the old drivers/cuda toolkit should be uninstalled (the latest installer also handles that part on linux).

jcuda commented 5 years ago

OK, after leaving my PC alone for one hour while "Preparing for installation" (in which it seemingly did nothing - no CPU load, no IO...), it proceeded with some other toolkit parts. It then said ~"Installing the driver". And after another hour or so (again, with seemingly doing nothing) it finished the installation :1st_place_medal: So regardless of whether NVIDIA severely messed up things there or just wants to drive people crazy by showing an indeterminate progress bar for 2 hours, I'll tackle the actual update now.

jcuda commented 5 years ago

So with some delay (apologies for that), the update for CUDA 10.1 should be done now.

There had not been sooo many changes, actually, but some of them may impact users on the API level:

Some runtime launch configuration calls are deprecated
Texture- and surface management functions have been deprecated
The nvGRAPH library is deprecated
Important: The cudaError codes have been renumbered! (Pity for those who did error checks with if (error==37) ... instead of if (error == cudaError.cudaErrorInvalidSurface) ... ...)

Additionally, the CUSPARSE and CUSOLVER libraries have been extended considerably. This did not cause much effort per se, because they are largely covered by automatic code generation. But there are currently very few samples/tests for these libraries. I'll have to allocate more time to port some of the samples from https://docs.nvidia.com/cuda/cusolver/index.html#cuSolverRF_examples to Java.

As an upside of the delay: The current state includes JCudnn for cuDNN 7.6.0, which has been released only a few days ago. (Imagine I had done the update last week - that would have been annoying...)

The native libraries for Linux and Mac can be built based on the current state of the repos, which is also tagged as version-10.1.0-RC01.

If there are any problems, please let me know.

blueberry commented 5 years ago

I've been struggling with the following error when building JCuda 10.1 on linux with CUDA 10.1 and cuDNN 7.6:

Preparation steps: I followed the jcuda-main building instructions. cmake was compaining about not finding CUDA_cuDNN_LIBRARY, so I had to set this manually in cmake-gui, and it works now.

Then it came to make. The build goes fine (with some warnings), but then JCusparse break with a int/int64_t conversion error:

` make
[ 16%] Built target JCudaCommonJNI [ 23%] Built target JCudaDriver [ 30%] Built target JCudaRuntime [ 36%] Built target JNvrtc [ 43%] Built target JCublas [ 50%] Built target JCublas2 [ 56%] Built target JCufft [ 63%] Built target JCurand [ 66%] Building CXX object jcusparse/JCusparseJNI/bin/CMakeFiles/JCusparse.dir/src/JCusparse.cpp.o /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp: In function ‘jint Java_jcuda_jcusparse_JCusparse_cusparseCooGetNative(JNIEnv, jclass, jobject, jobject, jobject, jobject, jobject, jobject, jobject, jintArray, jintArray, jintArray)’: /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50540:75: error: cannot convert ‘int’ to ‘int64_t’ {aka ‘long int’} cusparseStatus_t jniResult_native = cusparseCooGet(spMatDescr_native, rows_native, cols_native, nnz_native, cooRowInd_native, cooColInd_native, cooValues_native, idxType_native, idxBase_native, valueType_native); ^~~ In file included from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse_common.hpp:36, from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:30: /usr/local/cuda/include/cusparse.h:7421:43: note: initializing argument 2 of ‘cusparseStatus_t cusparseCooGet(cusparseSpMatDescr_t, int64_t, int64_t, int64_t*, void, void, void*, cusparseIndexType_t, cusparseIndexBase_t, cudaDataType)’ int64_t* rows,


/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp: In function ‘jint Java_jcuda_jcusparse_JCusparse_cusparseSpMatSetNumBatchesNative(JNIEnv*, jclass, jobject, jint)’:
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50658:41: error: ‘cusparseSpMatSetNumBatches’ was not declared in this scope
     cusparseStatus_t jniResult_native = cusparseSpMatSetNumBatches(spMatDescr_native, batchCount_native);
                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50658:41: note: suggested alternative: ‘cusparseSpMatSetValues’
     cusparseStatus_t jniResult_native = cusparseSpMatSetNumBatches(spMatDescr_native, batchCount_native);
                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~
                                         cusparseSpMatSetValues
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp: In function ‘jint Java_jcuda_jcusparse_JCusparse_cusparseSpMatGetNumBatchesNative(JNIEnv*, jclass, jobject, jintArray)’:
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50696:41: error: ‘cusparseSpMatGetNumBatches’ was not declared in this scope
     cusparseStatus_t jniResult_native = cusparseSpMatGetNumBatches(spMatDescr_native, batchCount_native);
                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50696:41: note: suggested alternative: ‘cusparseSpMatGetValues’
     cusparseStatus_t jniResult_native = cusparseSpMatGetNumBatches(spMatDescr_native, batchCount_native);
                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~
                                         cusparseSpMatGetValues
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp: In function ‘jint Java_jcuda_jcusparse_JCusparse_cusparseDnMatGetNative(JNIEnv*, jclass, jobject, jlongArray, jlongArray, jlongArray, jobject, jintArray, jintArray)’:
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50859:77: error: invalid conversion from ‘size_t*’ {aka ‘long unsigned int*’} to ‘int64_t*’ {aka ‘long int*’} [-fpermissive]
     cusparseStatus_t jniResult_native = cusparseDnMatGet(dnMatDescr_native, rows_native, cols_native, ld_native, valuesPtr_native, type_native, order_native);
                                                                             ^~~~~~~~~~~
In file included from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse_common.hpp:36,
                 from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:30:
/usr/local/cuda/include/cusparse.h:7496:45: note:   initializing argument 2 of ‘cusparseStatus_t cusparseDnMatGet(cusparseDnMatDescr_t, int64_t*, int64_t*, int64_t*, void**, cudaDataType*, cusparseOrder_t*)’
                  int64_t*                   rows,
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50859:90: error: invalid conversion from ‘size_t*’ {aka ‘long unsigned int*’} to ‘int64_t*’ {aka ‘long int*’} [-fpermissive]
     cusparseStatus_t jniResult_native = cusparseDnMatGet(dnMatDescr_native, rows_native, cols_native, ld_native, valuesPtr_native, type_native, order_native);
                                                                                          ^~~~~~~~~~~
In file included from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse_common.hpp:36,
                 from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:30:
/usr/local/cuda/include/cusparse.h:7497:45: note:   initializing argument 3 of ‘cusparseStatus_t cusparseDnMatGet(cusparseDnMatDescr_t, int64_t*, int64_t*, int64_t*, void**, cudaDataType*, cusparseOrder_t*)’
                  int64_t*                   cols,
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp: In function ‘jint Java_jcuda_jcusparse_JCusparse_cusparseDnMatGetStridedBatchNative(JNIEnv*, jclass, jobject, jintArray, jlongArray)’:
/home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:50947:108: error: invalid conversion from ‘size_t*’ {aka ‘long unsigned int*’} to ‘int64_t*’ {aka ‘long int*’} [-fpermissive]
     cusparseStatus_t jniResult_native = cusparseDnMatGetStridedBatch(dnMatDescr_native, batchCount_native, batchStride_native);
                                                                                                            ^~~~~~~~~~~~~~~~~~
In file included from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse_common.hpp:36,
                 from /home/dragan/workspace/java/jcuda/jcusparse/JCusparseJNI/src/JCusparse.cpp:30:
/usr/local/cuda/include/cusparse.h:7519:57: note:   initializing argument 3 of ‘cusparseStatus_t cusparseDnMatGetStridedBatch(cusparseDnMatDescr_t, int*, int64_t*)’
                              int64_t*                   batchStride);
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
make[2]: *** [jcusparse/JCusparseJNI/bin/CMakeFiles/JCusparse.dir/build.make:63: jcusparse/JCusparseJNI/bin/CMakeFiles/JCusparse.dir/src/JCusparse.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:533: jcusparse/JCusparseJNI/bin/CMakeFiles/JCusparse.dir/all] Error 2
make: *** [Makefile:84: all] Error 2
`

blueberry commented 5 years ago

I forgot to mention that searching for the solution led me to suggestions to use -fpermissive flag, but that did not help.

blueberry commented 5 years ago

Another detail: I'm doing this with CUDA 10.1 Update 1, the latest release that I've downloaded today.

blueberry commented 5 years ago

Something strange is happening with JCusolver even on macOS. It seems it has some leftovers from 10.0.

I managed to build native libraries, and when I run mvn clean install in jcuda-main (according to @jcuda guide), it goes well until it reaches jcusolver 10.0.0!!!!! see the tail of the output:

[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] JCuda .............................................. SUCCESS [ 2.037 s] [INFO] jcuda-common ....................................... SUCCESS [ 1.411 s] [INFO] jcuda-natives ...................................... SUCCESS [ 0.282 s] [INFO] jcuda .............................................. SUCCESS [ 11.281 s] [INFO] jcublas-natives .................................... SUCCESS [ 0.242 s] [INFO] jcublas ............................................ SUCCESS [ 6.913 s] [INFO] jcufft-natives ..................................... SUCCESS [ 0.141 s] [INFO] jcufft ............................................. SUCCESS [ 3.010 s] [INFO] jcurand-natives .................................... SUCCESS [ 0.194 s] [INFO] jcurand ............................................ SUCCESS [ 2.669 s] [INFO] jcusparse-natives .................................. SUCCESS [ 0.233 s] [INFO] jcusparse .......................................... SUCCESS [ 8.841 s] [INFO] jcusolver-natives 10.0.0 ........................... SUCCESS [ 0.182 s] [INFO] jcusolver 10.0.0 ................................... FAILURE [ 1.368 s] [INFO] jnvgraph-natives ................................... SKIPPED [INFO] jnvgraph ........................................... SKIPPED [INFO] jcudnn-natives ..................................... SKIPPED [INFO] jcudnn ............................................. SKIPPED [INFO] jcuda-main 10.1.0 .................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 39.331 s [INFO] Finished at: 2019-05-28T17:43:03+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project jcusolver: Could not resolve dependencies for project org.jcuda:jcusolver:jar:10.0.0: The following artifacts could not be resolved: org.jcuda:jcuda-natives:jar:apple-x86_64:10.0.0, org.jcuda:jcusparse-natives:jar:apple-x86_64:10.0.0: Could not find artifact org.jcuda:jcuda-natives:jar:apple-x86_64:10.0.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :jcusolver

blueberry commented 5 years ago

@jcuda IMPORTANT: It seems that the problem is that you forgot to push the recent updates to https://github.com/jcuda/jcusolver so the code for 10.1.0 is not on GitHub yet.

jcuda commented 5 years ago

Indeed, there was a conflict in jcusolver that caused my doPushAll.bat file to bail out (needed a git pull before). Sorry about that *blushes*. The latest version should now be in the repo.

jcuda commented 5 years ago

The fact that the cuDNN path has to be set manually is due to the fact that cuDNN is not part of the "core" CUDA package, and can thus not be found by the FindCUDA.cmake script. So this is expected.

I'm still trying to figure out the reason for the JCusparse error message, though. In fact, JCusparseSp did not change between 10.0 and 10.1. The confusing thing is that it seems to come from the cusparse.h header, as of the message cusparse.h:7519:57, but I may have to take a closer look at this if the problem remains.

blueberry commented 5 years ago

@jcuda I've found the source of the error: I was using the latest CUDA Toolkit 10.1 Update 1 (10.1.168) (May 2019) while you seem to use 10.1.105 from February. The offending functions were removed from the header in the new version (WTF!). I've installed 10.1.105 and now the build passes. I'll upload the binaries tonight.

blueberry commented 5 years ago

Here are linux jars from the jcuda-main/output folder:

jcuda-linux-jars.zip

blueberry commented 5 years ago

All cublas tests in neanderthal pass (more than 1000). Nice!

blueberry commented 5 years ago

And the macos jars:

jcuda-1.10.1.105-osx-jars.zip

(nevermind the 1.10.blabla typo. the version is 10.1.105)

jcuda commented 5 years ago

Great, thanks @blueberry !

I'll schedule the release ASAP. Fortunately, we have a holiday tomorrow, so I might be able to do it tomorrow, but likely not later than during the weekend.

The fact that they...

Removed functions between 10.0 and 10.1
Removed even more functions between 10.1.105 and 10.1.168
Entirely deprecated the nvGRAPH library between 10.0 and 10.1 ... shows that they are not taking https://semver.org/ toooo seriously :-/

jcuda commented 5 years ago

So after a (somewhat embarrassingly long) delay, the bundles for JCuda 10.1.0 have been uploaded and should soon be available in Maven Central:

<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcuda</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcublas</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcufft</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcusparse</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcusolver</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcurand</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jnvgraph</artifactId>
    <version>10.1.0</version>
</dependency>
<dependency>
    <groupId>org.jcuda</groupId>
    <artifactId>jcudnn</artifactId>
    <version>10.1.0</version>
</dependency>

A huge thanks and :+1: to @blueberry for your continued support (and patience), and for providing the Linux- and MacOS natives!

_{(I'll leave this issue open until the READMEs have been updated - and the website, which is still at 0.9.2. ""Fortunately"", in a few months, most of my contracts end, maybe then I'll find a bit more time for maintenance and updates...)}

blueberry commented 5 years ago

Thank you for a wonderful software, Marco!

jcuda commented 5 years ago

The website and README (actually, only USAGE.md) have been updated.

jcuda commented 5 years ago

@blueberry Is what I said in https://forum.byte-welt.net/t/why-jcuda-10-1-requires-usr-local-bfm-lib64-libstdc-so-6-version-cxxabi-1-3-8/21225/2?u=marco13 correct, i.e. are the linux natives compiled with a recent version of GCC (>= 4.9)?

blueberry commented 5 years ago

I think it was GCC 6.

jcuda / jcuda-main

CUDA 10.1 #29