Make it work with the GPU

[kwalcock]

Onnx supports many forms of hardware acceleration, including GPUs. For a Java interface on Linux and Windows, there is CUDA/cuDNN and also CUDA/TensorRT. So far I have only tried the former and only on Linux. Apple does not do CUDA, but it has CoreML. A page of documentation on that says "The CoreML EP can be used via the C or C++ APIs currently. Additional support via the Objective-C API is in progress." so that Java might not really be included yet.

However, speedup is not necessarily impressive. For the TokenClassifierTimerApp that is part of the scala-transformers project, I measured the values below for 10,000 sentences. I suspect that parallelizing on the GPU may not be possible. There are several reports along those lines: Parallel execution mode is deprecated, Parallel execution mode does not support the CUDA Execution Provider.

Timing (minutes)	Serial	Parallel
CPU	3:56	1:45
GPU	3:04	crash

This was done on a significantly low-end GPU and a higher end CPU, so maybe it could be faster. Parallelism on the CPU was more effective.

One of the GPU crash messages has to do with sequential execution:

2023-05-24 13:02:41.408722743 [E:onnxruntime:, sequential_executor.cc:494 ExecuteKernel] Non-zero status code returned while running Gemm node. Name:'Gemm_1373' Status Message: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:124 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char, const char, ERRTYPE, const char) [with ERRTYPE = cublasStatus_t; bool THRW = true; std::conditional_t<THRW, void, onnxruntime::common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:117 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char, const char, ERRTYPE, const char) [with ERRTYPE = cublasStatus_t; bool THRW = true; std::conditional_t<THRW, void, onnxruntime::common::Status> = void] CUBLAS failure 3: CUBLAS_STATUS_ALLOC_FAILED ; GPU=0 ; hostname=KWA-Aorus-Ubuntu ; expr=cublasCreate(&cublashandle);

[kwalcock]

It is working with the GPU under Windows as well. This is with an older computer. I have not verified that each of the runs gets the same answer. This one worked even in parallel on the GPU.

Timing (minutes)	Serial	Parallel
CPU	12:07	4:28
GPU	4:40	1:39

[kwalcock]

A regression-type test shows that the CPU and GPU versions get the same results and on the CPU, serial and parallel versions do as well. On the GPU, even when the parallel version doesn't crash, the results are different. Therefore:

Config	Serial	Parallel
CPU	✔	✔
GPU	✔	❌

[MihaiSurdeanu]

Thanks @kwalcock ! The speedup is impressive, so it's worth investigating what differences we observe. I suspect that some differences may be explainable simply due to hardware differences (e.g., rounding at fewer decimal points may cause ties that get solved randomly). But can you please generate a diff? Hopefully that will tell us if it's a more serious issue. Thanks!

[kwalcock]

Here is an example of the differences. However, both runs are on the GPU, so there should be no hardware problems, and the differences change between runs so that the values below can't be replicated. It seems like a threading issue.

589
Mr. Efthynoos GEORGOULIS Inspector of Forests , Ministry of Agriculture , Rue Anthistenous 26 , Athens Mr. Christos MOULOOPOULOS Professor of Forestry at the University of Salonika Italy Mr. Aldo PAVARI Director of the Forestry Research Station , Florence Mr. Giulio SACCHI Director General of the Forest Service , Ministry of Agriculture and Forests , Rome Mr. Giuseppe Ugo PAPI Secretary General of the Italian National FAO Committee , Ministry of Agriculture and Forests , Rome .
  actual: O B-PER I-PER O O B-ORG O B-ORG I-ORG I-ORG O B-PER I-PER O O B-LOC O B-PER I-PER O O I-ORG O O B-ORG I-ORG I-ORG B-LOC O B-PER I-PER O O O B-ORG I-ORG I-ORG I-LOC B-LOC B-PER B-PER I-PER B-MISC O O O B-ORG I-ORG I-LOC B-ORG I-ORG I-ORG I-ORG I-ORG I-PER B-LOC B-PER B-PER I-PER I-PER O I-ORG O O B-MISC B-ORG I-ORG I-ORG O B-ORG I-ORG I-ORG I-ORG I-ORG O B-LOC I-LOC NNP NNP NNP NNP IN NNPS , NNP IN NNP , NNP NNP CD , NNP NNP NNP NNP NNP IN NNP IN DT NNP IN NNP NNP NNP NNP NNP NNP IN DT NNP NNP NNP , NNP NNP NNP NNP NNP NNP IN DT NNP NNP , NNP IN NNP CC NNPS , NNP NNP NNP NNP NNP NNP NNP IN DT JJ NNP NNP NNP , NNP IN NNP CC NNPS , NNP . B-NP I-NP B-NP B-NP B-PP B-NP O B-NP B-PP B-NP O B-NP I-NP B-NP O B-NP B-NP I-NP I-NP B-NP B-PP B-NP B-PP B-NP I-NP B-PP B-NP B-NP B-NP I-NP I-NP B-NP B-PP B-NP I-NP I-NP I-NP O B-NP I-NP I-NP I-NP B-NP I-NP B-PP B-NP I-NP I-NP O B-NP B-PP B-NP I-NP I-NP O B-NP B-NP I-NP I-NP I-ADVP B-NP I-NP B-PP B-NP I-NP I-NP I-NP I-NP O B-NP B-PP B-NP I-NP I-NP O B-NP I-ADVP 2 1 1 -1 1 -2 -3 -4 1 -2 -7 1 -9 -1 -11 -13 2 1 -17 -1 1 -2 2 1 -5 1 -2 -9 2 1 1 -1 4 3 2 1 -5 -29 4 2 1 2 1 -2 3 2 1 -4 -41 5 1 -2 -1 -2 -45 4 3 2 1 2 1 -2 5 4 3 2 1 -6 -1 -2 1 -2 -1 -2 -5 -7 -56 compound compound compound dep case nmod punct appos case nmod punct compound appos dep punct appos compound compound dep dep case nmod case det nmod case nmod dep compound compound dep dep case det compound compound nmod punct compound compound compound compound compound dep case det compound nmod punct conj case nmod cc conj punct appos compound compound compound dep compound dep case det amod compound compound nmod punct appos case nmod cc conj punct appos punct
expected: O B-PER I-PER O O B-ORG O B-ORG I-ORG I-ORG O B-PER I-PER O O B-LOC O B-PER I-PER O O I-ORG O O B-ORG I-ORG I-ORG B-LOC O B-PER I-PER O O O B-ORG I-ORG I-ORG I-LOC B-LOC B-PER B-PER I-PER B-MISC O O O B-ORG I-ORG I-LOC B-ORG I-ORG I-ORG I-ORG I-ORG I-PER B-LOC B-PER B-PER I-PER I-PER O I-ORG O O B-MISC B-ORG I-ORG I-ORG O B-ORG I-ORG I-ORG I-ORG I-ORG O B-LOC I-LOC NNP NNP NNP NNP IN NNPS , NNP IN NNP , NNP NNP CD , NNP NNP NNP NNP NNP IN NNP IN DT NNP IN NNP NNP NNP NNP NNP NNP IN DT NNP NNP NNP , NNP NNP NNP NNP NNP NNP IN DT NNP NNP , NNP IN NNP CC NNPS , NNP NNP NNP NNP NNP NNP NNP IN DT JJ NNP NNP NNP , NNP IN NNP CC NNPS , NNP . B-NP I-NP B-NP B-NP B-PP B-NP O B-NP B-PP B-NP O B-NP I-NP B-NP O B-NP B-NP I-NP I-NP B-NP B-PP B-NP B-PP B-NP I-NP B-PP B-NP B-NP B-NP I-NP I-NP B-NP B-PP B-NP I-NP I-NP I-NP O B-NP I-NP I-NP I-NP B-NP I-NP B-PP B-NP I-NP I-NP O B-NP B-PP B-NP I-NP I-NP O B-NP B-NP I-NP I-NP I-ADVP B-NP I-NP B-PP B-NP I-NP I-NP I-NP I-NP O B-NP B-PP B-NP I-NP I-NP O B-NP I-ADVP 2 1 1 -1 1 -2 -3 -4 1 -2 -7 1 -9 -1 -11 -13 2 1 -17 -1 1 -2 2 1 -5 1 -2 -9 2 1 1 -1 4 3 2 1 -5 -27 4 2 1 2 1 -2 3 2 1 -4 -41 5 1 -2 -1 -2 -45 4 3 2 1 2 1 -2 5 4 3 2 1 -6 -1 -2 1 -2 -1 -2 -5 -7 -56 compound compound compound dep case nmod punct appos case nmod punct compound appos dep punct appos compound compound dep dep case nmod case det nmod case nmod dep compound compound dep dep case det compound compound nmod punct compound compound compound dep compound dep case det compound nmod punct conj case nmod cc conj punct appos compound compound compound dep compound dep case det amod compound compound nmod punct appos case nmod cc conj punct appos punct

730
As the sardines , anchovies , mackerel and perches constitute the major part of the pelagic fish p rduction on our coasts , more particularly on the west coast where totally different geographical and meteorological conditions prevail , three sub-stations have been established a t Bombay , Calicut and Karwar , separated from one another by a distance of about 300 miles .
  actual: O O O O O O O O O B-PER O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-LOC O B-LOC O B-LOC O O O O O O O O O O O O O IN DT NNS , NNS , NN CC NNS VBP DT JJ NN IN DT JJ NN NN NN IN PRP$ NNS , RBR RB IN DT NN NN WRB RB JJ JJ CC JJ NNS VBP , CD NNS VBP VBN VBN DT JJ NNP , NNP CC NNP , VBN IN CD DT IN DT NN IN RB CD NNS . B-SBAR B-NP I-NP I-NP I-NP I-NP I-NP I-NP I-NP B-VP B-NP I-NP I-NP B-PP B-NP I-NP I-NP I-NP I-NP B-PP B-NP I-NP O B-ADVP I-ADVP B-PP B-NP I-NP I-NP B-ADVP B-NP I-NP I-NP I-NP I-NP I-NP B-VP O B-NP I-NP B-VP I-VP I-VP B-NP I-NP I-NP I-NP I-NP I-NP I-NP O B-VP B-PP B-NP B-NP B-PP B-NP I-NP B-PP B-NP I-NP I-NP O 9 1 7 -1 -2 -3 -4 -5 -6 33 2 1 -3 5 4 3 2 1 -6 2 1 -3 -1 1 4 3 2 1 8 7 1 4 3 -1 -2 1 -8 5 1 3 2 1 0 2 1 -3 -1 -2 -3 -4 -5 -6 1 -2 -1 2 1 -6 3 1 1 -4 -18 mark det compound punct conj punct conj cc conj acl:relcl det amod dobj case det amod compound compound nmod case nmod:poss nmod punct advmod advmod case det compound nmod advmod advmod amod amod cc conj nsubj advcl punct nummod nsubj aux auxpass root det dep dobj punct conj cc conj punct acl case nmod dep case det nmod case advmod nummod nmod punct
expected: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-LOC O B-LOC O B-LOC O O O O O O O O O O O O O IN DT NNS , NNS , NN CC NNS VBP DT JJ NN IN DT JJ NN NN NN IN PRP$ NNS , RBR RB IN DT NN NN WRB RB JJ JJ CC JJ NNS VBP , CD NNS VBP VBN VBN DT JJ NNP , NNP CC NNP , VBN IN CD DT IN DT NN IN RB CD NNS . B-SBAR B-NP I-NP I-NP I-NP I-NP I-NP I-NP I-NP B-VP B-NP I-NP I-NP B-PP B-NP I-NP I-NP I-NP I-NP B-PP B-NP I-NP O B-ADVP I-ADVP B-PP B-NP I-NP I-NP B-ADVP B-NP I-NP I-NP I-NP I-NP I-NP B-VP O B-NP I-NP B-VP I-VP I-VP B-NP I-NP I-NP I-NP I-NP I-NP I-NP O B-VP B-PP B-NP B-NP B-PP B-NP I-NP B-PP B-NP I-NP I-NP O 9 1 7 -1 -2 -3 -4 -5 -6 33 2 1 -3 5 4 3 2 1 -6 2 1 -3 -1 1 4 3 2 1 8 7 1 4 3 -1 -2 1 -8 5 1 3 2 1 0 2 1 -3 -1 -2 -3 -4 -5 -6 1 -2 -1 2 1 -6 3 1 1 -4 -18 mark det compound punct conj punct conj cc conj advcl det amod dobj case det amod compound compound nmod case nmod:poss nmod punct advmod advmod case det compound nmod advmod advmod amod amod cc conj nsubj advcl punct nummod nsubj aux auxpass root det dep dobj punct conj cc conj punct acl case nmod dep case det nmod case advmod nummod nmod punct

815
The spawning condition ( Stage IV of the I.C.E.S .
  actual: O O O O O O O O O O DT VBG NN -LRB- NNP NNP IN DT NNP . B-NP O I-NP O I-NP I-NP B-PP B-NP I-NP O 1 -2 -2 1 1 -2 2 1 -1 -1 det dep dep punct compound dep case det dep punct
expected: O O O O O O O O B-LOC O DT NN NN -LRB- NN NNP IN DT NNP . B-NP I-NP I-NP O B-NP I-NP B-PP B-NP I-NP O 2 1 0 2 1 -3 2 1 -3 -7 det compound root punct dep appos case det nmod punct

859
stage and becomes static at the 48 mm .
  actual: O O O O O O O O O NN CC VBZ JJ IN DT CD NN . I-NP O B-VP B-NP B-PP B-NP B-NP I-NP O -6 -1 0 1 2 1 1 0 -30 dep cc root amod case det nummod dep punct
expected: O O O O O O O O O NN CC VBZ JJ IN DT CD NN . B-NP O B-VP B-ADJP B-PP B-NP I-NP I-NP O 0 -1 -2 -1 3 2 1 -5 -8 root cc conj xcomp case det nummod nmod punct

1889
15 ( 1898 ) 357 ; in De Zeeen Nederl .
  actual: O O O O O O O O O O O NN NN NN NN NN NN NN NN NN NN NN I-NP I-NP I-NP B-NP I-NP I-NP I-NP I-NP I-NP I-NP I-NP 1 1 1 -1 1 1 1 1 1 1 1 dep dep dep dep dep dep dep dep dep dep dep
expected: O O O O O O O B-LOC I-LOC I-LOC O CD -LRB- CD -RRB- CD : IN NNP NNP NNP . B-NP O B-NP O B-NP O B-PP B-NP I-NP I-NP O 0 1 -2 -1 -4 -6 3 2 1 -10 -1 root punct appos punct dep punct case compound compound nmod punct

2267
Prod .
  actual: O O -LRB- . B-NP O 7 -1 mark punct
expected: O O NN . B-NP O 0 -1 root punct

2815
It has the further advantage of not requiring full and complete records on all of the catch .
  actual: O O O O O O O O O O O O O O O O O B-ORG PRP VBZ DT RBR NN IN RB VBG JJ CC JJ NNS IN DT IN DT VB . B-NP B-VP B-NP B-NP I-NP B-PP B-ADVP B-VP B-NP O B-ADJP B-NP B-PP B-NP B-PP B-NP I-NP O -2 -1 1 1 -2 -1 2 -1 -2 -1 -1 -1 -1 -2 -1 1 -1 -49 dep dep det advmod dobj case neg acl dep cc dep dobj dep det case det dep punct
expected: O O O O O O O O O O O O O O O O O O PRP VBZ DT JJ NN IN RB VBG JJ CC JJ NNS IN DT IN DT NN . B-NP B-VP B-NP I-NP I-NP B-PP B-VP I-VP B-NP I-NP I-NP I-NP B-PP B-NP B-PP B-NP I-NP O 1 0 2 1 -3 2 1 -3 3 -1 -2 -4 1 -2 2 1 -3 -16 nsubj root det amod dobj mark neg acl amod cc conj dobj case nmod case det nmod punct

3720
- - 3 SOME RECENT ADVANCES IN THE STUDY OF THE BIOLOGY AND RACIAL DIVISION OF PACIFIC TUNAS by Milner B. Schaefer ' '' ABSTRACT Valuable results are being obtained from research into those factors of the life history of thc tunas which are of basic in1 ortance to the dewlo ment and management of the fisieries for tl ~ cm .
  actual: O O O O O O O O O O O O O O O O B-LOC O O B-PER B-ORG B-ORG B-ORG O B-ORG O O O O O B-ORG B-ORG O B-ORG B-PER O O O O O O O O O O O O O O O O O O O O O O O O O O O : : CD DT JJ NNS IN DT NN IN DT NN CC JJ NN IN NNP NN IN NNP NNP NNP NNP NNP NNP NN NNS VBP VBG NN NNP NNP NNP NNP NNP POS '' NN NN NN NN NNS VBG NN IN JJ NN NN TO DT FW NN CC NN IN DT NNS IN SYM . NN . O O O B-NP I-NP I-NP B-PP B-NP I-NP B-PP B-NP I-NP I-NP I-NP I-NP B-PP B-NP I-NP B-PP B-NP B-NP I-NP B-NP I-NP B-NP I-NP I-NP B-VP I-VP I-NP I-NP B-NP I-NP I-NP B-NP B-NP O I-NP I-NP I-NP I-NP I-NP I-VP I-NP B-PP B-NP I-NP O B-PP B-NP I-NP I-NP I-NP I-NP B-PP B-NP I-NP B-PP B-NP O B-NP O 5 4 3 2 1 0 2 1 -3 2 1 -3 -1 1 -3 2 1 -6 1 5 1 1 1 2 1 1 5 5 5 -3 1 1 1 1 5 -1 3 1 2 2 1 5 5 1 2 -2 1 -4 3 2 1 -4 -1 -2 2 1 -5 3 2 1 -4 -41 dep punct dep det amod root case det nmod case det nmod cc amod conj case compound nmod case nmod:poss compound compound compound compound compound dep nsubj aux dep amod compound compound compound compound nmod:poss case punct compound compound compound dep nsubj dep mwe case amod nmod nmod case det compound nmod cc conj case det nmod case dep punct dep punct
expected: O O O O O O O O O O O O O O O O B-LOC I-LOC O O I-ORG I-PER O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O : : CD DT JJ NNS IN DT NN IN DT NN CC JJ NN IN NNP NNP IN NNP NNP NNP POS '' NN JJ NNS VBP VBG VBN IN NN IN DT NNS IN DT NN NN IN SYM NNS WDT VBP IN JJ NN NN TO DT FW NN CC NN IN DT NNS IN SYM . NN . O O B-LST B-NP I-NP I-NP B-PP B-NP I-NP B-PP B-NP I-NP I-NP I-NP I-NP B-PP B-NP I-NP B-PP B-NP I-NP I-NP B-NP I-NP I-NP I-NP I-NP B-VP I-VP I-VP B-PP B-NP B-PP B-NP I-NP B-PP B-NP I-NP I-NP B-PP B-NP I-NP B-NP B-VP B-PP B-NP I-NP O B-PP B-NP I-NP I-NP O I-NP B-PP B-NP I-NP B-PP B-NP O I-NP O 5 4 3 2 1 0 2 1 -3 2 1 -3 -1 1 -3 2 1 -6 8 2 1 5 -1 3 2 1 3 2 1 0 1 -2 2 1 -3 3 2 1 -4 2 1 -3 5 4 1 2 -1 -2 3 2 1 -4 -1 -2 2 1 -5 3 2 1 -4 -25 dep punct dep det amod dep case det nmod case det nmod cc amod conj case compound nmod case compound compound nmod:poss case punct compound amod nmod aux auxpass root case nmod case det nmod case det compound nmod case dep nmod nsubj cop case acl:relcl dep dep case det compound nmod cc conj case det nmod case dep punct dep punct

8486
11938 ) .
  actual: O O O CD . . B-NP O O 0 -1 -2 root punct punct
expected: O O O CD -RRB- . B-NP O O 0 -1 -2 root punct punct

8773
- The methods followed in the collec % ion , laboratory analysis and computation of the data used in this paper are given in the introductory section of the respective station list of Australian oceanographicaldataissuedhy C.S .
  actual: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-MISC O O O : DT NNS VBN IN DT NN NN NN , NN NN CC NN IN DT NNS VBN IN DT NN VBP VBN IN DT JJ NN IN DT JJ NN NN IN JJ JJ NNP . O B-NP I-NP B-VP B-PP B-NP I-NP I-NP I-NP I-NP I-NP I-NP I-NP I-NP B-PP B-NP I-NP B-VP B-PP B-NP I-NP B-VP I-VP B-PP B-NP I-NP I-NP B-PP B-NP I-NP I-NP I-NP B-PP B-NP I-NP I-NP O 22 1 20 -1 4 3 2 1 -5 -1 1 -3 -4 -5 2 1 -8 -1 2 1 -3 1 0 3 2 1 -4 4 3 2 1 -5 3 2 1 -4 -14 punct det nsubj acl case det compound compound nmod punct compound conj cc conj case det nmod acl case det nmod auxpass root case det amod nmod case det amod compound nmod case amod amod nmod punct
expected: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-MISC O O O : DT NNS VBN IN DT NN NN NN , NN NN CC NN IN DT NNS VBN IN DT NN VBP VBN IN DT JJ NN IN DT JJ NN NN IN JJ JJ NNP . O B-NP I-NP B-VP B-PP B-NP I-NP I-NP I-NP I-NP I-NP I-NP I-NP I-NP B-PP B-NP I-NP B-VP B-PP B-NP I-NP B-VP I-VP B-PP B-NP I-NP I-NP B-PP B-NP I-NP I-NP I-NP B-PP B-NP I-NP I-NP O 21 1 20 -1 4 3 2 1 -5 -1 1 -3 -4 -5 2 1 -8 -1 2 1 -3 1 0 3 2 1 -4 4 3 2 1 -5 3 2 1 -4 -14 punct det nsubj acl case det compound compound nmod punct compound conj cc conj case det nmod acl case det nmod auxpass root case det amod nmod case det amod compound nmod case amod amod nmod punct

9537
Some of the earliest batches of eggs secured in plankton behaved , however , in an entirely different manner .
  actual: O O O O O O O O O O O O O O O O O O O B-MISC DT IN DT JJS NNS IN NNS VBN IN NN VBN , RB , IN DT RB JJ NN . B-NP B-PP B-NP B-NP I-NP B-PP I-NP I-NP B-PP I-NP B-VP O B-ADVP O B-PP B-NP B-ADJP B-NP I-NP O 1 2 1 2 -4 2 -3 2 2 1 -1 -1 1 -1 2 -1 1 1 1 -41 det case det amod appos case dep amod case dep dep punct advmod punct case nmod:npmod advmod amod appos punct
expected: O O O O O O O O O O O O O O O O O O O O DT IN DT JJS NNS IN NNS VBN IN NN VBD , RB , IN DT RB JJ NN . B-NP B-PP B-NP I-NP I-NP B-PP B-NP B-VP B-PP B-NP B-VP O B-ADVP O B-PP B-NP I-NP I-NP I-NP O 10 3 2 1 -4 1 -2 -3 1 -2 0 -1 -2 -3 4 3 1 1 -8 -9 nsubj case det amod nmod case nmod acl case nmod root punct advmod punct case det advmod amod nmod punct

[MihaiSurdeanu]

Thanks! Yes, it does seem like the differences indicate a bug... May be due to parallelism. May be something else.

When you run the sequential code multiple times, do you get the same outputs on GPU and CPU on different runs?

[kwalcock]

So far, the three other variations have produced the same, consistent results. That's serial on either CPU or GPU and parallel on CPU.

[MihaiSurdeanu]

Thanks! I suggest we disable parallel runs on the GPU if we can then.

[kwalcock]

Although I have marked the CPU and GPU serial versions both as good, they are not the same. I have not seen differences in the tags, but the actual vectors that they calculate are different, usually in the 5th or later decimal point. This means that there are probably situations in which they could produce different answers. When writing a paper, one would want to specify some hardware or provide the output files in order to achieve a repeatable or verifiable process. The GPU parallel version is still inconsistent.

[MihaiSurdeanu]

Yes, this is a known issue in deep learning. There is a journal paper on this, if you want to read more.