Closed AuroraRAS closed 1 year ago
Can you build dlprimitives and run some basic tests - I want to see if there an issue with driver or something else
run some basic test from there https://github.com/artyom-beilis/dlprimitives/blob/master/docs/build.md#benchmarking to see if it runs at all.
Can you build dlprimitives and run some basic tests - I want to see if there an issue with driver or something else
run some basic test from there https://github.com/artyom-beilis/dlprimitives/blob/master/docs/build.md#benchmarking to see if it runs at all.
cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo
make
make test
Running tests...
Test project /home/ml/Projects/pytorch_dlprim/dlprimitives/build
Start 1: test_test_case_abs
1/33 Test #1: test_test_case_abs ............... Passed 1.04 sec
Start 2: test_test_case_activation
2/33 Test #2: test_test_case_activation ........ Passed 3.85 sec
Start 3: test_test_case_batchnorm
3/33 Test #3: test_test_case_batchnorm ......... Passed 13.00 sec
Start 4: test_test_case_concat
4/33 Test #4: test_test_case_concat ............ Passed 0.28 sec
Start 5: test_test_case_conv2d
5/33 Test #5: test_test_case_conv2d ............ Passed 303.66 sec
Start 6: test_test_case_conv2d_dsc
6/33 Test #6: test_test_case_conv2d_dsc ........ Passed 325.08 sec
Start 7: test_test_case_conv2d_gemm
7/33 Test #7: test_test_case_conv2d_gemm ....... Passed 342.08 sec
Start 8: test_test_case_conv2d_win
8/33 Test #8: test_test_case_conv2d_win ........***Failed 35.25 sec
Start 9: test_test_case_elementwise
9/33 Test #9: test_test_case_elementwise ....... Passed 28.48 sec
Start 10: test_test_case_global_pooling
10/33 Test #10: test_test_case_global_pooling .... Passed 13.58 sec
Start 11: test_test_case_hardtanh
11/33 Test #11: test_test_case_hardtanh .......... Passed 1.49 sec
Start 12: test_test_case_inner_product
12/33 Test #12: test_test_case_inner_product ..... Passed 51.32 sec
Start 13: test_test_case_log_softmax
13/33 Test #13: test_test_case_log_softmax ....... Passed 1.20 sec
Start 14: test_test_case_mse_loss
14/33 Test #14: test_test_case_mse_loss .......... Passed 1.39 sec
Start 15: test_test_case_nll_loss
15/33 Test #15: test_test_case_nll_loss .......... Passed 1.01 sec
Start 16: test_test_case_param
16/33 Test #16: test_test_case_param ............. Passed 0.46 sec
Start 17: test_test_case_pooling2d
17/33 Test #17: test_test_case_pooling2d ......... Passed 155.98 sec
Start 18: test_test_case_reduction
18/33 Test #18: test_test_case_reduction ......... Passed 57.60 sec
Start 19: test_test_case_slice
19/33 Test #19: test_test_case_slice ............. Passed 0.46 sec
Start 20: test_test_case_softmax
20/33 Test #20: test_test_case_softmax ........... Passed 1.17 sec
Start 21: test_test_case_softmax_loss
21/33 Test #21: test_test_case_softmax_loss ...... Passed 1.21 sec
Start 22: test_test_case_threshold
22/33 Test #22: test_test_case_threshold ......... Passed 1.40 sec
Start 23: test_test_case_tr_conv2d
23/33 Test #23: test_test_case_tr_conv2d ......... Passed 167.49 sec
Start 24: test_test_case_tr_conv2d_dsc
24/33 Test #24: test_test_case_tr_conv2d_dsc ..... Passed 180.49 sec
Start 25: test_test_case_tr_conv2d_gemm
25/33 Test #25: test_test_case_tr_conv2d_gemm .... Passed 186.64 sec
Start 26: test_test_case_tr_conv2d_win
26/33 Test #26: test_test_case_tr_conv2d_win ..... Passed 154.16 sec
Start 27: test_net
27/33 Test #27: test_net ......................... Passed 8.95 sec
Start 28: test_net_nonopt
28/33 Test #28: test_net_nonopt .................. Passed 8.94 sec
Start 29: test_json
29/33 Test #29: test_json ........................ Passed 0.01 sec
Start 30: test_random
30/33 Test #30: test_random ...................... Passed 0.97 sec
Start 31: test_context
31/33 Test #31: test_context ..................... Passed 0.29 sec
Start 32: test_util
32/33 Test #32: test_util ........................ Passed 29.67 sec
Start 33: test_broadcast_reduce
33/33 Test #33: test_broadcast_reduce ............ Passed 33.51 sec
97% tests passed, 1 tests failed out of 33
Total Test time (real) = 2112.12 sec
The following tests FAILED:
8 - test_test_case_conv2d_win (Failed)
Errors while running CTest
Output from these tests are in: /home/ml/Projects/pytorch_dlprim/dlprimitives/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [Makefile:71: test] Error 8
8/33 Test for LastTestsFailed.log
8/33 Testing: test_test_case_conv2d_win
8/33 Test: test_test_case_conv2d_win
Command: "~/Projects/pytorch_dlprim/dlprimitives/build/test_from_template" "0:0" "~/Projects/pytorch_dlprim/dlprimitives/tests/test_case_conv2d_win.json"
Directory: ~/Projects/pytorch_dlprim/dlprimitives/build
"test_test_case_conv2d_win" start time: Jan 18 00:42 CST
Output:
----------------------------------------------------------
Running tests for operator Convolution2D on AMD Radeon R9 200 Series (pitcairn, LLVM 15.0.6, DRM 3.49, 6.1.6-200.fc37.x86_64) on Clover
- Running test case 0 for options {"activation":"relu","bias":true,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":98,"channels_out":128,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":5,"pad":2,"stride":1}
-- test for shape [[1,98,7,7]] fwd,bwd
-- test for shape [[1,98,8,8]] fwd,bwd
-- test for shape [[1,98,4,4]] fwd,bwd
- Running test case 1 for options {"bias":false,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":1,"channels_out":1,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":3,"pad":1,"stride":1}
-- test for shape [[1,1,2,2]] fwd,bwd
-- test for shape [[1,1,7,7]] fwd,bwd
-- test for shape [[1,1,8,8]] fwd,bwd
-- test for shape [[1,1,4,4]] fwd,bwd
-- test for shape [[3,1,4,4]] fwd,bwd
-- test for shape [[2,1,7,7]] fwd,bwd
-- test for shape [[2,1,10,5]] fwd,bwd
-- test for shape [[2,1,10,10]] fwd,bwd
-- test for shape [[2,1,19,19]] fwd,bwd
-- test for shape [[2,1,20,20]] fwd,bwd
-- test for shape [[2,1,32,32]] fwd,bwd
-- test for shape [[64,1,64,64]] fwd,bwd
-- test for shape [[53,1,100,100]] fwd,bwd
- Running test case 2 for options {"bias":false,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":2,"channels_out":1,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":3,"pad":1,"stride":1}
-- test for shape [[1,2,2,2]] fwd,bwd
-- test for shape [[1,2,7,7]] fwd,bwd
-- test for shape [[1,2,8,8]] fwd,bwd
-- test for shape [[1,2,4,4]] fwd,bwd
-- test for shape [[3,2,4,4]] fwd,bwd
-- test for shape [[2,2,7,7]] fwd,bwd
-- test for shape [[2,2,10,5]] fwd,bwd
-- test for shape [[2,2,10,10]] fwd,bwd
-- test for shape [[2,2,19,19]] fwd,bwd
-- test for shape [[2,2,20,20]] fwd,bwd
-- test for shape [[2,2,32,32]] fwd,bwd
-- test for shape [[64,2,64,64]] fwd,bwd
-- test for shape [[53,2,100,100]] fwd,bwd
- Running test case 3 for options {"bias":false,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":1,"channels_out":2,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":3,"pad":1,"stride":1}
-- test for shape [[1,1,2,2]] fwd,bwd
-- test for shape [[1,1,7,7]] fwd,bwd
-- test for shape [[1,1,8,8]] fwd,bwd
-- test for shape [[1,1,4,4]] fwd,bwd
-- test for shape [[3,1,4,4]] fwd,bwd
-- test for shape [[2,1,7,7]] fwd,bwd
-- test for shape [[2,1,10,5]] fwd,bwd
-- test for shape [[2,1,10,10]] fwd,bwd
-- test for shape [[2,1,19,19]] fwd,bwd
-- test for shape [[2,1,20,20]] fwd,bwd
-- test for shape [[2,1,32,32]] fwd,bwd
-- test for shape [[64,1,64,64]] fwd,bwd
-- test for shape [[53,1,100,100]] fwd,bwd
- Running test case 4 for options {"bias":false,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":3,"channels_out":8,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":3,"pad":1,"stride":1}
-- test for shape [[1,3,2,2]] fwd,bwd
-- test for shape [[1,3,7,7]] fwd,bwd
-- test for shape [[1,3,8,8]] fwd,bwd
-- test for shape [[1,3,4,4]] fwd,bwd
-- test for shape [[3,3,4,4]] fwd,bwd
-- test for shape [[2,3,7,7]] fwd,bwd
-- test for shape [[2,3,10,5]] fwd,bwd
-- test for shape [[2,3,10,10]] fwd,bwd
-- test for shape [[2,3,19,19]] fwd,bwd
-- test for shape [[2,3,20,20]] fwd,bwd
-- test for shape [[2,3,32,32]] fwd,bwd
-- test for shape [[64,3,64,64]] fwd,bwd
-- test for shape [[53,3,100,100]] fwd,bwd
- Running test case 5 for options {"bias":false,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":128,"channels_out":64,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":3,"pad":1,"stride":1}
-- test for shape [[1,128,2,2]] fwd,bwd
-- test for shape [[1,128,7,7]] fwd,bwd
-- test for shape [[1,128,8,8]] fwd,bwd
-- test for shape [[1,128,4,4]] fwd,bwd
-- test for shape [[3,128,4,4]] fwd,bwd
-- test for shape [[2,128,7,7]] fwd,bwd
-- test for shape [[2,128,10,5]] fwd,bwd
-- test for shape [[2,128,10,10]] fwd,bwd
-- test for shape [[2,128,19,19]] fwd,bwd
-- test for shape [[2,128,20,20]] fwd,bwd
-- test for shape [[2,128,32,32]] fwd,bwd
-- test for shape [[64,128,64,64]] fwd,bwd
-- test for shape [[53,128,100,100]] fwd,bwd
- Running test case 6 for options {"activation":"relu","bias":true,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":3,"channels_out":8,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":3,"pad":1,"stride":1}
-- test for shape [[1,3,2,2]] fwd,bwd
-- test for shape [[1,3,7,7]] fwd,bwd
-- test for shape [[1,3,8,8]] fwd,bwd
-- test for shape [[1,3,4,4]] fwd,bwd
-- test for shape [[3,3,4,4]] fwd,bwd
-- test for shape [[2,3,7,7]] fwd,bwd
-- test for shape [[2,3,10,5]] fwd,bwd
-- test for shape [[2,3,10,10]] fwd,bwd
-- test for shape [[2,3,19,19]] fwd,bwd
-- test for shape [[2,3,20,20]] fwd,bwd
-- test for shape [[2,3,32,32]] fwd,bwd
-- test for shape [[64,3,64,64]] fwd,bwd
-- test for shape [[53,3,100,100]] fwd,bwd
- Running test case 7 for options {"activation":"relu","bias":true,"bwd_data_algo":"winograd","bwd_filter_algo":"winograd","channels_in":3,"channels_out":8,"dilate":1,"fwd_algo":"winograd","groups":1,"kernel":3,"pad":1,"stride":1}
-- test for shape [[1,3,2,2]] fwd,bwd
-- test for shape [[1,3,7,7]] fwd,bwd
-- test for shape [[1,3,8,8]] fwd,bwd
-- test for shape [[1,3,4,4]] fwd,bwd
-- test for shape [[3,3,4,4]] fwd,bwd
-- test for shape [[2,3,7,7]] fwd,bwd
-- test for shape [[2,3,10,5]] fwd,bwd
-- test for shape [[2,3,10,10]] fwd,bwd
-- test for shape [[2,3,19,19]] fwd,bwd
-- test for shape [[2,3,20,20]] fwd,bwd
-- test for shape [[2,3,32,32]] fwd,bwd
-- test for shape [[64,3,64,64]] fwd,bwd
-- test for shape [[53,3,100,100]] fwd,bwdComparison failed for tensor data:0 at 95569 expecting -0.378173=-0.378173*1 got -0.253815 for esp=1e-05
Comparison failed for tensor data:0 at 95570 expecting -0.273047=-0.273047*1 got -0.366986 for esp=1e-05
Comparison failed for tensor data:0 at 95571 expecting 0.65315=0.65315*1 got 0.766756 for esp=1e-05
Comparison failed for tensor data:0 at 95669 expecting -0.393208=-0.393208*1 got -0.219796 for esp=1e-05
Comparison failed for tensor data:0 at 95670 expecting -0.223676=-0.223676*1 got -0.373241 for esp=1e-05
Comparison failed for tensor data:0 at 95671 expecting 0.318421=0.318421*1 got 0.451013 for esp=1e-05
Comparison failed for tensor data:0 at 95769 expecting -0.571868=-0.571868*1 got -0.484194 for esp=1e-05
Comparison failed for tensor data:0 at 95770 expecting 0.243095=0.243095*1 got 0.184242 for esp=1e-05
Comparison failed for tensor data:0 at 95771 expecting -0.897846=-0.897846*1 got -0.803248 for esp=1e-05
Comparison failed for tensor data:0 at 105569 expecting -0.419033=-0.419033*1 got -0.250495 for esp=1e-05
FAILED: Computations Failed
<end of output>
Test time = 35.25 sec
----------------------------------------------------------
Test Failed.
"test_test_case_conv2d_win" end time: Jan 18 00:43 CST
"test_test_case_conv2d_win" time elapsed: 00:00:35
----------------------------------------------------------
./dlprim_benchmark 0:0 ../docs/nets_for_benchmark/resnet18-b16.js
Using: AMD Radeon R9 200 Series (pitcairn, LLVM 15.0.6, DRM 3.49, 6.1.6-200.fc37.x86_64) on Clover
Inputs
- data: (16,3,224,224)
Outputs
- loss: (16,1000)
Step -5 549.446
Step -4 172.131
Step -3 173.069
Step -2 173.027
Step -1 172.077
Step 0 172.158
Step 1 172.775
Step 2 173.315
Step 3 171.667
Step 4 172.834
Step 5 172.461
Step 6 172.799
Step 7 172.202
Step 8 172.687
Step 9 172.011
Step 10 171.831
Step 11 171.646
Step 12 172.743
Step 13 171.402
Step 14 171.953
Step 15 173.605
Step 16 172.521
Step 17 172.836
Step 18 172.724
Step 19 171.633
Time per sample: 10.774 ms
TOT time per batch: 172.390 ms
./dlprim_benchmark -b 0:0 ../docs/nets_for_benchmark/resnet18-b16.js
Using: AMD Radeon R9 200 Series (pitcairn, LLVM 15.0.6, DRM 3.49, 6.1.6-200.fc37.x86_64) on Clover
Inputs
- data: (16,3,224,224)
Outputs
- loss: (16,1000)
Step -5 944.391 562.550 381.841
Step -4 568.292 186.458 381.834
Step -3 569.105 185.943 383.161
Step -2 567.985 185.340 382.645
Step -1 569.030 186.229 382.801
Step 0 567.343 185.491 381.852
Step 1 568.028 185.983 382.045
Step 2 566.410 185.105 381.304
Step 3 568.072 185.484 382.588
Step 4 567.650 184.827 382.823
Step 5 568.139 185.818 382.321
Step 6 568.514 186.141 382.374
Step 7 567.538 185.382 382.156
Step 8 569.284 185.928 383.355
Step 9 568.451 186.001 382.450
Step 10 568.275 185.067 383.208
Step 11 566.267 184.486 381.781
Step 12 568.327 185.258 383.069
Step 13 568.942 186.307 382.635
Step 14 568.047 186.173 381.874
Step 15 566.231 184.442 381.788
Step 16 566.303 185.724 380.578
Step 17 565.987 185.685 380.302
Step 18 567.147 186.259 380.887
Step 19 565.409 184.231 381.178
Time per sample: 35.470 ms
FWD time per batch: 185.490 ms
BWD time per batch: 382.028 ms
TOT time per batch: 567.518 ms
Looks like a nice result for an old GPU.
No it seems that from dlprimitives side it works more or less ok. There is some accuracy failure with Wingord that can be investigated further but in general seems to be working.
pytorch 2.0.0.dev20230116
I see you use 2.0.0. Can you check against 1.13? I just want to make sure it isn't related to latest development changes.
Looks like a nice result for an old GPU.
No it seems that from dlprimitives side it works more or less ok. There is some accuracy failure with Wingord that can be investigated further but in general seems to be working.
pytorch 2.0.0.dev20230116
I see you use 2.0.0. Can you check against 1.13? I just want to make sure it isn't related to latest development changes.
I just switch pytorch to ver 1.13.1 and rebuild it, but RuntimeError: Invalid Device #0
still there.
python mnist.py --device ocl:0
Using device: privateuseone:0
Traceback (most recent call last):
File "/home/ml/Projects/pytorch_dlprim/mnist.py", line 162, in <module>
main()
File "/home/ml/Projects/pytorch_dlprim/mnist.py", line 148, in main
model = Net().to(device)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 989, in to
return self._apply(convert)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 641, in _apply
module._apply(fn)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 664, in _apply
param_applied = fn(param)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 987, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: Invalid Device #0
little change, first line output was Using device: ocl:0
, it's Using device: privateuseone:0
now.
Sorry for delay can you please add to CLTensor.h few debug lines to see why device/platform wasn't loaded:
diff --git a/src/CLTensor.h b/src/CLTensor.h
index 8e01bc5..90e5dc3 100644
--- a/src/CLTensor.h
+++ b/src/CLTensor.h
@@ -159,7 +159,8 @@ namespace ptdlprim {
try {
cl::Platform::get(&platforms);
}
- catch(cl::Error &) {
+ catch(cl::Error &e) {
+ fprintf(stderr,"Failed to get platforms list %s\n",e.what());
return;
}
for(size_t i=0;i<platforms.size();i++) {
@@ -167,13 +168,15 @@ namespace ptdlprim {
try{
platforms[i].getDevices(CL_DEVICE_TYPE_ALL, &devices);
}
- catch(cl::Error &)
+ catch(cl::Error &e)
{
+ fprintf(stderr,"Failed to get device list for platform %d: %s\n",int(i),e.what());
continue;
}
for(size_t j=0;j<devices.size();j++) {
std::unique_ptr<DevData> d(new DevData());
data_.push_back(std::move(d));
+ fprintf(stderr,"Found platform/device %d:%d\n",int(i),int(j));
data_.back()->name = std::to_string(i) + ":" + std::to_string(j);
}
}
Sorry for delay can you please add to CLTensor.h few debug lines to see why device/platform wasn't loaded:
diff --git a/src/CLTensor.h b/src/CLTensor.h index 8e01bc5..90e5dc3 100644 --- a/src/CLTensor.h +++ b/src/CLTensor.h @@ -159,7 +159,8 @@ namespace ptdlprim { try { cl::Platform::get(&platforms); } - catch(cl::Error &) { + catch(cl::Error &e) { + fprintf(stderr,"Failed to get platforms list %s\n",e.what()); return; } for(size_t i=0;i<platforms.size();i++) { @@ -167,13 +168,15 @@ namespace ptdlprim { try{ platforms[i].getDevices(CL_DEVICE_TYPE_ALL, &devices); } - catch(cl::Error &) + catch(cl::Error &e) { + fprintf(stderr,"Failed to get device list for platform %d: %s\n",int(i),e.what()); continue; } for(size_t j=0;j<devices.size();j++) { std::unique_ptr<DevData> d(new DevData()); data_.push_back(std::move(d)); + fprintf(stderr,"Found platform/device %d:%d\n",int(i),int(j)); data_.back()->name = std::to_string(i) + ":" + std::to_string(j); } }
That's OK. we have some new output here
python mnist.py --device=ocl:0
Using device: privateuseone:0
Failed to get platforms list clGetPlatformIDs
Traceback (most recent call last):
File "/home/ml/Projects/pytorch_dlprim/mnist.py", line 162, in <module>
main()
File "/home/ml/Projects/pytorch_dlprim/mnist.py", line 148, in main
model = Net().to(device)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 989, in to
return self._apply(convert)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 641, in _apply
module._apply(fn)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 664, in _apply
param_applied = fn(param)
File "/home/ml/.conda/envs/pytorch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 987, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: Invalid Device #0
Ok from what I see that the query of platforms list fails...
Looks like kind of installation of OpenCL drivers/runtime issue.
Can you check what libOpenCL.so the build/libpt_ocl.so is linked to and which is linked to clinfo? Check if there are several libOpenCL.so libraries exist.
Also look into /etc/OpenCL/vendors
directory to see which platforms are installed.
Ok from what I see that the query of platforms list fails...
Looks like kind of installation of OpenCL drivers/runtime issue.
Can you check what libOpenCL.so the build/libpt_ocl.so is linked to and which is linked to clinfo? Check if there are several libOpenCL.so libraries exist.
Also look into
/etc/OpenCL/vendors
directory to see which platforms are installed.
strace clinfo 2>&1 |grep libOpenCL.so
openat(AT_FDCWD, "/lib64/libOpenCL.so.1", O_RDONLY|O_CLOEXEC) = 3
strace python mnist.py --device=ocl:0 2>&1 |grep libOpenCL.so
openat(AT_FDCWD, "/lib64/libOpenCL.so.1", O_RDONLY|O_CLOEXEC) = 4
ls -l -a /etc/OpenCL/vendors
total 8
drwxr-xr-x. 2 root root 41 Jan 17 00:11 .
drwxr-xr-x. 3 root root 21 Jul 22 2022 ..
-rw-r--r--. 1 root root 19 Jan 12 04:39 mesa.icd
-rw-r--r--. 1 root root 22 Jan 12 04:39 rusticl.icd
I just built a shared library with cl::Platform::get(&platforms) and loaded it with CDLL in python. It works fine in system Python, but not in conda Python.
I think the error is caused by conda environment configuration, not related to pytorch_dlprim.
conda install -c conda-forge ocl-icd-system
conda install -c conda-forge pyopencl
It works fine now.
RuntimeError: Invalid Device in there, I don't know why.
python mnist.py --device=ocl:0
clinfo
conda list