search

lamikr / rocm_sdk_builder

Other
113 stars 8 forks source link

Collection of pytorch gpu benchmark results #63

Open lamikr opened 3 weeks ago

lamikr commented 3 weeks ago

Extensive GPU benchmarks with AMD gpus can now be run by following steps after building the rocm sdk. This version has now been synced with the upstream version which has fixed the pytorch 2.0 support in another way I had done earlier and it runs all the tests without running exceptions.

git clone https://github.com/lamikr/pytorch-gpu-benchmark
cd pytorch-gpu-benchmark
source /opt/rocm_sdk_611/bin/env_rocm.sh
./test.sh

It would be nice to collect results from different computers and create some comparison graphs. On my AMD RX 6800 test execution was about 50 minutes and results were saved to result-folder to following 8 files.

'AMD Radeon RX 6800_1_gpus__double_model_inference_benchmark.csv'  'AMD Radeon RX 6800_1_gpus__half_model_inference_benchmark.csv'
'AMD Radeon RX 6800_1_gpus__double_model_train_benchmark.csv'      'AMD Radeon RX 6800_1_gpus__half_model_train_benchmark.csv'
'AMD Radeon RX 6800_1_gpus__float_model_inference_benchmark.csv'    config.json
'AMD Radeon RX 6800_1_gpus__float_model_train_benchmark.csv'        system_info.txt

I have stored those from my benchmark run now to results/AMD_Radeon_RX_6800 folder of gpu benchmark.

So if you have done the tests, can you send them as a pull requests. At the moment the plot.ipynb code which should read the CSV files and generate pictures seems to be broken so that needs to be fixed...

eitch commented 4 days ago

Hi @lamikr I've added my tests: https://github.com/ryujaehun/pytorch-gpu-benchmark/pull/31

Should i send the PR to your fork?

eitch commented 4 days ago

I now also sent the PR to your fork. I sure hope someone can fix the plotting.

lamikr commented 2 days ago

Thanks eitch, I noticed your 7900 xtx benchmarks today and merged the results to that repo. I am not sure is the upstream of that benchmark anymore active.

Another tests I run quite often to check is vit example from this repo https://github.com/BrianPulfer/PapersReimplementations.git (dir src/cv/vit) and docs in https://medium.com/@brianpulfer/vision-transformers-from-scratch-pytorch-a-step-by-step-guide-96c3313c2e0c

Then this one should contain all kind of usefull things

https://github.com/ROCm/ROCmValidationSuite/blob/master/docs/ug1main.md

lamikr commented 2 days ago

I have not integrated the tensorflow back, it's debugging is so time consuming if something goes wrong as it always like to trigger rebuild of everything without allowing to just fix a thing and continue.

But part of the tensorflow is a tool called tensorboard which is pretty nice. I just tested to install it with "pip install tensorboard" and it seemed to work ok without not messing the python dependencies. So that's one alternative and maybe at some point the tensorflow could be added back to rocm sdk builder.