search

ROCm / rocAL

The AMD rocAL is designed to efficiently decode and process images and videos from a variety of storage formats and modify them through a processing graph programmable by the user.
https://rocm.docs.amd.com/projects/rocAL/en/develop/
MIT License
12 stars 14 forks source link

ROCm 6.2.1 - RC2:cherry-pick #196

Closed kiritigowda closed 3 months ago

kiritigowda commented 3 months ago

Version 2.0.0 Updates

kiritigowda commented 3 months ago

CMakeList Log

cmake ../rocAL/
-- The CXX compiler identification is GNU 9.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- rocAL Version -- 2.0.0
-- rocAL Install Path -- /opt/rocm
-- rocAL Backend set to -- HIP
-- rocAL Build Type -- Release
-- Using HALF -- 
    Includes:/opt/rocm/include
-- rocAL Built with float16 Support -- -DAMD_FP16_SUPPORT definition added
-- rocAL Developer Options
--      -D GPU_SUPPORT=ON [Turn ON/OFF GPU support (default:ON)]
--      -D BACKEND=HIP [Select rocAL Backend [options:CPU/OPENCL/HIP](default:HIP)]
--      -D BUILD_PYPACKAGE=ON [rocAL Python Package(default:ON)]
--      -D PYTHON_VERSION_SUGGESTED=OFF [User provided python version to use for rocAL Python Bindings(default:System Version)]
--      -D BUILD_WITH_AMD_ADVANCE=OFF [Turn ON/OFF Build for AMD advanced GPUs(default:OFF)]
-- Using AMD RPP -- 
    Libraries:/opt/rocm/lib/librpp.so 
    Includes:/opt/rocm/include/rpp
-- Found VX RPP Version: 3.0.1
-- Using MIVisionX -- 
    Libraries:/opt/rocm/lib/libopenvx.so 
    Includes:/opt/rocm/include/mivisionx
-- Using Turbo JPEG -- 
    Libraries:/lib/libturbojpeg.so 
    Includes:/usr/include
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 
-- Checking for module 'libavcodec'
--   Found libavcodec, version 58.54.100
-- Checking for module 'libavformat'
--   Found libavformat, version 58.29.100
-- Checking for module 'libavutil'
--   Found libavutil, version 56.31.100
-- Using FFMPEG -- 
    Libraries:/usr/lib/x86_64-linux-gnu/libavcodec.so;/usr/lib/x86_64-linux-gnu/libavformat.so;/usr/lib/x86_64-linux-gnu/libavutil.so;/usr/lib/x86_64-linux-gnu/libswscale.so 
    Includes:/usr/include/x86_64-linux-gnu
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Using LMDB -- 
    Libraries:/usr/lib/x86_64-linux-gnu/liblmdb.so 
    Includes:/usr/include
-- Performing Test STD_FILESYSTEM_PRESENT
-- Performing Test STD_FILESYSTEM_PRESENT - Success
-- Using HALF -- 
    Includes:/opt/rocm/include
-- NOTE: FindSndFile failed to find -- SndFile
-- rocal -- AMDGPU_TARGETS: gfx908;gfx90a;gfx940;gfx941;gfx942;gfx1030;gfx1031;gfx1032;gfx1100;gfx1101;gfx1102
-- rocAL -- Using HIP -- Path:/opt/rocm Version:6.2.41133   Compiler:clang
-- ROCm Augmentation Library -- rocAL dependencies found
-- rocAL built with HIP Backend
-- rocAL built with ENABLE_SIMD
-- rocAL built with FFmpeg Video Decode Functionality
-- rocAL built with std Filesystem
-- NOTE: rocAL built without SndFile - Audio Functionalities will not be supported
-- NOTE: rocAL built without Audio support - Audio Functionalities will not be enabled
-- rocAL -- CMAKE_CXX_FLAGS: -O3 -DNDEBUG -fPIC -fopenmp -Wall -mavx2 -mfma -mf16c -Wno-deprecated-declarations -std=gnu++17
-- rocAL -- Link Libraries: /opt/rocm/lib/librpp.so;/opt/rocm/lib/libopenvx.so;/opt/rocm/lib/libvx_rpp.so;OpenMP::OpenMP_CXX;Threads::Threads;/lib/libturbojpeg.so;/usr/local/lib/libprotobuf.so;-lpthread;/usr/lib/x86_64-linux-gnu/liblmdb.so;$<TARGET_OBJECTS:rocAL_hip>;opencv_calib3d;opencv_core;opencv_dnn;opencv_features2d;opencv_flann;opencv_gapi;opencv_highgui;opencv_imgcodecs;opencv_imgproc;opencv_ml;opencv_objdetect;opencv_photo;opencv_stitching;opencv_video;opencv_videoio;/usr/lib/x86_64-linux-gnu/libavcodec.so;/usr/lib/x86_64-linux-gnu/libavformat.so;/usr/lib/x86_64-linux-gnu/libavutil.so;/usr/lib/x86_64-linux-gnu/libswscale.so
-- Using AMD RPP -- 
    Libraries:/opt/rocm/lib/librpp.so 
    Includes:/opt/rocm/include/rpp
-- Found VX RPP Version: 3.0.1
-- Using MIVisionX -- 
    Libraries:/opt/rocm/lib/libopenvx.so 
    Includes:/opt/rocm/include/mivisionx
-- Using Turbo JPEG -- 
    Libraries:/lib/libturbojpeg.so 
    Includes:/usr/include
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Using HALF -- 
    Includes:/opt/rocm/include
-- rocAL PyBind Backend set to -- HIP
-- rocAL PyBind -- Using HIP -- Path:/opt/rocm  Version:6.2.41133   Compiler:clang
-- NOTE: rocAL PyBind default installation path -- /usr/lib/python3.8 
-- NOTE: rocAL pybind requires Python3, Found --  /usr/bin/python3.8 
-- rocal_pybind -- Link Libraries: /opt/rocm/lib/librpp.so;/opt/rocm/lib/libopenvx.so;/opt/rocm/lib/libvx_rpp.so;/lib/libturbojpeg.so;rocal
-- rocal_pybind -- CMAKE_CXX_FLAGS: -O3 -DNDEBUG -fPIC
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/__init__.py to rocAL_pybind/amd/__init__.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/__init__.py to rocAL_pybind/amd/rocal/__init__.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/decoders.py to rocAL_pybind/amd/rocal/decoders.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/fn.py to rocAL_pybind/amd/rocal/fn.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/noise.py to rocAL_pybind/amd/rocal/noise.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/pipeline.py to rocAL_pybind/amd/rocal/pipeline.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/__init__.py to rocAL_pybind/amd/rocal/plugin/__init__.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/generic.py to rocAL_pybind/amd/rocal/plugin/generic.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/pytorch.py to rocAL_pybind/amd/rocal/plugin/pytorch.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/tf.py to rocAL_pybind/amd/rocal/plugin/tf.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/random.py to rocAL_pybind/amd/rocal/random.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/readers.py to rocAL_pybind/amd/rocal/readers.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/reductions.py to rocAL_pybind/amd/rocal/reductions.py
-- Copying /home/kiriti/develop/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/types.py to rocAL_pybind/amd/rocal/types.py
-- Detected rocAL Python wheel filename amd_rocal-2.0.0-cp38-cp38-linux_x86_64.whl
-- Radeon Augmentation Library Python Binding - rocal_pybind module added 
-- The C compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- rocal-test INFO:ROCM_PATH Set -- /opt/rocm
-- rocal-test: Using rocAL -- 
    Libraries:/opt/rocm/lib/librocal.so 
    Includes:/opt/rocm/include/rocal
-- AMD ROCm rocAL RunTime Package --  rpp, mivisionx, rocdecode
-- AMD ROCm rocAL Dev Package --  rocal, rpp-dev, mivisionx-dev, liblmdb-dev, libprotobuf-dev, rocdecode-dev
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kiriti/develop/rocal-kiriti/build
kiritigowda commented 3 months ago

make and make pypackage

make -j8
[  1%] Built target rocAL_hip
[ 97%] Built target rocal
[100%] Built target rocal_pybind
kiriti@simon:~/develop/rocal-kiriti/build$ sudo cmake --build . --target PyPackageInstall
[sudo] password for kiriti: 
[  1%] Built target rocAL_hip
[ 97%] Built target rocal
[ 99%] Built target rocal_pybind
Scanning dependencies of target PyPackageBuild
[ 99%] Building python wheel package
Processing /home/kiriti/develop/rocal-kiriti/build
Building wheels for collected packages: amd-rocal
  Building wheel for amd-rocal (setup.py) ... done
  Created wheel for amd-rocal: filename=amd_rocal-2.0.0-cp38-cp38-linux_x86_64.whl size=4158943 sha256=3c988ce7f6c0d75d5a5e6511afda86695545596d680e62660527285224a145b2
  Stored in directory: /tmp/pip-ephem-wheel-cache-4pnp22f8/wheels/16/2c/fd/4e70e92f01feb24bb1164fca95e50c4536fcdb34eed0d6c371
Successfully built amd-rocal
[ 99%] Built target PyPackageBuild
Scanning dependencies of target PyPackageInstall
[100%] Installing wheel file
Processing ./amd_rocal-2.0.0-cp38-cp38-linux_x86_64.whl
Installing collected packages: amd-rocal
  Attempting uninstall: amd-rocal
    Found existing installation: amd-rocal 2.0.0
    Uninstalling amd-rocal-2.0.0:
      Successfully uninstalled amd-rocal-2.0.0
Successfully installed amd-rocal-2.0.0
[100%] Built target PyPackageInstall
kiritigowda commented 3 months ago

CTEST -- @SundarRajan98 can you look the the failed tests?

ctest
Test project /home/kiriti/develop/rocal-kiriti/build
      Start  1: basic_test_cpu
 1/14 Test  #1: basic_test_cpu .....................   Passed    4.23 sec
      Start  2: basic_test_gpu
 2/14 Test  #2: basic_test_gpu .....................   Passed    0.46 sec
      Start  3: basic_test_gray
 3/14 Test  #3: basic_test_gray ....................   Passed    0.74 sec
      Start  4: basic_test_rgb
 4/14 Test  #4: basic_test_rgb .....................   Passed    0.84 sec
      Start  5: dataloader_multithread_cpu
 5/14 Test  #5: dataloader_multithread_cpu .........   Passed    5.24 sec
      Start  6: dataloader_multithread_gpu
 6/14 Test  #6: dataloader_multithread_gpu .........***Exception: SegFault  1.05 sec
      Start  7: performance_tests_cpu
 7/14 Test  #7: performance_tests_cpu ..............   Passed    4.44 sec
      Start  8: performance_tests_gpu
 8/14 Test  #8: performance_tests_gpu ..............   Passed    0.84 sec
      Start  9: performance_tests_with_depth_cpu
 9/14 Test  #9: performance_tests_with_depth_cpu ...   Passed    5.40 sec
      Start 10: performance_tests_with_depth_gpu
10/14 Test #10: performance_tests_with_depth_gpu ...   Passed    1.77 sec
      Start 11: unit_tests_cpu
11/14 Test #11: unit_tests_cpu .....................   Passed   24.58 sec
      Start 12: unit_tests_gpu
12/14 Test #12: unit_tests_gpu .....................   Passed   19.25 sec
      Start 13: unit_tests_gray
13/14 Test #13: unit_tests_gray ....................   Passed    7.62 sec
      Start 14: video_tests
14/14 Test #14: video_tests ........................***Failed  101.23 sec

86% tests passed, 2 tests failed out of 14

Total Test time (real) = 177.71 sec

The following tests FAILED:
      6 - dataloader_multithread_gpu (SEGFAULT)
     14 - video_tests (Failed)
Errors while running CTest
kiritigowda commented 3 months ago

Failure 1

ctest -R dataloader_multithread_gpu -VV
UpdateCTestConfiguration  from :/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
Parse Config file:/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
Parse Config file:/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
Test project /home/kiriti/develop/rocal-kiriti/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 6
    Start 6: dataloader_multithread_gpu

6: Test command: /home/kiriti/develop/rocal-kiriti/build/tests/cpp_api/dataloader_multithread/dataloader_multithread "/opt/rocm/share/rocal/test/data/images/AMD-tinyDataSet" "1"
6: Test timeout computed to be: 1500
6: Number of GPUs: 1
6: Running on GPU: 0
6: shard_id: 0
6: Running on GPU: 0
6: shard_id: 1
6: OK: OpenVX using GPU device - 0: AMD Radeon PRO W6800 [gfx1030] with 60 CUs on PCI bus 0b:00.0
6: OK: loaded 125 kernels from libvx_rpp.so
6: OK: OpenVX using GPU device - 0: AMD Radeon PRO W6800 [gfx1030] with 60 CUs on PCI bus 0b:00.0
6: OK: loaded 125 kernels from libvx_rpp.so
6: Remaining images 128
6: Augmented copies count 1
6: output width 224 output height 3584 color planes 3 n 1
6: Remaining images 128
6: Augmented copies count 1
6: output width 224 output height 3584 color planes 3 n 1
6: free(): invalid size
1/1 Test #6: dataloader_multithread_gpu .......Child aborted***Exception:   1.32 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   1.32 sec

The following tests FAILED:
      6 - dataloader_multithread_gpu (Child aborted)
Errors while running CTest
kiritigowda commented 3 months ago

Failure 2

ctest -R video_tests -VV
UpdateCTestConfiguration  from :/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
Parse Config file:/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
Parse Config file:/home/kiriti/develop/rocal-kiriti/build/DartConfiguration.tcl
Test project /home/kiriti/develop/rocal-kiriti/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 14
    Start 14: video_tests

14: Test command: /usr/bin/ctest "--build-and-test" "/home/kiriti/develop/rocal-kiriti/rocAL/tests/cpp_api/video_tests" "/home/kiriti/develop/rocal-kiriti/build/tests/cpp_api/video_tests" "--build-generator" "Unix Makefiles" "--test-command" "video_tests" "/opt/rocm/share/rocal/test/data/videos/AMD_driving_virtual_20.mp4"
14: Test timeout computed to be: 1500
14: Internal cmake changing into directory: /home/kiriti/develop/rocal-kiriti/build/tests/cpp_api/video_tests
14: ======== CMake output     ======
14: -- video_tests INFO:ROCM_PATH Set -- /opt/rocm
14: -- OpenCV Found -- Version-4.6.X Supported
14: Configuring done
14: Generating done
14: Build files have been written to: /home/kiriti/develop/rocal-kiriti/build/tests/cpp_api/video_tests
14: ======== End CMake output ======
14: Change Dir: /home/kiriti/develop/rocal-kiriti/build/tests/cpp_api/video_tests
14: 
14: Run Clean Command:/usr/bin/make clean
14: 
14: Run Build Command(s):/usr/bin/make && [ 50%] Building CXX object CMakeFiles/video_tests.dir/video_tests.cpp.o
14: [100%] Linking CXX executable video_tests
14: [100%] Built target video_tests
14: 
14: Running test command: "/home/kiriti/develop/rocal-kiriti/build/tests/cpp_api/video_tests/video_tests" "/opt/rocm/share/rocal/test/data/videos/AMD_driving_virtual_20.mp4"
14: Test command failed: /home/kiriti/develop/rocal-kiriti/build/tests/cpp_api/video_tests/video_tests
14: Batch size : 1
14: Sequence length : 1
14: Frame step : 1
14: Frame stride : 1
14: OK: OpenVX using GPU device - 0: AMD Radeon PRO W6800 [gfx1030] with 60 CUs on PCI bus 0b:00.0
14: OK: loaded 125 kernels from libvx_rpp.so
14: 
14: VIDEO READER
14: 
14: Remaining images 601
14: Augmented copies count 1
14: output width 2048 output height 1024 color planes 3
14: Load     time 1621
14: Decode   time 89600509
14: Process  time 952952
14: Transfer time 89540744
14: Processed 601 images/frames
14: Total Elapsed Time 89 sec 615579 us 
14: 
14: *** Exception executing: Segmentation fault
1/1 Test #14: video_tests ......................***Failed   94.50 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =  94.50 sec

The following tests FAILED:
     14 - video_tests (Failed)
Errors while running CTest
SundarRajan28 commented 3 months ago

@kiritigowda Tested this PR on both bare metal and dockers and the ctests are passing Bare metal (rocm 6.3): Installed latest rpp-dev and mivisionx-dev packages in bare metal and built this PR. Ctests are passing

root@x1000c4s5b1n0:/home/svaithiy/sundar/unittests_qa/rocAL/build# ctest
Test project /home/svaithiy/sundar/unittests_qa/rocAL/build
      Start  1: basic_test_cpu
 1/14 Test  #1: basic_test_cpu .....................   Passed    3.19 sec
      Start  2: basic_test_gpu
 2/14 Test  #2: basic_test_gpu .....................   Passed    0.42 sec
      Start  3: basic_test_gray
 3/14 Test  #3: basic_test_gray ....................   Passed    0.55 sec
      Start  4: basic_test_rgb
 4/14 Test  #4: basic_test_rgb .....................   Passed    0.73 sec
      Start  5: dataloader_multithread_cpu
 5/14 Test  #5: dataloader_multithread_cpu .........   Passed    4.38 sec
      Start  6: dataloader_multithread_gpu
 6/14 Test  #6: dataloader_multithread_gpu .........   Passed    1.88 sec
      Start  7: performance_tests_cpu
 7/14 Test  #7: performance_tests_cpu ..............   Passed    3.36 sec
      Start  8: performance_tests_gpu
 8/14 Test  #8: performance_tests_gpu ..............   Passed    0.60 sec
      Start  9: performance_tests_with_depth_cpu
 9/14 Test  #9: performance_tests_with_depth_cpu ...   Passed    4.51 sec
      Start 10: performance_tests_with_depth_gpu
10/14 Test #10: performance_tests_with_depth_gpu ...   Passed    1.60 sec
      Start 11: unit_tests_cpu
11/14 Test #11: unit_tests_cpu .....................   Passed   18.36 sec
      Start 12: unit_tests_gpu
12/14 Test #12: unit_tests_gpu .....................   Passed   15.25 sec
      Start 13: unit_tests_gray
13/14 Test #13: unit_tests_gray ....................   Passed    6.11 sec
      Start 14: video_tests
14/14 Test #14: video_tests ........................   Passed   88.27 sec

100% tests passed, 0 tests failed out of 14

Total Test time (real) = 149.24 sec

Docker (rocm 6.1): Installed ToT rpp and MIVISIONX from source and built this PR. Ctests are passing

root@x1001c1s1b0n0:/data/unittests_qa/rocAL/build# ctest                                                                                                                                                                                         
Test project /data/unittests_qa/rocAL/build                                                 
      Start  1: basic_test_cpu                                                                                                                                                           
 1/14 Test  #1: basic_test_cpu .....................   Passed    2.96 sec                                                                                                                
      Start  2: basic_test_gpu                                                                                                                                                           
 2/14 Test  #2: basic_test_gpu .....................   Passed    0.38 sec                                                                                                                
      Start  3: basic_test_gray                                                                                                                                                          
 3/14 Test  #3: basic_test_gray ....................   Passed    0.46 sec                                                                                                                
      Start  4: basic_test_rgb                                                              
 4/14 Test  #4: basic_test_rgb .....................   Passed    0.61 sec                                                                                                                
      Start  5: dataloader_multithread_cpu                                                                                                                                               
 5/14 Test  #5: dataloader_multithread_cpu .........   Passed    4.11 sec                                                                                                                
      Start  6: dataloader_multithread_gpu                                                  
 6/14 Test  #6: dataloader_multithread_gpu .........   Passed    1.70 sec                                                                                                                
      Start  7: performance_tests_cpu                                                       
 7/14 Test  #7: performance_tests_cpu ..............   Passed    3.15 sec                                                                                                                
      Start  8: performance_tests_gpu                                                                                                                                                    
 8/14 Test  #8: performance_tests_gpu ..............   Passed    0.53 sec                                                                                                                
      Start  9: performance_tests_with_depth_cpu                                                                                                                                         ug-24
 9/14 Test  #9: performance_tests_with_depth_cpu ...   Passed    4.25 sec                                                                                                                
      Start 10: performance_tests_with_depth_gpu                                            
10/14 Test #10: performance_tests_with_depth_gpu ...   Passed    1.53 sec                                                                                                                
      Start 11: unit_tests_cpu                                                              
11/14 Test #11: unit_tests_cpu .....................   Passed   18.25 sec                                                                                                                
      Start 12: unit_tests_gpu                                                              
12/14 Test #12: unit_tests_gpu .....................   Passed   15.13 sec                                                                                                                
      Start 13: unit_tests_gray                                                                                                                                                          
13/14 Test #13: unit_tests_gray ....................   Passed    6.04 sec                                                                                                                
      Start 14: video_tests                                                                 
14/14 Test #14: video_tests ........................   Passed   87.45 sec                                                                                                                

100% tests passed, 0 tests failed out of 14                                                                                                                                              

Total Test time (real) = 146.58 sec

Please let me know if I need to test it on a specific rocm version.

kiritigowda commented 3 months ago

@SundarRajan28 thanks for verifying. Need to test on 6.2.0 public branch for both TOT develop and this PR. The PR will replace 6.2.0 release.

rrawther commented 3 months ago

@SundarRajan28 : Are we good to merge this PR. I am assuming you have tested all our training benchmarks on this?

SundarRajan28 commented 3 months ago

@rrawther @kiritigowda I have tested all the rocAL trainings on this PR with latest rocm 6.2 docker - confirmed convergence and performance for all trainings similar to ToT develop. I have tested the C++ and python unit tests with this PR and they are passing as expected