search

ROCm / rocAL

The AMD rocAL is designed to efficiently decode and process images and videos from a variety of storage formats and modify them through a processing graph programmable by the user.
https://rocm.docs.amd.com/projects/rocAL/en/master/index.html
MIT License
10 stars 13 forks source link

rocAL - Setup updates #135

Closed kiritigowda closed 2 months ago

kiritigowda commented 2 months ago
LakshmiKumar23 commented 2 months ago

@kiritigowda Can you add this to support audio PRs coming in?

sudo apt-get install libsndfile1-dev
LakshmiKumar23 commented 2 months ago

@kiritigowda CI fails for this PR. Please take a look:

ERROR_CHECK failed with status:25600

+ mkdir -p build/release

+ cd build/release

+ cmake -DBACKEND=HIP -DCMAKE_BUILD_TYPE=Release ../..

-- The CXX compiler identification is GNU 11.4.0

-- Detecting CXX compiler ABI info

-- Detecting CXX compiler ABI info - done

-- Check for working CXX compiler: /usr/bin/c++ - skipped

-- Detecting CXX compile features

-- Detecting CXX compile features - done

-- rocAL Version -- 2.0.0

-- rocAL Install Path -- /opt/rocm

-- rocAL Backend set to -- HIP

-- rocAL Build Type -- Release

-- NOTE: FindHALF failed to find -- half.hpp

-- rocAL Built with float16 Support OFF

-- ROCm Version -- 6.2.40092

-- rocAL Developer Options

--      -D GPU_SUPPORT=ON [Turn ON/OFF GPU support (default:ON)]

--      -D BACKEND=HIP [Select rocAL Backend [options:CPU/OPENCL/HIP](default:HIP)]

--      -D BUILD_PYPACKAGE=ON [rocAL Python Package(default:ON)]

--      -D BUILD_WITH_AMD_ADVANCE=ON [rocAL support for advanced GPU(default:OFF)]

--      -D PYTHON_VERSION_SUGGESTED=OFF [User provided python version to use for rocAL Python Bindings(default:System Version)]

-- NOTE: FindAMDRPP failed to find -- rpp

-- NOTE: FindMIVisionX failed to find -- openvx

-- Using Turbo JPEG -- 

    Libraries:/lib/libturbojpeg.so 

    Includes:/usr/include

-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2") 

-- Checking for module 'libavcodec'

--   Found libavcodec, version 58.134.100

-- Checking for module 'libavformat'

--   Found libavformat, version 58.76.100

-- Checking for module 'libavutil'

--   Found libavutil, version 56.70.100

-- Using FFMPEG -- 

    Libraries:/usr/local/lib/libavcodec.so;/usr/local/lib/libavformat.so;/usr/local/lib/libavutil.so;/usr/local/lib/libswscale.so 

    Includes:/usr/local/include

-- Using LMDB -- 

    Libraries:/usr/lib/x86_64-linux-gnu/liblmdb.so 

    Includes:/usr/include

-- Performing Test STD_FILESYSTEM_PRESENT

-- Performing Test STD_FILESYSTEM_PRESENT - Success

-- NOTE: FindHALF failed to find -- half.hpp

-- rocAL -- Using HIP -- Path:/opt/rocm Version:6.2.40092   Compiler:clang

-- NOTE: rocAL library requires AMD's rpp library, Not Found 

-- NOTE: rocAL library requires AMD's MIVisionX library, Not Found 

-- NOTE: rocAL library requires RapidJSON, Not Found 

-- NOTE: rocAL library requires HALF, Not Found 

CMake Error at rocAL/CMakeLists.txt:328 (message):

  -- rocAL dependencies not satisfied

-- Configuring incomplete, errors occurred!

See also "/var/jenkins_home/workspace/main_precheckin_rocAL_PR-135/EYgH3HVzL/rocal/build/release/CMakeFiles/CMakeOutput.log".

++ nproc

+ make -j16

make: *** No targets specified and no makefile found.  Stop.

+ sudo cmake --build . --target PyPackageInstall

gmake: Makefile: No such file or directory

gmake: *** No rule to make target 'Makefile'.  Stop.

+ sudo make install

make: *** No rule to make target 'install'.  Stop.

+ ldd -v /opt/rocm/lib/librocal.so

ldd: /opt/rocm/lib/librocal.so: No such file or directory

script returned exit code 1
kiritigowda commented 2 months ago

@LakshmiKumar23 -- this PR will make the Setup install only packages for some of our deps and the CI does not leet us install software. We need to add deps into test dockers. Need to test on separate docker

kiritigowda commented 2 months ago

@LakshmiKumar23 can you review this PR?

LakshmiKumar23 commented 2 months ago

@LakshmiKumar23 can you review this PR?

Already added comments before. Please add sndfile install

LakshmiKumar23 commented 2 months ago

@kiritigowda Fails with without sudo on bare-metal. We should mention the script needs sudo acccess. Also maybe print the error name instead of number. 25600 does not help much

E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
E: Unable to lock directory /var/lib/apt/lists/
W: Problem unlinking the file /var/cache/apt/pkgcache.bin - RemoveCaches (13: Permission denied)
W: Problem unlinking the file /var/cache/apt/srcpkgcache.bin - RemoveCaches (13: Permission denied)
ERROR_CHECK failed with status:25600
kiritigowda commented 2 months ago 
rocAL Dependencies Installed with rocAL-setup.py V-2.0.0

root@simon:~# mkdir build
root@simon:~# cd build/
root@simon:~/build# cmake /root/hostDrive/rocal-kiriti/rocAL/
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- rocAL Version -- 2.0.0
-- rocAL Install Path -- /opt/rocm
-- rocAL Backend set to -- HIP
-- rocAL Build Type -- Release
-- Using HALF -- 
    Includes:/opt/rocm/include
-- rocAL Built with float16 Support -- -DAMD_FP16_SUPPORT definition added
-- ROCm Version -- 6.2.40092
-- rocAL Developer Options
--      -D GPU_SUPPORT=ON [Turn ON/OFF GPU support (default:ON)]
--      -D BACKEND=HIP [Select rocAL Backend [options:CPU/OPENCL/HIP](default:HIP)]
--      -D BUILD_PYPACKAGE=ON [rocAL Python Package(default:ON)]
--      -D BUILD_WITH_AMD_ADVANCE=ON [rocAL support for advanced GPU(default:OFF)]
--      -D PYTHON_VERSION_SUGGESTED=OFF [User provided python version to use for rocAL Python Bindings(default:System Version)]
-- Using AMD RPP -- 
    Libraries:/opt/rocm/lib/librpp.so 
    Includes:/opt/rocm/include/rpp
-- Using MIVisionX -- 
    Libraries:/opt/rocm/lib/libopenvx.so 
    Includes:/opt/rocm/include/mivisionx
-- Using Turbo JPEG -- 
    Libraries:/lib/libturbojpeg.so 
    Includes:/usr/include
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'libavcodec'
--   Found libavcodec, version 58.134.100
-- Checking for module 'libavformat'
--   Found libavformat, version 58.76.100
-- Checking for module 'libavutil'
--   Found libavutil, version 56.70.100
-- Using FFMPEG -- 
    Libraries:/usr/local/lib/libavcodec.so;/usr/local/lib/libavformat.so;/usr/local/lib/libavutil.so;/usr/local/lib/libswscale.so 
    Includes:/usr/local/include
-- Using LMDB -- 
    Libraries:/usr/lib/x86_64-linux-gnu/liblmdb.so 
    Includes:/usr/include
-- Performing Test STD_FILESYSTEM_PRESENT
-- Performing Test STD_FILESYSTEM_PRESENT - Success
-- Using HALF -- 
    Includes:/opt/rocm/include
-- rocAL -- Using HIP -- Path:/opt/rocm Version:6.2.40092   Compiler:clang
-- ROCm Augmentation Library -- rocAL dependencies found
-- rocAL built with HIP Backend
-- rocAL built with ENABLE_SIMD
-- rocAL built with FFmpeg Video Decode Functionality
-- rocAL built with std Filesystem
-- rocAL -- CMAKE_CXX_FLAGS: -O3 -DNDEBUG -fPIC -fopenmp -Wall -mavx2 -mfma -mf16c -Wno-deprecated-declarations -std=gnu++17
-- rocAL -- Link Libraries: /opt/rocm/lib/librpp.so;/opt/rocm/lib/libopenvx.so;/opt/rocm/lib/libvx_rpp.so;OpenMP::OpenMP_CXX;Threads::Threads;/lib/libturbojpeg.so;/usr/lib/x86_64-linux-gnu/libprotobuf.so;/usr/lib/x86_64-linux-gnu/liblmdb.so;$<TARGET_OBJECTS:rocAL_hip>;opencv_calib3d;opencv_core;opencv_dnn;opencv_features2d;opencv_flann;opencv_gapi;opencv_highgui;opencv_imgcodecs;opencv_imgproc;opencv_ml;opencv_objdetect;opencv_photo;opencv_stitching;opencv_video;opencv_videoio;/usr/local/lib/libavcodec.so;/usr/local/lib/libavformat.so;/usr/local/lib/libavutil.so;/usr/local/lib/libswscale.so
-- Using AMD RPP -- 
    Libraries:/opt/rocm/lib/librpp.so 
    Includes:/opt/rocm/include/rpp
-- Using MIVisionX -- 
    Libraries:/opt/rocm/lib/libopenvx.so 
    Includes:/opt/rocm/include/mivisionx
-- Using Turbo JPEG -- 
    Libraries:/lib/libturbojpeg.so 
    Includes:/usr/include
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Using HALF -- 
    Includes:/opt/rocm/include
-- rocAL PyBind Backend set to -- HIP
-- rocAL PyBind -- Using HIP -- Path:/opt/rocm  Version:6.2.40092   Compiler:clang
-- NOTE: rocAL PyBind default installation path -- /usr/lib/python3.10 
-- NOTE: rocAL pybind requires Python3, Found --  /usr/bin/python3.10 
-- rocal_pybind -- Link Libraries: /opt/rocm/lib/librpp.so;/opt/rocm/lib/libopenvx.so;/opt/rocm/lib/libvx_rpp.so;/lib/libturbojpeg.so;rocal
-- rocal_pybind -- CMAKE_CXX_FLAGS: -O3 -DNDEBUG -fPIC
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/__init__.py to rocAL_pybind/amd/__init__.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/__init__.py to rocAL_pybind/amd/rocal/__init__.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/decoders.py to rocAL_pybind/amd/rocal/decoders.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/fn.py to rocAL_pybind/amd/rocal/fn.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/noise.py to rocAL_pybind/amd/rocal/noise.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/pipeline.py to rocAL_pybind/amd/rocal/pipeline.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/__init__.py to rocAL_pybind/amd/rocal/plugin/__init__.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/generic.py to rocAL_pybind/amd/rocal/plugin/generic.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/pytorch.py to rocAL_pybind/amd/rocal/plugin/pytorch.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/plugin/tf.py to rocAL_pybind/amd/rocal/plugin/tf.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/random.py to rocAL_pybind/amd/rocal/random.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/readers.py to rocAL_pybind/amd/rocal/readers.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/reductions.py to rocAL_pybind/amd/rocal/reductions.py
-- Copying /root/hostDrive/rocal-kiriti/rocAL/rocAL_pybind/amd/rocal/types.py to rocAL_pybind/amd/rocal/types.py
-- Detected rocAL Python wheel filename amd_rocal-2.0.0-cp310-cp310-linux_x86_64.whl
-- Radeon Augmentation Library Python Binding - rocal_pybind module added 
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- rocal-test INFO:ROCM_PATH Set -- /opt/rocm
-- rocal-test requires rocAL. Install rocAL before running CTests
-- AMD ROCm rocAL RunTime Package --  rpp, mivisionx
-- AMD ROCm rocAL Dev Package --  rocal, rpp-devel, mivisionx-devel, lmdb-devel
-- AMD ROCm rocAL RunTime Package --  rpp, mivisionx
-- AMD ROCm rocAL Dev Package --  rocal, rpp-dev, mivisionx-dev, liblmdb-dev
-- Configuring done
-- Generating done
-- Build files have been written to: /root/build