saudet
commented
8 months ago If it works with GitHub Actions, that's alright
Closed archenroot closed 8 months ago
saudet
commented
8 months ago If it works with GitHub Actions, that's alright
archenroot
commented
8 months ago Ok, I will try to look at GitHub actions configuration to replicate it locally.
archenroot
commented
8 months ago @saudet - ok I looked into the github action script and extracted only related content and just testing if it works:
export SUDO=$(which sudo)
mkdir -p .ccache
echo "max_size = 2.0G" > .ccache/ccache.conf
echo "hash_dir = false" >> .ccache/ccache.conf
echo "sloppiness = file_macro,include_file_ctime,include_file_mtime,pch_defines,time_macros" >> .ccache/ccache.conf
export ARCH=amd64
export PREFIX=x86_64-linux-gnu
export ARCH_CUDA=x86_64
export CUDA=cuda-repo-rhel8-12-3-local-12.3.1_545.23.08-1.x86_64.rpm
export CUDNN=8.9.7.29-1.cuda12.2.x86_64
export NCCL=2.19.3-1+cuda12.3.x86_64
export NVCOMP=nvcomp_3.0.5_x86_64_12.x
$SUDO dpkg --list
$SUDO apt-get update
$SUDO apt-get -y install gnupg
source /etc/os-release
export CODENAME=$UBUNTU_CODENAME
if [[ ! "$ARCH" == "amd64" ]]; then
# https://github.com/actions/runner-images/issues/675
$SUDO gem install apt-spy2
$SUDO apt-spy2 check
$SUDO apt-spy2 fix --commit
$SUDO sed -i 's/azure\.//' /etc/apt/apt-mirrors.txt /etc/apt/sources.list
$SUDO cat /etc/apt/apt-mirrors.txt /etc/apt/sources.list
$SUDO apt-get update
# https://github.com/actions/runner-images/issues/4589
$SUDO apt-add-repository -y ppa:ondrej/php
$SUDO apt-get -y install ppa-purge
$SUDO ppa-purge -y ppa:ondrej/php
fi
if [[ "$ARCH" == "i386" ]]; then
$SUDO dpkg --add-architecture $ARCH
TOOLCHAIN="gcc-$PREFIX g++-$PREFIX gfortran-$PREFIX"
elif [[ ! "$ARCH" == "amd64" ]]; then
$SUDO dpkg --add-architecture $ARCH
$SUDO sed -i 's/deb http/deb [arch=amd64] http/g' /etc/apt/sources.list
$SUDO echo deb [arch=$ARCH] http://ports.ubuntu.com/ubuntu-ports $CODENAME main restricted universe multiverse | $SUDO tee -a /etc/apt/sources.list
$SUDO echo deb [arch=$ARCH] http://ports.ubuntu.com/ubuntu-ports $CODENAME-updates main restricted universe multiverse | $SUDO tee -a /etc/apt/sources.list
$SUDO echo deb [arch=$ARCH] http://ports.ubuntu.com/ubuntu-ports $CODENAME-backports main restricted universe multiverse | $SUDO tee -a /etc/apt/sources.list
$SUDO echo deb [arch=$ARCH] http://ports.ubuntu.com/ubuntu-ports $CODENAME-security main restricted universe multiverse | $SUDO tee -a /etc/apt/sources.list
TOOLCHAIN="gcc-$PREFIX g++-$PREFIX gfortran-$PREFIX linux-libc-dev-$ARCH-cross binutils-multiarch"
fi
$SUDO apt-get update
$SUDO apt-get -y install gcc-multilib g++-multilib gfortran-multilib python3 python2.7 python3-minimal python2.7-minimal rpm libasound2-dev:$ARCH freeglut3-dev:$ARCH libfontconfig-dev:$ARCH libgtk2.0-dev:$ARCH libusb-dev:$ARCH libusb-1.0-0-dev:$ARCH libffi-dev:$ARCH libbz2-dev:$ARCH zlib1g-dev:$ARCH libxcb1-dev:$ARCH
$SUDO apt-get -y install pkg-config ccache clang $TOOLCHAIN openjdk-8-jdk ant python2 python3-pip swig git file wget unzip tar bzip2 gzip patch autoconf-archive autogen automake cmake make libtool bison flex perl nasm ragel curl libcurl4-openssl-dev libssl-dev libffi-dev libbz2-dev zlib1g-dev rapidjson-dev
$SUDO python3 -m pip install gdown || $SUDO python3 -m pip install gdown
echo "ARCH=$ARCH" >> $GITHUB_ENV
echo "PREFIX=$PREFIX" >> $GITHUB_ENV
echo Installing CUDA, cuDNN, nvCOMP, etc
curl -LO https://developer.download.nvidia.com/compute/cuda/12.3.1/local_installers/$CUDA
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libcudnn8-$CUDNN.rpm
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libcudnn8-devel-$CUDNN.rpm
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libnccl-$NCCL.rpm
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libnccl-devel-$NCCL.rpm
$SUDO rpm -i --force --ignorearch --nodeps $CUDA libcudnn*.rpm libnccl*.rpm
rm -f *.rpm *.tgz *.txz *.tar.*
pushd /var/cuda-repo-rhel8-12-3-local/; $SUDO rpm -i --force --ignorearch --nodeps cuda*.rpm libc*.rpm libn*.rpm; $SUDO rm *.rpm; popd
$SUDO ln -sf /usr/local/cuda/lib64/ /usr/local/cuda/lib
$SUDO ln -sf /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/libcuda.so
$SUDO ln -sf /usr/local/cuda/lib64/stubs/libnvidia-ml.so /usr/local/cuda/lib64/libnvidia-ml.so
$SUDO mv /usr/include/cudnn* /usr/include/nccl* /usr/local/cuda/include/
$SUDO mv /usr/lib64/libcudnn* /usr/lib64/libnccl* /usr/local/cuda/lib64/
if [[ -n ${NVCOMP:-} ]]; then
echo "installing nvcomp"
curl -LO https://developer.download.nvidia.com/compute/nvcomp/3.0.5/local_installers/$NVCOMP.tgz
$SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/lib64/ --strip-components=1 lib/ || $SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/lib64/ --strip-components=2 nvcomp-3.0.5-ctk-12.2/lib/
$SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/include/ --strip-components=1 include/ || $SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/include/ --strip-components=2 nvcomp-3.0.5-ctk-12.2/include/
rm -f $NVCOMP.tgz
fi
@saudet - the only strange fact is that you use RPM (CentOS, Fedora, RHEL) packages on Ubuntu (where we have Debian deb). Don't you want to recreate this flow to be based on the latest LTS Ubuntu image, ie. 22.04 LTS and Nividia DEB packages installation?
saudet
commented
8 months ago They only have RHEL packages for ppc64le...
archenroot
commented
8 months ago I look at deploy-ubuntu/action.yml and see this:
name: Deploy on Ubuntu
runs:
using: composite
...
...
if [[ -n ${ARCH_CUDA:-} ]] && [[ -n ${CI_DEPLOY_NEED_CUDA:-} ]]; then
echo Installing CUDA, cuDNN, nvCOMP, etc
curl -LO https://developer.download.nvidia.com/compute/cuda/12.3.1/local_installers/$CUDA
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libcudnn8-$CUDNN.rpm
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libcudnn8-devel-$CUDNN.rpm
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libnccl-$NCCL.rpm
curl -LO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/$ARCH_CUDA/libnccl-devel-$NCCL.rpm
$SUDO rpm -i --force --ignorearch --nodeps $CUDA libcudnn*.rpm libnccl*.rpm
rm -f *.rpm *.tgz *.txz *.tar.*
pushd /var/cuda-repo-rhel8-12-3-local/; $SUDO rpm -i --force --ignorearch --nodeps cuda*.rpm libc*.rpm libn*.rpm; $SUDO rm *.rpm; popd
$SUDO ln -sf /usr/local/cuda/lib64/ /usr/local/cuda/lib
$SUDO ln -sf /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/libcuda.so
$SUDO ln -sf /usr/local/cuda/lib64/stubs/libnvidia-ml.so /usr/local/cuda/lib64/libnvidia-ml.so
$SUDO mv /usr/include/cudnn* /usr/include/nccl* /usr/local/cuda/include/
$SUDO mv /usr/lib64/libcudnn* /usr/lib64/libnccl* /usr/local/cuda/lib64/
if [[ -n ${NVCOMP:-} ]]; then
curl -LO https://developer.download.nvidia.com/compute/nvcomp/3.0.5/local_installers/$NVCOMP.tgz
$SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/lib64/ --strip-components=1 lib/ || $SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/lib64/ --strip-components=2 nvcomp-3.0.5-ctk-12.2/lib/
$SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/include/ --strip-components=1 include/ || $SUDO tar -xvf $NVCOMP.tgz -C /usr/local/cuda/include/ --strip-components=2 nvcomp-3.0.5-ctk-12.2/include/
rm -f $NVCOMP.tgz
fiSo cuda build is using RPM on Ubuntu...do I miss something from github action?
archenroot
commented
8 months ago I cannot see neither local or network DEB nvidia packages installed in that ubuntu deployment shell script... thx for helping me understand
saudet
commented
8 months ago If you can find Ubuntu packages for ppc64le, sure
archenroot
commented
8 months ago aha, these doesn't exists... thx for hint.
@saudet - hi buddy,
I have seen your latest commit which upgrades to the latest Nvidia Cuda stack, I installed the following files:
I installed it via following procedure taken from Nvidia install guides:
when I query for stack installed I get for cuda:
and for cudnn
I am on Ubuntu 22.04.3 LTS. my nvcc:
cudnn:
All above seems to match your commit https://github.com/bytedeco/javacpp-presets/commit/a885e6c58268eee4510525c9597c5b90a169d7a2
but my local mvn install fails. javacpp-presets-cuda-12.3-failure.txt
Seems to me I have something totally wrong against your config in GitHub actions as soon as your commit passed the CICD.