Add DeepSpeed easyblock

easybuilders / easybuild-easyblocks

Collection of easyblocks that implement support for building and installing software with EasyBuild.

https://easybuild.io

GNU General Public License v2.0

106 stars 285 forks source link

Add DeepSpeed easyblock #3450

Open VRehnberg opened 1 month ago

VRehnberg commented 1 month ago

Adds an EasyBlock for building DeepSpeed. Main reason for not handling it in the easyconfig is to access cuda_compute_capabilities.

Also expands PythonPackage easyblock so that PATH and LD_LIBRARY_PATH is exported for runtest with testinstall as well (compared to only PYTHONPATH originally).

VRehnberg commented 1 month ago

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

FAIL (build issue) DeepSpeed-0.14.5-foss-2023a-CUDA-12.1.1.eb (partial log available at https://gist.github.com/VRehnberg/3537c6b4776752b8e1cf3abbb8f9e06d)

Build succeeded for 0 out of 1 (1 easyconfigs in total) alvis1-03 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8 See https://gist.github.com/VRehnberg/0ee552005c7319653c59e7beb3a63b7e for a full test report.

VRehnberg commented 1 month ago

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

FAIL (build issue) DeepSpeed-0.14.5-foss-2023a-CUDA-12.1.1.eb (partial log available at https://gist.github.com/VRehnberg/2773836659824c214e275a5f2ec29721)

Build succeeded for 0 out of 1 (1 easyconfigs in total) alvis1-02 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8 See https://gist.github.com/VRehnberg/62dc55136208c4601668fbff0a174866 for a full test report.

VRehnberg commented 1 month ago

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

FAIL (build issue) DeepSpeed-0.14.5-foss-2023a-CUDA-12.1.1.eb (partial log available at https://gist.github.com/VRehnberg/a82a4faf68919c4a5aaee509bfe9eaa9)

Build succeeded for 0 out of 1 (1 easyconfigs in total) alvis1-02 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8 See https://gist.github.com/VRehnberg/00bb6c0b928ef06dc1c7f9ab2a23b3c2 for a full test report.