Request of installation of LLVM library on Casper

[sjsprecious]

Sorry if I post this request at the wrong place but I would like the latest LLVM library to be installed on Casper so that we could explore some just-in-time compiling features later.

[benkirk]

What is your need date?

I ask because our current plan is to update the operating system on Casper in the early 2023 timeframe, and include a totally fresh user software stack based off of Spack. This would be much easier to include in that update, we estimate around March 2023 for user access.

[sjsprecious]

Thanks @benkirk. I plan to test the LLVM module before the beginning of NCAR/NOAA Open Hackathon, which will be the end of February 2023. But if you think it is easier to include it during the update of Casper in March 2023, that is fine and I can try to find a way to work around.

[benkirk]

What features & dependent libraries might you need? Is it simply the compiler with CUDA support?

I'll play with this and see if we can meet your need date with just the compiler, not fully integrated with the existing Casper module tree.

[sjsprecious]

Thanks @benkirk for your information. I would let @mattldawson answer your questions since he is more familiar with the LLVM requirement for the JIT compiling.

[mattldawson]

hi @benkirk - we've been using just the core LLVM project to explore JIT-compiling some of the model code (CPU only). We don't need clang or any of the other additional projects (but I guess it wouldn't hurt to have them, and I'm not sure what Jian might need for GPU stuff once we get to that point.) We're using these libraries from the LLVM project: support core orcjit native irreader

I've just been testing things locally, but yesterday I built the LLVM v15.0.6 project on Casper here: /glade/scratch/mattdawson/llvm-15 from source here: /glade/scratch/mattdawson/llvm-project-15.0.6.src, to try out our tests on Casper, if this is useful. (We might be able to just use this for the Hack-a-thon if that works better with the Casper upgrade plans.)

I will try to figure out with Jian if there is anything we might need in addition for the Hack-a-thon

[sjsprecious]

@mattldawson: Wow, thanks for your update and I did not realize that you were trying to build LLVM on Casper already. I could try to use your LLVM path and run your JIT example on Casper to examine whether it works on my end. If yes, probably it is enough for the Hackathon but we could always add more libraries later if they are required for GPU.

[benkirk]

@mattldawson & @sjsprecious

Thanks, looks like you are set for a little bit anyway!

I've installed llvm@15.0.4 with CUDA support on Casper as well: /glade/work/benkirk/my_spack_playground/deploy/view/CentOS-compilers/llvm/15.0.4

I've not yet kicked the tires much with the CUDA support, other than getting it to compile with a consistent cuda and cuda_arch.

==> Installed packages
-- linux-centos7-x86_64 / gcc@11.3.0 ----------------------------
zk5vnam llvm@15.0.4+clang+compiler-rt+cuda+flang+gold+internal_unwind~ipo+libcxx~link_llvm_dylib+lld+lldb+llvm_dylib~mlir+omp_as_runtime~omp_debug~omp_tsan+polly~python~split_dwarf~z3 build_system=cmake build_type=Release cuda_arch=80 patches=3dc9bc1 shlib_symbol_version=none targets=none version_suffix=none
vzv5l7r     binutils@2.38~gas+gold~headers~interwork+ld~libiberty~lto+nls+plugins build_system=autotools libs=shared,static
zl7eejh         diffutils@3.8 build_system=autotools
7i6nsqc             libiconv@1.16 build_system=autotools libs=shared,static
o5yhcaq         gettext@0.21.1+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools
2yjvplf             tar@1.34 build_system=autotools zip=pigz
ua2dk5n                 pigz@2.7 build_system=makefile
ougyety                 zstd@1.5.2+programs build_system=makefile compression=none libs=shared,static
qiv4z2x     cmake@3.24.3~doc+ncurses+ownlibs~qt build_system=generic build_type=Release
gxlv7fe         openssl@1.1.1s~docs~shared build_system=generic certs=mozilla
kw6oooa             ca-certificates-mozilla@2022-10-11 build_system=generic
jqtrzzw     cuda@11.8.0~allow-unsupported-compilers~dev build_system=generic
3qzwztv         libxml2@2.10.3~python build_system=autotools
6ip3hzq     hwloc@2.8.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static
fwljxhm         libpciaccess@0.16 build_system=autotools
qkwvmfy             libtool@2.4.7 build_system=autotools
nsvf4s2                 m4@1.4.19+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7
llzrwrn                     libsigsegv@2.13 build_system=autotools
bf3yj4s             util-macros@1.19.3 build_system=autotools
5kvx74h     libedit@3.1-20210216 build_system=autotools
efdk47z     libelf@0.8.13 build_system=autotools
s7ljr3d     libffi@3.4.2 build_system=autotools
3a5alko     ncurses@6.3~symlinks+termlib abi=none build_system=autotools
aqb4ow7     ninja@1.11.1 build_system=generic
4kpdbiy     perl-data-dumper@2.173 build_system=perl
6qo6hp5         perl@5.36.0+cpanm+shared+threads build_system=generic
abgzjy5             berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc
rzt66kl     pkgconf@1.8.0 build_system=autotools
qz37yfl     python@3.10.8+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060
2auw3wr         bzip2@1.0.8~debug~pic+shared build_system=generic
cvodpnn         expat@2.4.8+libbsd build_system=autotools
ovf6sww             libbsd@0.11.5 build_system=autotools
xtaemos                 libmd@1.0.4 build_system=autotools
kbqlols         gdbm@1.23 build_system=autotools
lbxcvam         readline@8.1.2 build_system=autotools
6ixd27k         sqlite@3.39.4+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools
sujcpgq         util-linux-uuid@2.38.1 build_system=autotools
dn2w46o         xz@5.2.7~pic build_system=autotools libs=shared,static
apr7omr     zlib@1.2.13+optimize+pic+shared build_system=makefile

[sjsprecious]

Thanks @benkirk. I could confirm that I am able to use Matt's LLVM build on Casper to run a test successfully. I could try your build for the Hackathon since we will need the GPU support and let you know if I run into any issue later. Thank you for your help!

[vanderwb]

@sjsprecious - are you okay with this being closed, since it sounds like you have a solution and this isn't relevant to Gust at present anyway?

[sjsprecious]

Yes, Brian. It could be closed as complete. Thanks Ben's help on this issue.