search

fiedl / hole-ice-scripts

Example scripts for the hole-ice extension of clsim in the IceCube simulation framework.
0 stars 0 forks source link

Build in Zeuthen #9

Closed fiedl closed 2 years ago

fiedl commented 2 years ago
fiedl commented 2 years ago

Sync folders with mutagen

[2022-09-09 11:34:31] fiedl@fiedl-mbp ~/icecube/icetray main 79f9588ad
▶ brew install mutagen-io/mutagen/mutage
▶ brew install mutagen-io/mutagen/mutagen-compose
▶ mutagen sync create --stage-mode=internal --name icetray-zeuthen ~/icecube/icetray fiedl@warp-zeuthen.desy.de:/afs/ifh.de/group/amanda/scratch/fiedl/icetray
▶ mutagen sync create --stage-mode=internal --name hole-ice-scripts-zeuthen ~/icecube/hole-ice-scripts fiedl@warp-zeuthen.desy.de:/afs/ifh.de/group/amanda/scratch/fiedl/hole-ice-scripts

Sync später beenden:

[2022-09-09 11:52:26] fiedl@fiedl-mbp ~/icecube/icetray main 79f9588ad
▶ mutagen sync terminate icetray-zeuthen
▶ mutagen sync terminate hole-ice-scripts-zeuthen
fiedl commented 2 years ago

Build icetray on zeuthen

[2022-09-09 13:14:46] fiedl@kepler00 /afs/ifh.de/group/amanda/scratch/fiedl/icetray-build
▶ source /cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh
▶ cmake -D CMAKE_BUILD_TYPE=Debug -D SYSTEM_PACKAGES=true ../icetray
-- + clsim
-- +-- Building with SSE extensions for x86/amd64/x86_64.
-- +-- python [symlinks]
-- +-- OpenCL not found, skipping support
-- +-- CUDA support disabled (CUDA >= 10 not found)
fiedl commented 2 years ago

https://icecube-spno.slack.com/archives/C02KQL9KN/p1662722634332219

[2022-09-09 13:14:46] fiedl@kepler00 /afs/ifh.de/group/amanda/scratch/fiedl/icetray-build
▶ /cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh
▶ module load cuda/11.6
▶ export OPENCL_VENDOR_PATH=/cvmfs/icecube.opensciencegrid.org/distrib/OpenCL_RHEL_7_x86_64
▶ cmake -D CMAKE_BUILD_TYPE=Debug -D SYSTEM_PACKAGES=true ../icetray
-- + clsim
-- +-- Building with SSE extensions for x86/amd64/x86_64.
-- +-- python [symlinks]
-- +-- Build clsim with OpenCL support
-- --- no tabulator (OpenCL  is too old)
-- +-- CUDA architectures: 35 37 52 61 75 80 86
-- +-- CUDA support (found CUDA 11.6.124)
-- +-- Geant4 support
-- +-- clsim-pybindings
-- +-- /cvmfs/icecube.opensciencegrid.org/py3-v4.1.0/../data/safeprimes_base32.gz data file exists, skipping download
-- +-- gmp support (make_safeprimes utility)
fiedl commented 2 years ago

Or, with a symlink to access everything from ~/icecube:

[2022-09-09 14:20:59] fiedl@kepler00 ~
▶ ln -s /afs/ifh.de/group/amanda/scratch/fiedl/icecube ~/icecube

[2022-09-09 14:22:36] fiedl@kepler00 ~/icecube
▶ mkdir icetray hole-ice-scripts

[2022-09-09 14:23:47] fiedl@fiedl-mbp ~/icecube
▶ mutagen sync create --stage-mode=internal --name icetray-zeuthen ~/icecube/icetray fiedl@warp-zeuthen.desy.de:~/icecube/icetray
▶ mutagen sync create --stage-mode=internal --name hole-ice-scripts-zeuthen ~/icecube/hole-ice-scripts fiedl@warp-zeuthen.desy.de:~/icecube/hole-ice-scripts

[2022-09-09 14:28:48] fiedl@kepler00 ~/icecube/icetray-build
▶ /cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh
▶ module load cuda/11.6
▶ export OPENCL_VENDOR_PATH=/cvmfs/icecube.opensciencegrid.org/distrib/OpenCL_RHEL_7_x86_64
▶ cmake -D CMAKE_BUILD_TYPE=Debug -D SYSTEM_PACKAGES=true ../icetray
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
OPENCL_LIBRARIES

[2022-09-09 14:36:24] fiedl@kepler00 ~/icecube/icetray-build
▶ make -j 4
make: *** Keine Targets angegeben und keine »make«-Steuerdatei gefunden.  Schluss
fiedl commented 2 years ago

With the help of jvs, this is how icetray can be built in zeuthen:

  1. Optional: Use scratch space for quota reasons

    [2022-09-09 14:20:59] fiedl@kepler00 ~
▶ ln -s /afs/ifh.de/group/amanda/scratch/fiedl/icecube ~/icecube

  2. Optional: Sync local dev machine with zeuthen

    [2022-09-09 14:23:47] fiedl@fiedl-mbp ~/icecube
▶ mutagen sync create --stage-mode=internal --name icetray-zeuthen ~/icecube/icetray fiedl@warp-zeuthen.desy.de:~/icecube/icetray
▶ mutagen sync create --stage-mode=internal --name hole-ice-scripts-zeuthen ~/icecube/hole-ice-scripts fiedl@warp-zeuthen.desy.de:~/icecube/hole-ice-scripts

  3. For opencl support, we need a newer cmake version

    [2022-09-09 16:28:48] fiedl@kepler00 ~/icecube
▶ wget -qO- https://github.com/Kitware/CMake/releases/download/v3.24.1/cmake-3.24.1.tar.gz | tar xz

[2022-09-09 16:30:43] fiedl@kepler00 ~/icecube/cmake-3.24.1
▶ cmake .
▶ make -j 8
▶ export PATH=~/icecube/cmake-3.24.1/bin:$PATH
▶ cmake --version
cmake version 3.24.1

  4. Finally, build icetray

    [2022-09-09 16:39:46] fiedl@kepler00 ~/icecube/icetray-build
▶ /cvmfs/icecube.opensciencegrid.org/py3-v4.2.1/setup.sh
▶ module load cuda/11.6
▶ export OPENCL_VENDOR_PATH=/cvmfs/icecube.opensciencegrid.org/distrib/OpenCL_RHEL_7_x86_64/etc/OpenCL/vendors
▶ cmake -D CMAKE_BUILD_TYPE=Debug -D SYSTEM_PACKAGES=true ../icetray
▶ make -j 8
fiedl commented 2 years ago

Test propagation script in zeuthen

[2022-09-09 17:24:37] fiedl@kepler00 ~/icecube/hole-ice-scripts
▶ ~/icecube/icetray-build/env-shell.sh
▶ scripts/generate_photons.py
Traceback (most recent call last):
  File "scripts/generate_photons.py", line 63, in <module>
    main()
  File "scripts/generate_photons.py", line 16, in main
    random_number_generator = phys_services.I3SPRNGRandomService(
AttributeError: module 'icecube.phys_services' has no attribute 'I3SPRNGRandomService'
fiedl commented 2 years ago

With this workaround (https://github.com/fiedl/hole-ice-scripts/commit/de2bdd398d0c9af8ff213833d2b9ed85a438f360), the scripts can be executed:

[2022-09-09 17:50:55] fiedl@kepler00 ~/icecube/hole-ice-scripts
▶ ~/icecube/icetray-build/env-shell.sh
▶ scripts/generate_photons.py
▶ scripts/propagate_photons.py

Finally works 👍

fiedl commented 2 years ago

Mutagen

I'm using mutagen to sync my local icetray source folder and my local script folder to the corresponding folders in zeuthen. Then I can work on the files locally, open files locally in steamshovel, but run scripts remotely in zeuthen.

With password-less ssh into zeuthen (https://github.com/fiedl/hole-ice-scripts/issues/13), the mutagen commands should be:

Install mutagen

[2022-09-09 11:34:31] fiedl@fiedl-mbp ~/icecube/icetray main 79f9588ad
▶ brew install mutagen-io/mutagen/mutage
▶ brew install mutagen-io/mutagen/mutagen-compose

Create directory syncs

[2022-09-09 11:34:31] fiedl@fiedl-mbp ~/icecube/icetray main 79f9588ad
▶ kinit --renewable fiedl@IFH.DE
▶ mutagen sync create --stage-mode=internal --name icetray-zeuthen ~/icecube/icetray fiedl@ice-wgs-gpu.ifh.de:~/icecube/icetray
▶ mutagen sync create --stage-mode=internal --name hole-ice-scripts-zeuthen ~/icecube/hole-ice-scripts fiedl@ice-wgs-gpu.ifh.de:~/icecube/hole-ice-scripts

Check sync session

[2022-09-10 13:53:29] fiedl@fiedl-mbp ~/icecube
▶ mutagen sync list

Resume sync sessions after network has been down

[2022-09-10 14:33:36] fiedl@fiedl-mbp ~/icecube
▶ mutagen sync resume icetray-zeuthen hole-ice-scripts-zeuthen

Terminate sync sessions

[2022-09-10 14:33:36] fiedl@fiedl-mbp ~/icecube
▶ mutagen sync terminate icetray-zeuthen hole-ice-scripts-zeuthen
fiedl commented 2 years ago

With --sync-mode=two-way-resolved, the local directory takes precedence in case of conflicts.

https://mutagen.io/documentation/synchronization

[2022-09-10 14:33:36] fiedl@fiedl-mbp ~/icecube
▶ mutagen sync create --stage-mode=internal --sync-mode=two-way-resolved --name hole-ice-scripts-zeuthen ~/icecube/hole-ice-scripts fiedl@ice-wgs-gpu.ifh.de:~/icecube/hole-ice-scripts
▶ mutagen sync create --stage-mode=internal --sync-mode=two-way-resolved --name icetray-zeuthen ~/icecube/icetray fiedl@ice-wgs-gpu.ifh.de:~/icecube/icetray