pyhf
and funcX
Example code for vCHEP 2021 paper "Distributed statistical inference with pyhf enabled through funcX"
Create a Python 3 virtual environment and then install the pyhf
and funcX
dependencies in requirements.txt
.
(distributed-inference) $ python -m pip install --upgrade pip setuptools wheel
(distributed-inference) $ python -m pip install -r requirements.txt
To install a reproducible environment that is consistent down to the hash level, use pip-compile
to compile a lock file from requirements.txt
and install it following the pip-secure-install
recommendations.
(distributed-inference) $ bash compile_dependencies.sh
(distributed-inference) $ bash secure_install.sh
On EXPANSE, to use a Python 3.7+ runtime Conda must be used, so create a Conda environment from the expanse-environment.yml
provided, which uses the different requirements.txt
files to provide the dependencies.
$ conda env create -f expanse-environment.yml
$ conda activate distributed-inference
Once a GPU session has been entered, source the setup_expanse_funcx_test_env.sh
shell script to activate the environment and load all required modules
(distributed-inference) $ . setup_expanse_funcx_test_env.sh
EXPANSE has the following Nvidia drivers and GPUs:
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Thu_Jun_11_22:26:38_PDT_2020
Cuda compilation tools, release 11.0, V11.0.194
Build cuda_11.0_bu.TC445_37.28540450_0
$ nvidia-smi --list-gpus
GPU 0: Tesla V100-SXM2-32GB (UUID: GPU-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX)
Create a file named endpoint_id.txt
in the top level of this repository and save your funcX endpoint ID into the file.
(distributed-inference) $ touch endpoint_id.txt
This will be read in during the run.
Pass the config JSON file for the analysis you want to run to fit_analysis.py
(distributed-inference) $ python fit_analysis.py -c config/1Lbb.json -b numpy
$ python fit_analysis.py --help
usage: fit_analysis.py [-h] [-c CONFIG_FILE] [-b BACKEND]
configuration arguments provided at run time from the CLI
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
config file
-b BACKEND, --backend BACKEND
pyhf backend str alias