search

aertslab / pycisTopic

pycisTopic is a Python module to simultaneously identify cell states and cis-regulatory topics from single cell epigenomics data.
Other
58 stars 12 forks source link

OpenBLAS warning #141

Closed TingTingShao closed 6 months ago

TingTingShao commented 6 months ago

Hi,

I reinstalled the package as follows:

conda create --name scenic2 python=3.11 -y
conda activate  scenic2
git clone https://github.com/aertslab/scenicplus.git
cd scenicplus
pip install .

But there is always a warning when I do the dimension reduction with cistopic, target cell, and the topic binarization step. With this warning, the process seems stopped.

I canceled the job, and tried to rerun, and it showed the incomplete-job error, so I thought it might be running. I let the job run again for another 10 hours, but it still stopped at 

localrule topic_binary:
    input: /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/cistopic_obj.pkl, /scratch/leuven/351/vsc35107/flags/pycistopic_visual.done
    output: /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/bi_regions.png, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/bi_cell.png, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/binarized_cell_topic.pkl, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/region_bin_topics_top_3k.pkl, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/region_bin_topics_otsu.pkl, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/region_bin_topics_otsu.png, /scratch/leuven/351/vsc35107/flags/topic_binary.done
    jobid: 2
    reason: Missing output files: /scratch/leuven/351/vsc35107/flags/topic_binary.done, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/region_bin_topics_top_3k.pkl, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/binarized_cell_topic.pkl, /lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse/01/030results_cistopic/030visual/region_bin_topics_otsu.pkl; Input files updated by another job: /scratch/leuven/351/vsc35107/flags/pycistopic_visual.done
    resources: tmpdir=/tmp

export ANT_HOME="/vsc-hard-mounts/leuven-data/351/vsc35107/master_thesis/apache-ant-1.10.14"
export BASH_ENV="/usr/share/lmod/lmod/init/bash"
export CONDA_DEFAULT_ENV="scenic2"
export CONDA_EXE="/data/leuven/351/vsc35107/miniconda3/bin/conda"
export CONDA_PREFIX="/data/leuven/351/vsc35107/miniconda3/envs/scenic2"
export CONDA_PROMPT_MODIFIER="(scenic2) "
export CONDA_PYTHON_EXE="/data/leuven/351/vsc35107/miniconda3/bin/python"
export CONDA_SHLVL="1"
export DEBUGINFOD_URLS="https://debuginfod.centos.org/ "
export ENVIRONMENT="BATCH"
export GQUOTE_DIR="/apps/leuven/goldreports/"
export HISTCONTROL="ignoredups"
export HISTSIZE="1000"
export HISTTIMEFORMAT="%F %T "
export HOME="/user/leuven/351/vsc35107"
export HOSTNAME="k28l03"
export IBUTILS_NODE_NAME_MAP_FILE_PATH="/etc/rdma/ib-node-name-map"
export JAVA_HOME="/vsc-hard-mounts/leuven-data/351/vsc35107/master_thesis/jdk1.8.0_202"
export KMP_DUPLICATE_LIB_OK="True"
export KMP_INIT_AT_FORK="FALSE"
export LANG="en_US.UTF-8"
export LC_ALL="C.UTF-8"
export LD_LIBRARY_PATH="/data/leuven/351/vsc35107/miniconda3/envs/snap2/lib:"
export LESSOPEN="||/usr/bin/lesspipe.sh %s"
export LMOD_CMD="/usr/share/lmod/lmod/libexec/lmod"
export LMOD_DIR="/usr/share/lmod/lmod/libexec"
export LMOD_PKG="/usr/share/lmod/lmod"
export LMOD_ROOT="/usr/share/lmod"
export LMOD_SETTARG_FULL_SUPPORT="no"
export LMOD_VERSION="8.7.32"
export LMOD_sys="Linux"
export LOGNAME="vsc35107"
export LT_SCRATCH="/scratch/leuven_lt/351/vsc35107"
export MAIL="/user/leuven/351/vsc35107/inbox"
export MANPATH="/usr/share/lmod/lmod/share/man::/opt/puppetlabs/puppet/share/man"
export MODULEPATH="/apps/leuven/etc/modules:/apps/leuven/common/modules/all:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core"
export MODULEPATH_ROOT="/usr/share/modulefiles"
export MODULESHOME="/usr/share/lmod/lmod"
export OLDPWD="/lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse"
export PATH="/data/leuven/351/vsc35107/miniconda3/envs/scenic2/bin:/data/leuven/351/vsc35107/miniconda3/condabin:/bin:/sbin"
export PERL5LIB="/user/leuven/351/vsc35107/perl5/lib/perl5:/user/leuven/351/vsc35107/perl5/lib/perl5/"
export PERL_LOCAL_LIB_ROOT="/user/leuven/351/vsc35107/perl5"
export PERL_MB_OPT="--install_base \"/user/leuven/351/vsc35107/perl5\""
export PERL_MM_OPT="INSTALL_BASE=/user/leuven/351/vsc35107/perl5"
export POLARS_ALLOW_EXTENSION="true"
export PWD="/lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse"
export RAY_CLIENT_MODE="0"
export RUST_BACKTRACE="1"
export SHELL="/bin/bash"
export SHLVL="3"
export SLURMD_NODENAME="k28l03"
export SLURM_CLUSTER_NAME="wice"
export SLURM_CONF="/etc/slurm/slurm.conf"
export SLURM_CPUS_ON_NODE="72"
export SLURM_CPUS_PER_TASK="1"
export SLURM_EXPORT_ENV="NONE"
export SLURM_GET_USER_ENV="1"
export SLURM_GPUS_PER_NODE="0"
export SLURM_GTIDS="0"
export SLURM_JOBID="61412522"
export SLURM_JOB_ACCOUNT="lp_alethea"
export SLURM_JOB_CPUS_PER_NODE="72"
export SLURM_JOB_END_TIME="1715696729"
export SLURM_JOB_GID="2535107"
export SLURM_JOB_ID="61412522"
export SLURM_JOB_NAME="r03"
export SLURM_JOB_NODELIST="k28l03"
export SLURM_JOB_NUM_NODES="1"
export SLURM_JOB_PARTITION="bigmem"
export SLURM_JOB_QOS="lp_alethea"
export SLURM_JOB_START_TIME="1715437529"
export SLURM_JOB_UID="2535107"
export SLURM_JOB_USER="vsc35107"
export SLURM_LOCALID="0"
export SLURM_MEM_PER_CPU="28000"
export SLURM_NNODES="1"
export SLURM_NODEID="0"
export SLURM_NODELIST="k28l03"
export SLURM_NODE_ALIASES="(null)"
export SLURM_NPROCS="72"
export SLURM_NTASKS="72"
export SLURM_PRIO_PROCESS="0"
export SLURM_PROCID="0"
export SLURM_SCRIPT_CONTEXT="prolog_task"
export SLURM_SUBMIT_DIR="/lustre1/project/stg_00079/students/tingting/data/sun/snap2_PFC_synapse"
export SLURM_SUBMIT_HOST="k28i14"
export SLURM_TASKS_PER_NODE="72"
export SLURM_TASK_PID="3748025"
export SLURM_TOPOLOGY_ADDR="spine_icelake.km28.k28.k28l03"
export SLURM_TOPOLOGY_ADDR_PATTERN="switch.switch.switch.node"
export SLURM_WORKING_CLUSTER="wice:tier2-p-slurm-3:6817:9984:109"
export SSH_AGENT_PID="3747905"
export SSH_AUTH_SOCK="/var/run/cert-inject/agent136942647"
export S_COLORS="auto"
export TERM="xterm-256color"
export TMPDIR="/tmp"
export UCX_NET_DEVICES="mlx5_0:1"
export USER="vsc35107"
export VISUAL="vim"
export VSC_ARCH_LOCAL="icelake"
export VSC_ARCH_SUFFIX=""
export VSC_CLUSTER_TOOLCHAIN_VERSION="2021a"
export VSC_DATA="/data/leuven/351/vsc35107"
export VSC_DATA_VO="/does/not/exist"
export VSC_DATA_VO_USER="/does/not/exist/vsc35107"
export VSC_HOME="/user/leuven/351/vsc35107"
export VSC_INSTITUTE="leuven"
export VSC_INSTITUTE_CLUSTER="wice"
export VSC_INSTITUTE_LOCAL="leuven"
export VSC_INSTITUTE_TIER=""
export VSC_OS_LOCAL="rocky8"
export VSC_QUOTE="TRUE"
export VSC_SCRATCH="/scratch/leuven/351/vsc35107"
export VSC_SCRATCH_GLOBAL="/site_scratch"
export VSC_SCRATCH_NODE="/tmp"
export VSC_SCRATCH_SITE="/scratch/leuven/351/vsc35107"
export XDG_CACHE_HOME="/data/leuven/351/vsc35107/.cache"
export _CE_CONDA=""
export _CE_M=""
export _ModuleTable001_="X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0ge30sCm1wYXRoQSA9IHsKIi9hcHBzL2xldXZlbi9ldGMvbW9kdWxlcyIsICIvYXBwcy9sZXV2ZW4vY29tbW9uL21vZHVsZXMvYWxsIgosICIvZXRjL21vZHVsZWZpbGVzIiwgIi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMiLCAiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9MaW51eCIKLCAiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlIiwgIi91c3Ivc2hhcmUvbG1vZC9sbW9kL21vZHVsZWZpbGVzL0NvcmUiLAp9LApzeXN0ZW1CYXNlTVBBVEggPSAiL2V0Yy9tb2R1bGVmaWxlczovdXNyL3No"
export _ModuleTable002_="YXJlL21vZHVsZWZpbGVzOi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvTGludXg6L3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlOi91c3Ivc2hhcmUvbG1vZC9sbW9kL21vZHVsZWZpbGVzL0NvcmUiLAp9Cg=="
export _ModuleTable_Sz_="2"
export __LMOD_REF_COUNT_MODULEPATH="/apps/leuven/etc/modules:2;/apps/leuven/common/modules/all:1;/etc/modulefiles:1;/usr/share/modulefiles:1;/usr/share/modulefiles/Linux:1;/usr/share/modulefiles/Core:1;/usr/share/lmod/lmod/modulefiles/Core:1"
export which_declare="declare -f"
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.

No process at all, could you help me with this?

Thanks tingting

TingTingShao commented 6 months ago

BTW, I tried with solutions suggested here:

https://stackoverflow.com/questions/75619847/python-sklearn-openblas-error-for-kmeans

but nothing worked for me.

ghuls commented 6 months ago

You can set those variables before running pycistopic (or python):

export OPENBLAS_NUM_THREADS="8"

pycistopic command

For example on HPC I have set the following in my ~/.bashrc, so everything that uses Intel math library or OpenBlas does not use all cores (some python or R library) as in general above 8 threads does not give much speedup anymore (most of the time the speed stays the same, but the CPU usage is higher):

# Restrict Intel math library to max 8 threads.
export MKL_NUM_THREADS=8
# Restrict OpenBlas library to max 8 threads.
export OMP_NUM_THREADS=8
# Restrict Polars to max 8 threads.
export POLARS_MAX_THREADS=8
TingTingShao commented 6 months ago

Thanks, it worked.

TingTing