number of thread Error at step cell_cluster when stereoPipeline_v7.1_manual_part2.sh

[wzhang42]

Hi, STOmics developer, I am running the stereoPipeline_v7.1_manual_part2.sh. It always exited at the step cell_cluster and throw the Error message as " The number of threads must be between 1 and n". I tried several rounds, including to provide more or less cpu nodes /threads, this issues are still there. Even I use only 1 cpu node and 1 thread. This similar message still throw out as ValueError: The number of threads must be between 1 and 1. It seems that the issue is from some python code which are from the inside singularity - SAW_7.1.sif. I have no idea on how to solve this issue and greatly need your help. The following is cut from one of the run log file.

2024-09-24 08:57:12.435781: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. Traceback (most recent call last): File "main/cell_cluster.py", line 137, in File "main/cell_cluster.py", line 131, in main File "main/cell_cluster.py", line 55, in cell_cluster File "/usr/local/python3/lib/python3.8/site-packages/stereo/core/st_pipeline.py", line 39, in wrapped res = func(*args, kwargs) File "/usr/local/python3/lib/python3.8/site-packages/stereo/core/st_pipeline.py", line 782, in neighbors neighbor, dists, connectivities = find_neighbors(x=self.result[pca_res_key].values, method=method, n_pcs=n_pcs, File "/usr/local/python3/lib/python3.8/site-packages/stereo/algorithm/neighbors.py", line 172, in find_neighbors knn_indices, knn_distances, forest = neighbor.compute_neighbors_umap( File "/usr/local/python3/lib/python3.8/site-packages/stereo/algorithm/neighbors.py", line 282, in compute_neighbors_umap knn_indices, knn_dists, forest = nearestneighbors( File "/usr/local/python3/lib/python3.8/site-packages/umap/umap.py", line 328, in nearest_neighbors knn_searchindex = NNDescent( File "/usr/local/python3/lib/python3.8/site-packages/pynndescent/pynndescent.py", line 780, in init numba.set_num_threads(self.n_jobs) File "/usr/local/python3/lib/python3.8/site-packages/numba/np/ufunc/parallel.py", line 607, in set_num_threads snt_check(n) File "/usr/local/python3/lib/python3.8/site-packages/numba/np/ufunc/parallel.py", line 569, in snt_check raise ValueError(msg) ValueError: The number of threads must be between 1 and 6** Command exited with non-zero status 1 Command being timed: "singularity exec ../SAW_7.1.sif cellCluster -i /research/groups/cab/projects/BrainAltas/common/STOmics/mousebrain2/Manual_Image_Registration/Manual_part2_Pipeline_Run/041.cellcut/A03397C6.adjusted.cellbin.gef -o /research/groups/cab/projects/BrainAltas/common/STOmics/mousebrain2/Manual_Image_Registration/Manual_part2_Pipeline_Run/051.cellcluster/A03397C6.adjusted.cell.cluster.h5ad" User time (seconds): 61.41 System time (seconds): 69.49 Percent of CPU this job got: 231% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:56.56 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 2194756 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 545 Minor (reclaiming a frame) page faults: 827431 Voluntary context switches: 41198 Involuntary context switches: 82451547 Swaps: 0 File system inputs: 796318 File system outputs: 8 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 1

[Clouate]

@wzhang42 Hi, how many CPU cores in your machine? This is due to an issue with the NUMBA package itself. Your machine may have previously set the environment variable NUMBA_NUM_THREADS too small, so you could try setting the environment variable NUMBA_NUM_THREADS, which is better to be larger than the number of CPU cores. I suggest that you could try the following code for testing.

export NUMBA_NUM_THREADS=32
# better to be larger than the number of CPU cores
export HDF5_USE_FILE_LOCKING=FALSE
outdir=/path/to/your/SAW_output_directory
export NUMBA_CACHE_DIR=${outDir}/tmp
export MPLCONFIGDIR=${outDir}/tmp
singularity exec ${sif} cellCluster -i  ${outDir}/041.cellcut/${SN}.adjusted.cellbin.gef -o ${outDir}/051.cellcluster/${SN}.adjusted.cell.cluster.h5ad

[wzhang42]

Thank you so much, your suggestions works.