search

aertslab / scenicplus

SCENIC+ is a python package to build gene regulatory networks (GRNs) using combined or separate single-cell gene expression (scRNA-seq) and single-cell chromatin accessibility (scATAC-seq) data.
Other
186 stars 29 forks source link

Resource temporarily unavailable when run_scenicplus #348

Closed HarryShen668 closed 7 months ago

HarryShen668 commented 7 months ago

Hi @SeppeDeWinter

I encountered an error when running my run_scenicplus.py script in the terminal to generate the scplus_obj object

import dill
import scanpy as sc
import os
import warnings
warnings.filterwarnings("ignore")
import pandas
import pyranges
# Set stderr to null to avoid strange messages from ray
import sys
_stderr = sys.stderr
null = open(os.devnull,'wb')
work_dir = '..../work_dir '
tmp_dir = '.../tmp/scratch/leuven/330/vsc33053/'
adata = sc.read_h5ad(os.path.join(work_dir, 'h5/data_filter.h5ad'))
cistopic_obj = dill.load(open(os.path.join(work_dir, 'scenicplus/scATAC/cistopic_obj.pkl'), 'rb'))
menr = dill.load(open(os.path.join(work_dir, 'scenicplus/motifs/menr.pkl'), 'rb'))
adata.obs['barcode'] = adata.obs.index +'___'+ adata.obs['orig.ident']
adata.obs = adata.obs.set_index('barcode')
adata.raw.var.index = adata.raw.var['_index']
from scenicplus.scenicplus_class import create_SCENICPLUS_object
import numpy as np
scplus_obj = create_SCENICPLUS_object(
    GEX_anndata = adata.raw.to_adata(),
    cisTopic_obj = cistopic_obj,
    menr = menr
)
scplus_obj.X_EXP = np.array(scplus_obj.X_EXP.todense())
biomart_host = "http://apr2020.archive.ensembl.org/"
ray_tmp_dir='.../tmp/'
from scenicplus.wrappers.run_scenicplus import run_scenicplus
try:
    run_scenicplus(
        scplus_obj = scplus_obj,
        variable = ['GEX_celltype'],
        species = 'hsapiens',
        assembly = 'hg38',
        tf_file = '.../TF/utoronto_human_tfs_v_1.01.txt',
        save_path = os.path.join(work_dir, 'scenicplus'),
        biomart_host = biomart_host,
        upstream = [1000, 150000],
        downstream = [1000, 150000],
        calculate_TF_eGRN_correlation = True,
        calculate_DEGs_DARs = True,
        export_to_loom_file = True,
        export_to_UCSC_file = True,
        path_bedToBigBed = '.../cistarget',
        n_cpu = 66,
        _temp_dir = os.path.join(ray_tmp_dir, 'ray_spill'))
except Exception as e:
    #in case of failure, still save the object
    dill.dump(scplus_obj, open(os.path.join(work_dir, 'scenicplus/scplus_obj.pkl'), 'wb'), protocol=-1)
    raise(e)

However, during run_scenicplus(), the script encounters the error and fails.

2024-04-06 03:46:51,904 SCENIC+_wrapper INFO     .../scenicplus folder already exists.
2024-04-06 03:46:51,904 SCENIC+_wrapper INFO     Merging cistromes
2024-04-06 03:48:16,475 SCENIC+_wrapper INFO     Getting search space
2024-04-06 03:48:19,362 R2G          INFO     Downloading gene annotation from biomart dataset: hsapiens_gene_ensembl
2024-04-06 03:48:50,323 R2G          INFO     Downloading chromosome sizes from: http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes
2024-04-06 03:48:51,703 R2G          INFO     Extending promoter annotation to 10 bp upstream and 10 downstream
Warning! Start and End columns now have different dtypes: int32 and int64
Warning! Start and End columns now have different dtypes: int32 and int64
2024-04-06 03:48:53,404 R2G          INFO     Extending search space to:
                                    150000 bp downstream of the end of the gene.
                                    150000 bp upstream of the start of the gene.
Warning! Start and End columns now have different dtypes: int32 and int64
Warning! Start and End columns now have different dtypes: int32 and int64
2024-04-06 03:49:12,864 R2G          INFO     Intersecting with regions.
join: Strand data from other will be added as strand data to self.
If this is undesired use the flag apply_strand_suffix=False.
To turn off the warning set apply_strand_suffix to True or False.
Warning! Start and End columns now have different dtypes: int32 and int64
2024-04-06 03:49:14,632 R2G          INFO     Calculating distances from region to gene
2024-04-06 03:52:08,746 R2G          INFO     Imploding multiple entries per region and gene
2024-04-06 03:56:41,177 R2G          INFO     Done!
2024-04-06 03:56:42,025 SCENIC+_wrapper INFO     Inferring region to gene relationships
2024-04-06 03:56:42,493 R2G          INFO     Calculating region to gene importances, using GBM method
2024-04-06 03:57:01,595 INFO worker.py:1743 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 

initializing:   0%|          | 0/19021 [00:00<?, ?it/s]
initializing:   0%|          | 1/19021 [00:00<1:31:47,  3.45it/s]
initializing:   0%|          | 2/19021 [00:00<1:01:14,  5.18it/s]
initializing:   0%|          | 3/19021 [00:00<1:01:23,  5.16it/s]
initializing:   0%|          | 4/19021 [00:00<59:38,  5.31it/s]  
initializing:   0%|          | 5/19021 [00:00<58:04,  5.46it/s]
initializing:   0%|          | 6/19021 [00:01<59:12,  5.35it/s]
initializing:   0%|          | 7/19021 [00:01<57:11,  5.54it/s]
initializing:   0%|          | 8/19021 [00:01<58:34,  5.41it/s]
initializing:   0%|          | 9/19021 [00:01<58:13,  5.44it/s]
initializing:   0%|          | 10/19021 [00:01<52:44,  6.01it/s]
initializing:   0%|          | 11/19021 [00:02<53:49,  5.89it/s]
initializing:   0%|          | 12/19021 [00:02<55:57,  5.66it/s][2024-04-06 03:57:08,486 E 125110 125110] logging.cc:97: Unhandled exception: St12system_error. what(): Resource temporarily unavailable
[2024-04-06 03:57:09,454 E 125110 125110] logging.cc:104: Stack trace: 
 .../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0xfe373a) [0x7f7a70fa773a] ray::operator<<()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0xfe5e78) [0x7f7a70fa9e78] ray::TerminateHandler()
.../anaconda3/envs/scenicplus/bin/../lib/libstdc++.so.6(+0xb643c) [0x7f7bb468943c] __cxxabiv1::__terminate()
.../anaconda3/envs/scenicplus/bin/../lib/libstdc++.so.6(+0xb648e) [0x7f7bb468948e] __cxxabiv1::__unexpected()
.../anaconda3/envs/scenicplus/bin/../lib/libstdc++.so.6(__cxa_rethrow+0) [0x7f7bb4689680] __cxa_rethrow
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x54e857) [0x7f7a70512857] std::__throw_system_error()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x1108869) [0x7f7a710cc869]
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0xfe31a6) [0x7f7a70fa71a6] ray::parallel_memcopy()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x5e341c) [0x7f7a705a741c] __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x5a8471) [0x7f7a7056c471] __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x5dc0f7) [0x7f7a705a00f7] __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x67e0a2) [0x7f7a706420a2] __pyx_pf_3ray_7_raylet_10CoreWorker_58put_serialized_object_and_increment_local_ref()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x67e5eb) [0x7f7a706425eb] __pyx_pw_3ray_7_raylet_10CoreWorker_59put_serialized_object_and_increment_local_ref()
python(+0x201626) [0x5600a298d626] cfunction_call
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x5a32df) [0x7f7a705672df] __Pyx_PyObject_Call()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x6944de) [0x7f7a706584de] __pyx_f_3ray_7_raylet_prepare_args_and_increment_put_refs()
.../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_raylet.so(+0x69d938) [0x7f7a70661938] __pyx_pw_3ray_7_raylet_10CoreWorker_73submit_task()
python(+0x20ace8) [0x5600a2996ce8] method_vectorcall_VARARGS_KEYWORDS
python(PyObject_Vectorcall+0x2c) [0x5600a29869cc] PyObject_Vectorcall
python(_PyEval_EvalFrameDefault+0x716) [0x5600a2979e36] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x181) [0x5600a299d4c1] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x49f9) [0x5600a297e119] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x181) [0x5600a299d4c1] _PyFunction_Vectorcall
python(PyObject_Call+0x130) [0x5600a29a71e0] PyObject_Call
python(_PyEval_EvalFrameDefault+0x49f9) [0x5600a297e119] _PyEval_EvalFrameDefault
python(+0x2303a4) [0x5600a29bc3a4] _PyObject_VectorcallTstate.lto_priv.14
python(+0x22fb8e) [0x5600a29bbb8e] method_vectorcall
python(PyObject_Call+0x130) [0x5600a29a71e0] PyObject_Call
python(_PyEval_EvalFrameDefault+0x49f9) [0x5600a297e119] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x181) [0x5600a299d4c1] _PyFunction_Vectorcall
python(PyObject_Call+0x130) [0x5600a29a71e0] PyObject_Call
python(_PyEval_EvalFrameDefault+0x49f9) [0x5600a297e119] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x181) [0x5600a299d4c1] _PyFunction_Vectorcall
python(PyObject_Call+0x130) [0x5600a29a71e0] PyObject_Call
python(_PyEval_EvalFrameDefault+0x49f9) [0x5600a297e119] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x181) [0x5600a299d4c1] _PyFunction_Vectorcall
python(PyObject_Call+0x130) [0x5600a29a71e0] PyObject_Call
python(_PyEval_EvalFrameDefault+0x49f9) [0x5600a297e119] _PyEval_EvalFrameDefault
python(+0x2a442d) [0x5600a2a3042d] _PyEval_Vector
python(PyEval_EvalCode+0x9f) [0x5600a2a2fabf] PyEval_EvalCode
python(+0x2c2a1a) [0x5600a2a4ea1a] run_eval_code_obj
python(+0x2be593) [0x5600a2a4a593] run_mod
python(+0x2d3930) [0x5600a2a5f930] pyrun_file
python(_PyRun_SimpleFileObject+0x1ae) [0x5600a2a5f2ce] _PyRun_SimpleFileObject
python(_PyRun_AnyFileObject+0x44) [0x5600a2a5eff4] _PyRun_AnyFileObject
python(Py_RunMain+0x374) [0x5600a2a596f4] Py_RunMain
python(Py_BytesMain+0x37) [0x5600a2a1fa77] Py_BytesMain
/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f7c61dbf3d5] __libc_start_main
python(+0x29391d) [0x5600a2a1f91d]

*** SIGABRT received at time=1712347029 on cpu 47 ***
PC: @     0x7f7c61dd3207  (unknown)  raise
    @     0x7f7c628825d0  1075206688  (unknown)
    @     0x7f7bb468943c  (unknown)  __cxxabiv1::__terminate()
    @ ... and at least 3 more frames
[2024-04-06 03:57:09,454 E 125110 125110] logging.cc:361: *** SIGABRT received at time=1712347029 on cpu 47 ***
[2024-04-06 03:57:09,454 E 125110 125110] logging.cc:361: PC: @     0x7f7c61dd3207  (unknown)  raise
[2024-04-06 03:57:09,454 E 125110 125110] logging.cc:361:     @     0x7f7c628825d0  1075206688  (unknown)
[2024-04-06 03:57:09,454 E 125110 125110] logging.cc:361:     @     0x7f7bb468943c  (unknown)  __cxxabiv1::__terminate()
[2024-04-06 03:57:09,454 E 125110 125110] logging.cc:361:     @ ... and at least 3 more frames
Fatal Python error: Aborted

Stack (most recent call first):
  File ".../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/remote_function.py", line 435 in invocation
  File ".../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/remote_function.py", line 468 in _remote
  File ".../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/util/tracing/tracing_helper.py", line 310 in _invocation_remote_span
  File ".../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 21 in auto_init_wrapper
  File ".../anaconda3/envs/scenicplus/lib/python3.11/site-packages/ray/remote_function.py", line 139 in _remote_proxy
  File ".../tools/scenicplus/src/scenicplus/enhancer_to_gene.py", line 545 in _score_regions_to_genes
  File ".../tools/scenicplus/src/scenicplus/enhancer_to_gene.py", line 654 in calculate_regions_to_genes_relationships
  File ".../tools/scenicplus/src/scenicplus/wrappers/run_scenicplus.py", line 143 in run_scenicplus
  File ".../run_scenicplus.py", line 33 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, ... , aiohttp._websocket, frozenlist._frozenlist, grpc._cython.cygrpc (total: 334)

And I tested my ray by

import ray
import os

ray_tmp_dir = '.../tmp/'
ray.init(num_cpus=5, _temp_dir=os.path.join(ray_tmp_dir + 'ray_spill'))

@ray.remote
def f(x):
    return x * x

futures = [f.remote(i) for i in range(4)]
print(ray.get(futures))

ray.shutdown()

The above script executed successfully, indicating that my Ray can initialize and perform parallel tasks without any apparent issues on my system. I tried to downgrade my Ray version but did not resolve the issue and instead got even more new errors.

Furthermore, I possess approximately 300GB of available memory, which should be ample to accommodate the demands.

Could you provide any guidance to trigger this error? Thanks!

Version:

SeppeDeWinter commented 7 months ago

Hi @HarryShen668

This version of SCENIC+ has not been tested using python 3.11 (only 3.8), so that might be the cause of the issue. However, I would strongly recommend to use the development branch of the code.

Please see: https://scenicplus.readthedocs.io/en/development/ for documentation.

Seppe

HarryShen668 commented 7 months ago

Hi, Thank you for your response and guidance.

I eventually discovered that the issue might have been caused by an excessive number of threads, which was resolved by reducing the number of threads and switching to a server with more abundant computing resources.

However, I currently have a downstream analysis question. After completing the SCENIC+ analysis(run_scenicplus) of my entire map and generating the SCENIC+ object (scplus_obj), I want to analyze eRegulons that exhibit differences between two different biological conditions (which I have already added to the metadata_cell in scplus_obj) within a specific cell type in the object. Is there a method to achieve this by the generated scplus_obj?

Looking forward to your insights. Thank you so much!

SeppeDeWinter commented 7 months ago

Hi @HarryShen668

Please refer to these discussion thread related to your question, https://github.com/aertslab/scenicplus/discussions/124, https://github.com/aertslab/scenicplus/discussions/236, https://github.com/aertslab/scenicplus/discussions/318, Unfortunately, there is no one specific function to do this. I would suggest comparing AUC values between the two conditions.

Best,

Seppe

HarryShen668 commented 7 months ago

Thank you Seppe