search

aertslab / scenicplus

SCENIC+ is a python package to build gene regulatory networks (GRNs) using combined or separate single-cell gene expression (scRNA-seq) and single-cell chromatin accessibility (scATAC-seq) data.
Other
178 stars 28 forks source link

A same bug as #146 in compute_qc_stats() under pandas1.5 #168

Closed aaronwolf1995 closed 1 year ago

aaronwolf1995 commented 1 year ago

Describe the bug When I was going through https://scenicplus.readthedocs.io/en/latest/pbmc_multiome_tutorial.html, I had the same error as in #146 when I was running compute_qc_stats(), and get the exact same plot: metadata_bc, profile_data_dict = compute_qc_stats( ... fragments_dict = fragments_dict, ... tss_annotation = annot, ... stats=['barcode_rank_plot', 'duplicate_rate', 'insert_size_distribution', 'profile_tss', 'frip'], ... label_list = None, ... path_to_regions = path_to_regions, ... n_cpu = 1, ... valid_bc = None, ... n_frag = 100, ... n_bc = None, ... tss_flank_window = 1000, ... tss_window = 50, ... tss_minimum_signal_window = 100, ... tss_rolling_window = 10, ... remove_duplicates = True, ... _temp_dir = os.path.join(tmp_dir + 'ray_spill'))

here are the error reports: 2023-06-20 15:14:10,635 cisTopic INFO Reading 10x_pbmc

2023-06-20 15:14:36,991 cisTopic INFO Computing barcode rank plot for 10x_pbmc 2023-06-20 15:14:36,991 cisTopic INFO Counting fragments 2023-06-20 15:14:38,989 cisTopic INFO Marking barcodes with more than 100 2023-06-20 15:14:39,015 cisTopic INFO Returning plot data 2023-06-20 15:14:39,016 cisTopic INFO Returning valid barcodes 2023-06-20 15:14:40,474 cisTopic INFO Computing duplicate rate plot for 10x_pbmc 2023-06-20 15:14:43,563 cisTopic INFO Return plot data 2023-06-20 15:14:43,667 cisTopic INFO Computing insert size distribution for 10x_pbmc 2023-06-20 15:14:43,668 cisTopic INFO Counting fragments 2023-06-20 15:14:44,339 cisTopic INFO Returning plot data 2023-06-20 15:14:56,280 cisTopic INFO Computing TSS profile for 10x_pbmc 2023-06-20 15:14:57,866 cisTopic INFO Formatting annnotation 2023-06-20 15:14:57,914 cisTopic INFO Creating coverage matrix 2023-06-20 15:14:59,049 cisTopic INFO Coverage matrix done

/root/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pycisTopic/qc.py:564: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use newframe = frame.copy() TSS_matrix[x] = 0

For your convenience, this is the output of metadata_bc: metadata_bc {'10x_pbmc': Log_total_nr_frag Log_unique_nr_frag ... FRIP TSS_enrichment CAGCATGTCTACCTAT-1 5.026496 4.839472 ... 0.318152 0.0 TTAAAGGCATAACGGG-1 4.381169 4.185910 ... 0.583198 0.0 GGACCTCAGTTCCCAC-1 4.422655 4.219585 ... 0.617310 0.0 CCTATATTCCTCACAC-1 4.796311 4.617074 ... 0.222257 0.0 TGAAACTGTGACATAT-1 5.756169 5.584738 ... 0.188154 0.0 ... ... ... ... ... ... TGCACTTGTCTTGCAT-1 4.443967 4.245291 ... 0.597408 0.0 TCTACCTCATGACTAT-1 4.466808 4.278616 ... 0.586712 0.0 GCCAGGTTCGTCAAGT-1 4.485466 4.287802 ... 0.668351 0.0 ACAGGATCACCTGCCT-1 4.387212 4.187098 ... 0.710172 0.0 CGAATATGTTCATTTG-1 4.381602 4.172106 ... 0.651349 0.0

[207 rows x 10 columns]}

It seems TSS_enrichment is all 0.

Thirdly, here is all the packages in my scenicplus environment:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge accessible-pygments 0.0.4 pypi_0 pypi adjusttext 0.8 pypi_0 pypi aiohttp 3.8.4 pypi_0 pypi aiohttp-cors 0.7.0 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi alabaster 0.7.13 pypi_0 pypi anndata 0.9.1 pypi_0 pypi annoy 1.17.3 pypi_0 pypi ansiwrap 0.8.4 pypi_0 pypi appdirs 1.4.4 pypi_0 pypi arboreto 0.1.6 pypi_0 pypi asttokens 2.2.1 pypi_0 pypi async-timeout 4.0.2 pypi_0 pypi attr 0.3.2 pypi_0 pypi attrs 23.1.0 pypi_0 pypi babel 2.12.1 pypi_0 pypi backcall 0.2.0 pypi_0 pypi backports-zoneinfo 0.2.1 pypi_0 pypi bbknn 1.5.1 pypi_0 pypi beautifulsoup4 4.12.2 pypi_0 pypi bidict 0.22.1 pypi_0 pypi bioservices 1.11.2 pypi_0 pypi bleach 6.0.0 pypi_0 pypi blessed 1.20.0 pypi_0 pypi blosc2 2.0.0 pypi_0 pypi bokeh 3.1.1 pypi_0 pypi boltons 23.0.0 pypi_0 pypi bs4 0.0.1 pypi_0 pypi bzip2 1.0.8 h7f98852_4 conda-forge ca-certificates 2023.5.7 hbcca054_0 conda-forge cachetools 5.3.1 pypi_0 pypi cattrs 23.1.2 pypi_0 pypi certifi 2023.5.7 pypi_0 pypi cffi 1.15.1 pypi_0 pypi charset-normalizer 3.1.0 pypi_0 pypi click 8.1.3 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi colorama 0.4.6 pypi_0 pypi colorful 0.5.5 pypi_0 pypi colorlog 6.7.0 pypi_0 pypi comm 0.1.3 pypi_0 pypi contourpy 1.1.0 pypi_0 pypi cryptography 41.0.1 pypi_0 pypi ctxcore 0.2.0 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.35 pypi_0 pypi cytoolz 0.12.1 pypi_0 pypi dask 2023.5.0 pypi_0 pypi dataclasses-json 0.5.8 pypi_0 pypi debugpy 1.6.7 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi dill 0.3.6 pypi_0 pypi distlib 0.3.6 pypi_0 pypi distributed 2023.5.0 pypi_0 pypi docutils 0.18.1 pypi_0 pypi easydev 0.12.1 pypi_0 pypi entrypoints 0.4 pypi_0 pypi et-xmlfile 1.1.0 pypi_0 pypi exceptiongroup 1.1.1 pypi_0 pypi executing 1.2.0 pypi_0 pypi fastjsonschema 2.17.1 pypi_0 pypi fbpca 1.0 pypi_0 pypi filelock 3.12.2 pypi_0 pypi fonttools 4.40.0 pypi_0 pypi frozendict 2.3.8 pypi_0 pypi frozenlist 1.3.3 pypi_0 pypi fsspec 2023.6.0 pypi_0 pypi future 0.18.3 pypi_0 pypi gensim 4.3.1 pypi_0 pypi geosketch 1.2 pypi_0 pypi gevent 22.10.2 pypi_0 pypi globre 0.1.5 pypi_0 pypi google-api-core 2.11.1 pypi_0 pypi google-auth 2.20.0 pypi_0 pypi googleapis-common-protos 1.59.1 pypi_0 pypi gpustat 1.1 pypi_0 pypi greenlet 2.0.2 pypi_0 pypi grequests 0.7.0 pypi_0 pypi grpcio 1.51.3 pypi_0 pypi gseapy 0.10.8 pypi_0 pypi h5py 3.8.0 pypi_0 pypi harmonypy 0.0.9 pypi_0 pypi idna 3.4 pypi_0 pypi igraph 0.10.4 pypi_0 pypi imageio 2.31.1 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 6.7.0 pypi_0 pypi importlib-resources 5.12.0 pypi_0 pypi interlap 0.2.7 pypi_0 pypi intervaltree 3.1.0 pypi_0 pypi ipykernel 6.23.2 pypi_0 pypi ipympl 0.9.3 pypi_0 pypi ipython 8.12.2 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 8.0.6 pypi_0 pypi jedi 0.18.2 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi joblib 1.2.0 pypi_0 pypi jsonpickle 3.0.1 pypi_0 pypi jsonschema 4.17.3 pypi_0 pypi jupyter-client 8.2.0 pypi_0 pypi jupyter-core 5.3.1 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jupyterlab-widgets 3.0.7 pypi_0 pypi kaleido 0.2.1 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi lazy-loader 0.2 pypi_0 pypi ld_impl_linux-64 2.40 h41732ed_0 conda-forge lda 2.0.0 pypi_0 pypi leidenalg 0.9.1 pypi_0 pypi levenshtein 0.21.1 pypi_0 pypi libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.1.0 he5830b7_0 conda-forge libgomp 13.1.0 he5830b7_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libsqlite 3.42.0 h2797004_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libzlib 1.2.13 hd590300_5 conda-forge llvmlite 0.40.1rc1 pypi_0 pypi locket 1.0.0 pypi_0 pypi loompy 3.0.7 pypi_0 pypi loomxpy 0.4.2 pypi_0 pypi lxml 4.9.2 pypi_0 pypi lz4 4.3.2 pypi_0 pypi macs2 2.2.8 pypi_0 pypi markupsafe 2.1.3 pypi_0 pypi marshmallow 3.19.0 pypi_0 pypi marshmallow-enum 1.5.1 pypi_0 pypi matplotlib 3.7.1 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mistune 3.0.1 pypi_0 pypi mizani 0.9.2 pypi_0 pypi msgpack 1.0.5 pypi_0 pypi mudata 0.2.3 pypi_0 pypi multidict 6.0.4 pypi_0 pypi multiprocessing-on-dill 3.5.0a4 pypi_0 pypi mypy-extensions 1.0.0 pypi_0 pypi natsort 8.3.1 pypi_0 pypi nbclient 0.8.0 pypi_0 pypi nbconvert 7.6.0 pypi_0 pypi nbformat 5.9.0 pypi_0 pypi nbsphinx 0.9.2 pypi_0 pypi nbsphinx-link 1.3.0 pypi_0 pypi ncls 0.0.68 pypi_0 pypi ncurses 6.4 hcb278e6_0 conda-forge nest-asyncio 1.5.6 pypi_0 pypi networkx 3.1 pypi_0 pypi numba 0.57.0 pypi_0 pypi numexpr 2.8.4 pypi_0 pypi numpy 1.24.3 pypi_0 pypi numpy-groupies 0.9.22 pypi_0 pypi numpydoc 1.5.0 pypi_0 pypi nvidia-ml-py 11.525.112 pypi_0 pypi opencensus 0.11.2 pypi_0 pypi opencensus-context 0.1.3 pypi_0 pypi openpyxl 3.1.2 pypi_0 pypi openssl 3.1.1 hd590300_1 conda-forge packaging 23.1 pypi_0 pypi pandas 1.5.0 pypi_0 pypi pandoc 2.3 pypi_0 pypi pandocfilters 1.5.0 pypi_0 pypi papermill 2.4.0 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.4.0 pypi_0 pypi patsy 0.5.3 pypi_0 pypi pbr 3.1.1 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.5.0 pypi_0 pypi pip 23.1.2 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pypi_0 pypi platformdirs 3.6.0 pypi_0 pypi plotly 5.15.0 pypi_0 pypi plotnine 0.12.1 pypi_0 pypi plumbum 1.8.2 pypi_0 pypi ply 3.11 pypi_0 pypi polars 0.18.3 pypi_0 pypi progressbar2 4.2.0 pypi_0 pypi prometheus-client 0.17.0 pypi_0 pypi prompt-toolkit 3.0.38 pypi_0 pypi protobuf 4.23.3 pypi_0 pypi psutil 5.9.5 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi py-spy 0.3.14 pypi_0 pypi pyarrow 12.0.1 pypi_0 pypi pyasn1 0.5.0 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pybedtools 0.9.0 pypi_0 pypi pybigwig 0.3.22 pypi_0 pypi pybiomart 0.2.0 pypi_0 pypi pycistarget 1.0.3.dev1+g3fde1ce pypi_0 pypi pycistopic 1.0.3.dev16+g5de8c6c pypi_0 pypi pycparser 2.21 pypi_0 pypi pydantic 1.10.9 pypi_0 pypi pydata-sphinx-theme 0.13.3 pypi_0 pypi pyfasta 0.5.2 pypi_0 pypi pygam 0.9.0 pypi_0 pypi pygments 2.15.1 pypi_0 pypi pynndescent 0.5.10 pypi_0 pypi pyopenssl 23.2.0 pypi_0 pypi pyparsing 3.1.0 pypi_0 pypi pyranges 0.0.127 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi pysam 0.21.0 pypi_0 pypi pyscenic 0.12.1+5.gb61e5e9 pypi_0 pypi python 3.8.17 he550d4f_0_cpython conda-forge python-dateutil 2.8.2 pypi_0 pypi python-igraph 0.10.4 pypi_0 pypi python-levenshtein 0.21.1 pypi_0 pypi python-utils 3.6.1 pypi_0 pypi pytz 2023.3 pypi_0 pypi pyvis 0.3.2 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi pyzmq 25.1.0 pypi_0 pypi rapidfuzz 3.1.1 pypi_0 pypi ray 2.5.0 pypi_0 pypi readline 8.2 h8228510_1 conda-forge requests 2.31.0 pypi_0 pypi requests-cache 1.0.1 pypi_0 pypi rsa 4.9 pypi_0 pypi scanorama 1.7.3 pypi_0 pypi scanpy 1.9.3 pypi_0 pypi scenicplus 0.1.dev468+g26677cb.d20230620 pypi_0 pypi scikit-image 0.21.0 pypi_0 pypi scikit-learn 1.2.2 pypi_0 pypi scipy 1.10.1 pypi_0 pypi scrublet 0.2.3 pypi_0 pypi seaborn 0.12.2 pypi_0 pypi session-info 1.0.0 pypi_0 pypi setuptools 67.7.2 pyhd8ed1ab_0 conda-forge six 1.16.0 pypi_0 pypi smart-open 6.3.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi sorted-nearest 0.0.39 pypi_0 pypi sortedcontainers 2.4.0 pypi_0 pypi soupsieve 2.4.1 pypi_0 pypi sphinx 6.2.1 pypi_0 pypi sphinx-book-theme 1.0.1 pypi_0 pypi sphinx-rtd-theme 1.2.2 pypi_0 pypi sphinxcontrib-applehelp 1.0.4 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.1 pypi_0 pypi sphinxcontrib-jquery 4.1 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi stack-data 0.6.2 pypi_0 pypi statistics 1.0.3.5 pypi_0 pypi statsmodels 0.14.0 pypi_0 pypi stdlib-list 0.8.0 pypi_0 pypi suds-community 1.1.2 pypi_0 pypi tables 3.8.0 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tblib 1.7.0 pypi_0 pypi tenacity 8.2.2 pypi_0 pypi texttable 1.6.7 pypi_0 pypi textwrap3 0.9.2 pypi_0 pypi threadpoolctl 3.1.0 pypi_0 pypi tifffile 2023.4.12 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tk 8.6.12 h27826a3_0 conda-forge tmtoolkit 0.12.0 pypi_0 pypi toolz 0.12.0 pypi_0 pypi tornado 6.3.2 pypi_0 pypi tqdm 4.65.0 pypi_0 pypi traitlets 5.9.0 pypi_0 pypi tspex 0.6.2 pypi_0 pypi typing 3.7.4.3 pypi_0 pypi typing-extensions 4.6.3 pypi_0 pypi typing-inspect 0.9.0 pypi_0 pypi umap-learn 0.5.3 pypi_0 pypi url-normalize 1.4.3 pypi_0 pypi urllib3 1.26.16 pypi_0 pypi virtualenv 20.21.0 pypi_0 pypi wcwidth 0.2.6 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi wheel 0.40.0 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.7 pypi_0 pypi wrapt 1.15.0 pypi_0 pypi xlrd 2.0.1 pypi_0 pypi xmltodict 0.13.0 pypi_0 pypi xyzservices 2023.5.0 pypi_0 pypi xz 5.2.6 h166bdaf_0 conda-forge yarl 1.9.2 pypi_0 pypi zict 3.0.0 pypi_0 pypi zipp 3.15.0 pypi_0 pypi zope-event 4.6 pypi_0 pypi zope-interface 6.0 pypi_0 pypi

Version (please complete the following information):

Thanks! I'm looking forward to your reply!

Annie133 commented 1 year ago

Hi,

have you solved the problem, I met the same issue and have no clue how to fix it. As mentioned at #146, I down the pands version to 1.5, but still have the same problem. Could you please share your experience if you fixed it.

Thanks and best.

Qotov commented 11 months ago

Same issue here (with pandas 1.5)

SeppeDeWinter commented 11 months ago

Hi @Qotov and @Annie133

Can you show the head of annot and one of your fragments files please?

Best,

Seppe