Pvalues_results value error

[kevinmgao]

Hi, I was trying to run cellphoneDB on single cell RNAseq of 113 cells. The process broke during the p value calculation, please see the error message below. Would appreciate any input/help!

As there is no R environment set up, some functionalities will be disabled, e.g. plot [ ][APP][29/04/21-21:13:47][WARNING] Latest local available version is v2.0.0, using it [ ][APP][29/04/21-21:13:47][WARNING] User selected downloaded database v2.0.0 is available, using it [ ][CORE][29/04/21-21:13:47][INFO] Initializing SqlAlchemy CellPhoneDB Core [ ][CORE][29/04/21-21:13:47][INFO] Using custom database at /home/kg69w/.cpdb/releases/v2.0.0/cellphone.db [ ][APP][29/04/21-21:13:47][INFO] Launching Method cpdb_statistical_analysis_local_method_launcher [ ][APP][29/04/21-21:13:47][INFO] Launching Method _set_paths [ ][APP][29/04/21-21:13:47][INFO] Launching Method _load_meta_counts [ ][CORE][29/04/21-21:13:48][INFO] Launching Method cpdb_statistical_analysis_launcher [ ][CORE][29/04/21-21:13:48][INFO] Launching Method _counts_validations [ ][CORE][29/04/21-21:13:48][INFO] [Cluster Statistical Analysis] Threshold:0.1 Iterations:1000 Debug-seed:-1 Threads:4 Precision:3 [ ][CORE][29/04/21-21:14:02][INFO] Running Real Analysis [ ][CORE][29/04/21-21:14:31][INFO] Running Statistical Analysis /share/pkg/python3/3.6_intel_pu3_2018.3.039/intelpython3/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters [ ][CORE][29/04/21-22:34:43][INFO] Building Pvalues result [ ][CORE][29/04/21-22:36:20][INFO] Building results [ ][APP][29/04/21-22:36:24][ERROR] Unexpected error Traceback (most recent call last): File "/home/kg69w/cpdb/lib/python3.6/site-packages/cellphonedb/src/api_endpoints/terminal_api/method_terminal_api_endpoints/method_terminal_commands.py", line 144, in statistical_analysis subsampler, File "/home/kg69w/cpdb/lib/python3.6/site-packages/cellphonedb/src/local_launchers/local_method_launcher.py", line 64, in cpdb_statistical_analysis_local_method_launcher subsampler File "/home/kg69w/cpdb/lib/python3.6/site-packages/cellphonedb/src/core/methods/method_launcher.py", line 76, in cpdb_statistical_analysis_launcher self.separator) File "/home/kg69w/cpdb/lib/python3.6/site-packages/cellphonedb/src/core/methods/cpdb_statistical_analysis_method.py", line 36, in call result_precision, File "/home/kg69w/cpdb/lib/python3.6/site-packages/cellphonedb/src/core/methods/cpdb_statistical_analysis_complex_method.py", line 119, in call counts_data File "/home/kg69w/cpdb/lib/python3.6/site-packages/cellphonedb/src/core/methods/cpdb_statistical_analysis_complex_method.py", line 201, in build_results pvalues_result = pd.concat([interactions_data_result, result_percent], axis=1, join='inner', sort=False) File "/share/pkg/python3/3.6_intel_pu3_2018.3.039/intelpython3/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 287, in concat return op.get_result() File "/share/pkg/python3/3.6_intel_pu3_2018.3.039/intelpython3/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 503, in get_result mgrs_indexers, self.new_axes, concat_axis=self.bm_axis, copy=self.copy, File "/share/pkg/python3/3.6_intel_pu3_2018.3.039/intelpython3/lib/python3.6/site-packages/pandas/core/internals/concat.py", line 84, in concatenate_block_managers return BlockManager(blocks, axes) File "/share/pkg/python3/3.6_intel_pu3_2018.3.039/intelpython3/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 149, in init self._verify_integrity() File "/share/pkg/python3/3.6_intel_pu3_2018.3.039/intelpython3/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 329, in _verify_integrity raise construction_error(tot_items, block.shape[1:], self.axes) ValueError: Shape of passed values is (284, 300), indices imply (283, 300)

Best, -Kevin

[prete]

Hi @kevinmgao what's your version of CellPhoneDB and pandas? (pip show cellphonedb pandas)

[kevinmgao]

hi @prete , here is my version information:

(cpdb) [kg69w@ghpcc06 20210429_cellphoneDB]$ pip show cellphonedb pandas Name: CellPhoneDB Version: 2.1.7 Summary: Inferring cell-cell communication Home-page: https://cellphonedb.org Author: TeichLab Author-email: contact@cellphonedb.org License: MIT Location: /home/kg69w/.local/lib/python3.6/site-packages Requires: flask, PyYAML, click, pika, geosketch, tqdm, h5py, numpy, Flask-RESTful, rpy2, pandas, requests, anndata, boto3, Flask-Testing, scikit-learn, SQLAlchemy Required-by:

Name: pandas Version: 1.1.5 Summary: Powerful data structures for data analysis, time series, and statistics Home-page: https://pandas.pydata.org Author: None Author-email: None License: BSD Location: /share/pkg/python3/3.6_intel_pu3_2018.3.039/intelpython3/lib/python3.6/site-packages Requires: python-dateutil, numpy, pytz Required-by: statsmodels, CellPhoneDB, anndata

[prete]

Your pandas version (1.1.5) doesn't match the requirements. Could you please downgrade to pandas 1.1.4 (pip install -U pandas==1.1.4) and then try again?

[prete]

Hi @kevinmgao did you manage to run it with the downgraded version of pandas?

[kevinmgao]

Hi @prete , apologies for the late reply,

I was able to successfully run most of my analysis. Interestingly, this specific error only came up on 2 out of 28 of my datasets, which turned out to include the samples with the smallest cell counts. I did run your test example data, and that ran with no issues. Not sure what exactly is weird with those 2 datasets.

I did attempt to change the version of pandas as you suggested, but I am new to using my institution's HPCC and wasn't able to successfully downgrade pandas.

I will look into contacting my HPCC admins to see if they can help with the pandas downgrade, otherwise would it help if I provided the files that had an issue with cellphonedb?

Thanks for your help!

[prete]

As far as your HPC env goes, I noticed that pandas is stored in a shared path (/share/pkg/python3/...). You want to install your own version in your environment. For that first activate your (cpdb) env and then install pandas using pip install --user -U pandas==1.1.4, the "--user" option will install the pacakge to your user directory (~/.local/)... that should be enough I guess.