KeyError: at plot_group_stats step

twoneu commented 2 years ago

Hi! Thanks again for a great package. I was running the code in the Quick Start tutorial and hit the following error when plotting the group level statistics:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/opt/anaconda3/lib/python3.9/site-packages/pandas/core/indexes/base.py:3621, in Index.get_loc(self, key, method, tolerance)
   3620 try:
-> 3621     return self._engine.get_loc(casted_key)
   3622 except KeyError as err:

File ~/opt/anaconda3/lib/python3.9/site-packages/pandas/_libs/index.pyx:136, in pandas._libs.index.IndexEngine.get_loc()

File ~/opt/anaconda3/lib/python3.9/site-packages/pandas/_libs/index.pyx:163, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:5198, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:5206, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'fdr_prop'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Input In [76], in <cell line: 18>()
      6 dict_df_stats
      8 dict_celltype_display_name = {
      9     "pyramidal_CA1": "Pyramidal CA1",
     10     "oligodendrocytes": "Oligodendrocyte",
   (...)
     15     "microglia": "Microglia",
     16 }
---> 18 scdrs.util.plot_group_stats(
     19     {
     20         trait: df_stats.rename(index=dict_celltype_display_name)
     21         for trait, df_stats in dict_df_stats.items()
     22     }
     23 )

File ~/opt/anaconda3/lib/python3.9/site-packages/scdrs/util.py:441, in plot_group_stats(dict_df_stats, df_fdr_prop, df_assoc_fdr, df_hetero_fdr)
    438 trait_list = list(dict_df_stats.keys())
    439 # compile df_fdr_prop, df_assoc_fdr, df_hetero_fdr from dict_df_stats
    440 df_fdr_prop = pd.concat(
--> 441     [dict_df_stats[trait]["fdr_prop"] for trait in trait_list], axis=1
    442 ).T
    443 df_assoc_fdr = pd.concat(
    444     [dict_df_stats[trait]["assoc_pval"] for trait in trait_list], axis=1
    445 ).T
    446 df_assoc_fdr = pd.DataFrame(
    447     multipletests(df_assoc_fdr.values.flatten(), method="fdr_bh")[1].reshape(
    448         df_assoc_fdr.shape
   (...)
    451     columns=df_assoc_fdr.columns,
    452 )

File ~/opt/anaconda3/lib/python3.9/site-packages/scdrs/util.py:441, in <listcomp>(.0)
    438 trait_list = list(dict_df_stats.keys())
    439 # compile df_fdr_prop, df_assoc_fdr, df_hetero_fdr from dict_df_stats
    440 df_fdr_prop = pd.concat(
--> 441     [dict_df_stats[trait]["fdr_prop"] for trait in trait_list], axis=1
    442 ).T
    443 df_assoc_fdr = pd.concat(
    444     [dict_df_stats[trait]["assoc_pval"] for trait in trait_list], axis=1
    445 ).T
    446 df_assoc_fdr = pd.DataFrame(
    447     multipletests(df_assoc_fdr.values.flatten(), method="fdr_bh")[1].reshape(
    448         df_assoc_fdr.shape
   (...)
    451     columns=df_assoc_fdr.columns,
    452 )

File ~/opt/anaconda3/lib/python3.9/site-packages/pandas/core/frame.py:3505, in DataFrame.__getitem__(self, key)
   3503 if self.columns.nlevels > 1:
   3504     return self._getitem_multilevel(key)
-> 3505 indexer = self.columns.get_loc(key)
   3506 if is_integer(indexer):
   3507     indexer = [indexer]

File ~/opt/anaconda3/lib/python3.9/site-packages/pandas/core/indexes/base.py:3623, in Index.get_loc(self, key, method, tolerance)
   3621     return self._engine.get_loc(casted_key)
   3622 except KeyError as err:
-> 3623     raise KeyError(key) from err
   3624 except TypeError:
   3625     # If we have a listlike key, _check_indexing_error will raise
   3626     #  InvalidIndexError. Otherwise we fall through and re-raise
   3627     #  the TypeError.
   3628     self._check_indexing_error(key)

KeyError: 'fdr_prop'`

I am able to generate df_fdr_prop using the following code snippet:

trait_list = list(dict_df_stats.keys())
df_fdr_prop = pd.concat(
            [
                dict_df_stats[trait]["n_fdr_0.1"] / dict_df_stats[trait]["n_cell"]
                for trait in trait_list
            ],
            axis=1,
        ).T

How should I proceed? Thank you!

KangchengHou commented 2 years ago

This should be due to some version inconsistency (we updated scdrs.util.plot_group_stats only very recently, sorry!)

Follows are options you can proceed:

update to the github master version:
```
git clone https://github.com/martinjzhang/scDRS.git
cd scDRS
pip install -e .
```
which will use the latest code https://github.com/martinjzhang/scDRS/blob/06e6df054295137bd8b686862234c7f9b4b64347/scdrs/util.py#L451
skip the plotting for now until we push another release (you should be able to run through other parts of the tutorial without calling this function)

vicscott commented 2 years ago

I met the same problem. So I would like to appreciate if there are some advice to plot like scdrs.util.plot_group_stats according to the scdrs_group.level1class file. And another question is that I don't known what happened to my scdrsgroup.level1class file that the nfdr* are all 0.

KangchengHou commented 2 years ago

hi @vicscott can you try

git clone https://github.com/martinjzhang/scDRS.git
cd scDRS
pip install -e .

to update to the latest version (we will soon push that to the pip). And can you replicate what you are seeing following https://martinjzhang.github.io/scDRS/notebooks/quickstart.html?

vicscott commented 2 years ago

My scrds version is v1.0.1, and it still report like above after I run these git and pip commands. By the way, thanks for your reply.

---Original--- From: "Kangcheng @.> Date: Fri, Sep 9, 2022 23:48 PM To: @.>; Cc: @.**@.>; Subject: Re: [martinjzhang/scDRS] KeyError: at plot_group_stats step (Issue #24)

hi @vicscott can you try git clone https://github.com/martinjzhang/scDRS.git cd scDRS pip install -e .
to update to the latest version (we will soon push that to the pip). And can you replicate what you are seeing following https://martinjzhang.github.io/scDRS/notebooks/quickstart.html?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

KangchengHou commented 2 years ago

that bug is fixed after v1.0.1 (and we will have a new version for that). For scdrs_group.level1class Could you confirm you are seeing different things than this?

vicscott commented 1 year ago

I found that the function called group_stats containing the argument "fdr_prop" was deprecated in method.py, but "fdr_prop" was still a requisite for the function called plot_group_stats.

---Original--- From: "Kangcheng @.> Date: Sat, Sep 10, 2022 00:08 AM To: @.>; Cc: @.**@.>; Subject: Re: [martinjzhang/scDRS] KeyError: at plot_group_stats step (Issue#24)

that bug is fixed after v1.0.1 (and we will have a new version for that). For scdrs_group.level1class Could you confirm you are seeing different things than this?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

KangchengHou commented 1 year ago

Hi please try pip install scdrs==1.0.2 and feel free to reopen if this bug pertains.

econnolly7 commented 1 year ago

Hi there. Awesome package. I just installed scdrs v1.0.2 and after running plot_group_stats() on the demo dataset from the quickstart guide I get the following error:

util.py:557: UserWarning: Tight layout not applied. The bottom and top margins cannot be made large enough to accommodate all axes decorations.

Let me know what I should do. Thanks so much!

martinjzhang commented 1 year ago

Hi,

This seems to be due to the figure size being too small. Having a larger figure should fix the problem, e.g., plt.figure(figsize=[width, height]) with large numbers for width and height.

I will wait for @KangchengHou for more detailed replies.

Best, Martin

KangchengHou commented 1 year ago

@econnolly7 yes it is due to plotting settings. I will add more options to the plotting functions and will reply further here.

But you should be able to proceed the rest of the tutorial (e.g., looking at the numerical results)

vicscott commented 1 year ago

I am sure that the version is v1.0.1,

vic @.***

------------------ 原始邮件 ------------------ 发件人: "martinjzhang/scDRS" @.>; 发送时间: 2022年9月10日(星期六) 凌晨0:08 @.>; @.**@.>; 主题: Re: [martinjzhang/scDRS] KeyError: at plot_group_stats step (Issue #24)

that bug is fixed after v1.0.1 (and we will have a new version for that). For scdrs_group.level1class Could you confirm you are seeing different things than this?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

KangchengHou commented 1 year ago

this should be addressed in f1b83e4. feel free to reopen

martinjzhang / scDRS

KeyError: at plot_group_stats step #24