ValueError: Unable to determine gene collection

[tingxie2020]

I receive the error message,

ValueError: Unable to determine gene collection. Please make sure the input dataset specifies either HUGO gene symbols or Entrez gene ID's.

W1_1.var.head() gene_ids feature_types highly_variable means dispersions dispersions_norm n_cells mt rb n_cells_by_counts mean_counts pct_dropout_by_counts total_counts Mrpl15 ENSMUSG00000033845 Gene Expression False 0.523846 1.410226 -0.317858 364 False False 364 0.125276 76.815287 196.682587 Lypla1 ENSMUSG00000025903 Gene Expression False 0.496954 1.360324 -0.604688 356 False False 356 0.117622 77.324841 184.666626 Tcea1 ENSMUSG00000033813 Gene Expression False 1.178549 1.267944 -0.668772 814 False False 814 0.410102 48.152866 643.860168 Atp6v1h ENSMUSG00000033793 Gene Expression False 0.555859 1.411481 -0.310645 389 False False 389 0.138000 75.222930 216.659286 Rb1cc1 ENSMUSG00000025907 Gene Expression False 1.298682 1.432109 0.011839 838 False False 838 0.449731 46.624204 706.077637

the gene_ids should be Entrez gene ids? I change the column name gene_ids to Entrez gene IDs or Gene stable ID or Entrez gene ids, but didn't work. or what code should I use to map the gene ids to Ensembl BioMart (http://useast.ensembl.org/biomart) ?

Thanks Ting

[tingxie2020]

I used code: annot = sc.queries.biomart_annotations( "mmusculus", ["ensembl_gene_id", "Entrez_gene_ID","start_position", "end_position", "chromosome_name"], ).set_index("ensembl_gene_id") annot

but got error: KeyError Traceback (most recent call last) File ~/ENTER/lib/python3.9/site-packages/pybiomart/dataset.py:243, in Dataset.query(self, attributes, filters, only_unique, use_attr_names) 242 try: --> 243 attr = self.attributes[name] 244 self._add_attr_node(dataset, attr)

KeyError: 'Entrez_gene_ID'

During handling of the above exception, another exception occurred:

BiomartException Traceback (most recent call last) Input In [65], in <cell line: 1>() ----> 1 annot = sc.queries.biomart_annotations( 2 "mmusculus", 3 ["ensembl_gene_id", "Entrez_gene_ID","start_position", "end_position", "chromosome_name"], 4 ).set_index("ensembl_gene_id") 5 annot

File ~/ENTER/lib/python3.9/site-packages/scanpy/queries/_queries.py:108, in biomart_annotations(org, attrs, host, use_cache) 74 @_doc_params(doc_org=_doc_org, doc_host=_doc_host, doc_use_cache=_doc_use_cache) 75 def biomart_annotations( 76 org: str, (...) 80 use_cache: bool = False, 81 ) -> pd.DataFrame: 82 """\ 83 Retrieve gene annotations from ensembl biomart. 84 (...) 106 >>> adata.var[annot.columns] = annot 107 """ --> 108 return simple_query(org=org, attrs=attrs, host=host, use_cache=use_cache)

File ~/ENTER/lib/python3.9/site-packages/scanpy/queries/_queries.py:70, in simple_query(org, attrs, filters, host, use_cache) 66 server = Server(host, use_cache=use_cache) 67 dataset = server.marts["ENSEMBL_MART_ENSEMBL"].datasets[ 68 "{}_gene_ensembl".format(org) 69 ] ---> 70 res = dataset.query(attributes=attrs, filters=filters, use_attr_names=True) 71 return res

File ~/ENTER/lib/python3.9/site-packages/pybiomart/dataset.py:246, in Dataset.query(self, attributes, filters, only_unique, use_attr_names) 244 self._add_attr_node(dataset, attr) 245 except KeyError: --> 246 raise BiomartException( 247 'Unknown attribute {}, check dataset attributes ' 248 'for a list of valid attributes.'.format(name)) 250 if filters is not None: 251 # Add filter elements. 252 for name, value in filters.items():

BiomartException: Unknown attribute Entrez_gene_ID, check dataset attributes for a list of valid attributes.

I don't know how to check mmusculus dataset attributes for the entrez_gene_ids. Could anyone help?

This is for CellO cell type annotation which requires Entrez gene ids or HUGO gene symbols, could anyone help?

Thanks Ting

[tingxie2020]

I have used R biomaRt to find out the mmusculus dataset attributes. It is "entrezgene_id". So I got annot dataframe which contains entrez gene ids.

But now I encountered another issue when I used:

cello_data.var[annot.columns] = annot

for map the my anndata (cello_data2) gene ids to entrez gene ids and add the entrez gene ids. I got the error:

ValueError Traceback (most recent call last) Input In [84], in <cell line: 1>() ----> 1 cello_data2.var[annot.columns] = annot

File ~/ENTER/lib/python3.9/site-packages/pandas/core/frame.py:3643, in DataFrame.setitem(self, key, value) 3641 self._setitem_frame(key, value) 3642 elif isinstance(key, (Series, np.ndarray, list, Index)): -> 3643 self._setitem_array(key, value) 3644 elif isinstance(value, DataFrame): 3645 self._set_item_frame_value(key, value)

File ~/ENTER/lib/python3.9/site-packages/pandas/core/frame.py:3687, in DataFrame._setitem_array(self, key, value) 3685 check_key_length(self.columns, key, value) 3686 for k1, k2 in zip(key, value.columns): -> 3687 self[k1] = value[k2] 3689 elif not is_list_like(value): 3690 for col in key:

File ~/ENTER/lib/python3.9/site-packages/pandas/core/frame.py:3655, in DataFrame.setitem(self, key, value) 3652 self._setitem_array([key], value) 3653 else: 3654 # set column -> 3655 self._set_item(key, value)

File ~/ENTER/lib/python3.9/site-packages/pandas/core/frame.py:3832, in DataFrame._set_item(self, key, value) 3822 def _set_item(self, key, value) -> None: 3823 """ 3824 Add series to DataFrame in specified column. 3825 (...) 3830 ensure homogeneity. 3831 """ -> 3832 value = self._sanitize_column(value) 3834 if ( 3835 key in self.columns 3836 and value.ndim == 1 3837 and not is_extension_array_dtype(value) 3838 ): 3839 # broadcast across multiple columns if necessary 3840 if not self.columns.is_unique or isinstance(self.columns, MultiIndex):

File ~/ENTER/lib/python3.9/site-packages/pandas/core/frame.py:4532, in DataFrame._sanitize_column(self, value) 4530 # We should never get here with DataFrame value 4531 if isinstance(value, Series): -> 4532 return _reindex_for_setitem(value, self.index) 4534 if is_list_like(value): 4535 com.require_length_match(value, self.index)

File ~/ENTER/lib/python3.9/site-packages/pandas/core/frame.py:10999, in _reindex_for_setitem(value, index) 10995 except ValueError as err: 10996 # raised in MultiIndex.from_tuples, see test_insert_error_msmgs 10997 if not value.index.is_unique: 10998 # duplicate axis

10999 raise err 11001 raise TypeError( 11002 "incompatible index of inserted column with frame index" 11003 ) from err 11004 return reindexed_value

File ~/ENTER/lib/python3.9/site-packages/pandas/core/frame.py:10994, in _reindex_for_setitem(value, index) 10992 # GH#4107 10993 try:

10994 reindexed_value = value.reindex(index)._values 10995 except ValueError as err: 10996 # raised in MultiIndex.from_tuples, see test_insert_error_msmgs 10997 if not value.index.is_unique: 10998 # duplicate axis

File ~/ENTER/lib/python3.9/site-packages/pandas/core/series.py:4672, in Series.reindex(self, *args, kwargs) 4668 raise TypeError( 4669 "'index' passed as both positional and keyword argument" 4670 ) 4671 kwargs.update({"index": index}) -> 4672 return super().reindex(kwargs)

File ~/ENTER/lib/python3.9/site-packages/pandas/core/generic.py:4966, in NDFrame.reindex(self, *args, **kwargs) 4963 return self._reindex_multi(axes, copy, fill_value) 4965 # perform the reindex on the axes -> 4966 return self._reindex_axes( 4967 axes, level, limit, tolerance, method, fill_value, copy 4968 ).finalize(self, method="reindex")

File ~/ENTER/lib/python3.9/site-packages/pandas/core/generic.py:4986, in NDFrame._reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy) 4981 new_index, indexer = ax.reindex( 4982 labels, level=level, limit=limit, tolerance=tolerance, method=method 4983 ) 4985 axis = self._get_axis_number(a) -> 4986 obj = obj._reindex_with_indexers( 4987 {axis: [new_index, indexer]}, 4988 fill_value=fill_value, 4989 copy=copy, 4990 allow_dups=False, 4991 ) 4992 # If we've made a copy once, no need to make another one 4993 copy = False

File ~/ENTER/lib/python3.9/site-packages/pandas/core/generic.py:5032, in NDFrame._reindex_with_indexers(self, reindexers, fill_value, copy, allow_dups) 5029 indexer = ensure_platform_int(indexer) 5031 # TODO: speed up on homogeneous DataFrame objects (see _reindex_multi) -> 5032 new_data = new_data.reindex_indexer( 5033 index, 5034 indexer, 5035 axis=baxis, 5036 fill_value=fill_value, 5037 allow_dups=allow_dups, 5038 copy=copy, 5039 ) 5040 # If we've made a copy once, no need to make another one 5041 copy = False

File ~/ENTER/lib/python3.9/site-packages/pandas/core/internals/managers.py:679, in BaseBlockManager.reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups, copy, consolidate, only_slice, use_na_proxy) 677 # some axes don't allow reindexing with dups 678 if not allow_dups: --> 679 self.axes[axis]._validate_can_reindex(indexer) 681 if axis >= self.ndim: 682 raise IndexError("Requested axis not found in manager")

File ~/ENTER/lib/python3.9/site-packages/pandas/core/indexes/base.py:4107, in Index._validate_can_reindex(self, indexer) 4105 # trying to reindex on an axis with duplicates 4106 if not self._index_as_unique and len(indexer): -> 4107 raise ValueError("cannot reindex on an axis with duplicate labels")

ValueError: cannot reindex on an axis with duplicate labels

so, how to solve this?

[tingxie2020]

When I used the code: annot = sc.queries.biomart_annotations( "mmusculus", ["hgnc_symbol", "entrezgene_id"], ) annot

it extract the hgnc_symbol and entrezgene_id, then I set the index: annot2=annot annot2.set_index("hgnc_symbol")

this time it let me run the code successfully. cello_data3.var[annot2.columns] = annot2

It didn't shown the duplicate index error. then I rename the entrezgene_id with Entrez gene ID

but when I run the CellO code: cello_resource_loc = "/opt/test_cello" model_prefix = "cello_data3" # <-- The trained model will be stored in a file called GSM3516666_LX682_NORMAL.model.dill

cello.scanpy_cello( cello_data3, 'leiden', cello_resource_loc, out_prefix=model_prefix ) It still shows the error: ValueError: Unable to determine gene collection. Please make sure the input dataset specifies either HUGO gene symbols or Entrez gene ID's.

I don't know what to do now. Any suggestions?

[tingxie2020]

by the way, right now the cello_data3 head looks like: cello_data3.var.head()

n_cells highly_variable means   dispersions dispersions_norm    highly_variable_nbatches    highly_variable_intersection    hgnc_symbol Entrez gene ID

Xkr4 4 False 0.000200 -0.470874 -1.075329 0 False NaN NaN Gm19938 12 False 0.000936 0.114140 -0.173123 0 False NaN NaN Rp1 31 False 0.004254 0.368517 0.485255 1 False NaN NaN Sox17 8 False 0.000919 0.401739 0.376563 0 False NaN NaN Mrpl15 2373 False 0.209986 0.241276 0.035849 0 False NaN NaN

somehow some genes don't have entrezgene ids.