Closed jpfeil closed 2 weeks ago
Yes, I also encounter this error when I run the mouse model, it seems to be the error of converting mouse gene to human gene.
Hi @jpfeil and @nierq,
Indeed there seems to be a server side issue with the conversion. @deeenes could you take a look please?
hello, I also encounter the same error。
>>> import pycurl
>>> import decoupler as dc
>>>
>>>
>>> dc.get_progeny(organism = 'Mus musculus',top =100)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/public/home/niuxy/Software/miniconda3/envs/py39/lib/python3.9/site-packages/decoupler/omnip.py", line 139, in get_progeny
p = translate_net(
File "/public/home/niuxy/Software/miniconda3/envs/py39/lib/python3.9/site-packages/decoupler/omnip.py", line 432, in translate_net
_target_organism = taxonomy.ensure_ncbi_tax_id(target_organism)
File "/public/home/niuxy/Software/miniconda3/envs/py39/lib/python3.9/site-packages/pypath/utils/taxonomy.py", line 367, in ensure_ncbi_tax_id
taxid_from_common_name(taxon_id) or
File "/public/home/niuxy/Software/miniconda3/envs/py39/lib/python3.9/site-packages/pypath/utils/taxonomy.py", line 283, in taxid_from_common_name
common_to_ncbi = get_db('common')
File "/public/home/niuxy/Software/miniconda3/envs/py39/lib/python3.9/site-packages/pypath/utils/taxonomy.py", line 435, in get_db
init_db(key)
File "/public/home/niuxy/Software/miniconda3/envs/py39/lib/python3.9/site-packages/pypath/utils/taxonomy.py", line 450, in init_db
ncbi_data = uniprot_input.uniprot_ncbi_taxids_2()
File "/public/home/niuxy/Software/miniconda3/envs/py39/lib/python3.9/site-packages/pypath/inputs/uniprot.py", line 1301, in uniprot_ncbi_taxids_2
for line in c.result:
TypeError: 'NoneType' object is not iterable
I now have an alternative solution to this problem. step 1: Using R packet progeny to obtain model data used by decoupler
library(progeny) model_data <- progeny::model_mouse_full
write.table(model_data ,'./progeny_mouse_geneset.txt')`
step2 In python gene_set = pd.read_csv("./progeny_mouse_geneset.txt",sep='\t') gene_set.rename(columns = {'gene':'target','pathway':'source','p.value':'p_value'},inplace = True) gene_set = gene_set.reindex(columns=['source','target','weight','p_value'])
gene_set = ( gene_set. sort_values('p_value'). groupby('source'). head(200). sort_values(['source', 'p_value']). reset_index() )
dc.run_mlm(mat=raw_data, net=gene_set, source='source', target='target', weight='weight', verbose=True, use_raw=False)
Or you can do progeny analysis directly in R:
library(Seurat) library(anndata) library(progeny) data <- read_h5ad('./all_adata_MC38.integrated_final_rawdata.h5ad') data_seurat <- CreateSeuratObject(counts = t(as.matrix(data$X)), meta.data = data$obs) #Standardized data is required For specific steps, please refer to the introduction of progeny:https://saezlab.github.io/progeny/articles/ProgenySingleCell.html
@xiangrong7, yours is not the same error, it happens on a completely different line. The error that you encountered is likely due to a random network failure, your computer failed to download a file. You can get rid of the error by trying again on a good quality network:
from pypath.share import curl
import decoupler as dc
with curl.cache_off():
dc.get_progeny(organism = 'Mus musculus', top = 100)
Hello @jpfeil,
Apologies about the slow answer. I have to correct my previous comment, actually your issue is very similar to @xiangrong7's, it is a transient networking error. The traceback is long, but the most downstream element that we can identify is the call at pypath/utils/mapping.py:2599
: it wants to translate secondary UniProt IDs to primary ones. Then normally a download happens which looks like this in the log:
116 [2023-01-18 15:12:04] [mapping] Requested to load ID translation table from `uniprot-sec` to `uniprot-pri`, organism: 9606.
117 [2023-01-18 15:12:04] [mapping] Chosen built-in defined ID translation table: resource=basic, id_type_a=uniprot-sec, id_type_b=uniprot-pri
118 [2023-01-18 15:12:04] [inputs] Selecting input method (step 1): module `uniprot.get_uniprot_sec`, method `None`.
119 [2023-01-18 15:12:04] [inputs] Selecting input method (step 2): module `pypath.inputs.uniprot`, method `get_uniprot_sec`.
120 [2023-01-18 15:12:04] [inputs] Importing module `pypath.inputs.uniprot`.
121 [2023-01-18 15:12:04] [mapping] Loading mapping table for organism `9606` with identifiers `uniprot-sec` and `uniprot-pri`, input type `file`
122 [2023-01-18 15:12:04] [mapping] Reader created for ID translation table, parameters: `ncbi_tax_id=9606, id_a=uniprot-sec, id_b=uniprot-pri, load_a_to_b=1, load_b_to_a=0, input_type=file (FileMapping)`.
123 [2023-01-18 15:12:04] [inputs] Selecting input method (step 1): module `uniprot.get_uniprot_sec`, method `None`.
124 [2023-01-18 15:12:04] [inputs] Selecting input method (step 2): module `pypath.inputs.uniprot`, method `get_uniprot_sec`.
125 [2023-01-18 15:12:04] [inputs] Importing module `pypath.inputs.uniprot`.
126 [2023-01-18 15:12:04] [uniprot_input] Loading list of all UniProt IDs for organism `9606` (only SwissProt: None).
127 [2023-01-18 15:12:04] [curl] Creating Curl object to retrieve data from `https://legacy.uniprot.org/uniprot/`
128 [2023-01-18 15:12:04] [curl] GET parameters added to the URL: `query=organism%3A9606&format=tab&columns=id`
129 [2023-01-18 15:12:04] [curl] Cache file path: `/home/denes/pypath/nocache/cache/590b20597cb4f25fe88d504e4dc430df-`
130 [2023-01-18 15:12:04] [curl] Setting up and calling pycurl.
131 [2023-01-18 15:12:25] [curl] Opening file `/home/denes/pypath/nocache/cache/590b20597cb4f25fe88d504e4dc430df-`
132 [2023-01-18 15:12:25] [curl] Extracting data from file type `plain`
133 [2023-01-18 15:12:25] [curl] Opening plain text file `/home/denes/pypath/nocache/cache/590b20597cb4f25fe88d504e4dc430df-`.
134 [2023-01-18 15:12:25] [curl] Contents of `/home/denes/pypath/nocache/cache/590b20597cb4f25fe88d504e4dc430df-` has been read and the file has been closed.
135 [2023-01-18 15:12:25] [curl] File at `https://legacy.uniprot.org/uniprot/?query=organism%3A9606&format=tab&columns=id` successfully retrieved. Resulted file type `plain text, unicode string`. Local file at
136 `/home/denes/pypath/nocache/cache/590b20597cb4f25fe88d504e4dc430df-`.
137 [2023-01-18 15:12:25] [curl] Creating Curl object to retrieve data from `ftp://ftp.expasy.org/databases/uniprot/current_release/knowledgebase/complete/docs/sec_ac.txt`
138 [2023-01-18 15:12:25] [curl] Cache file path: `/home/denes/pypath/nocache/cache/7814fe9dc734379a8c28d4b1478d2f85-sec_ac.txt`
139 [2023-01-18 15:12:25] [curl] Setting up and calling pycurl.
140 [2023-01-18 15:12:27] [curl] Opening file `/home/denes/pypath/nocache/cache/7814fe9dc734379a8c28d4b1478d2f85-sec_ac.txt`
141 [2023-01-18 15:12:27] [curl] Extracting data from file type `plain`
142 [2023-01-18 15:12:27] [curl] Opening plain text file `/home/denes/pypath/nocache/cache/7814fe9dc734379a8c28d4b1478d2f85-sec_ac.txt`.
143 [2023-01-18 15:12:27] [curl] File at `ftp://ftp.expasy.org/databases/uniprot/current_release/knowledgebase/complete/docs/sec_ac.txt` successfully retrieved. Resulted file type `plain text, file object`. Local file at
144 `/home/denes/pypath/nocache/cache/7814fe9dc734379a8c28d4b1478d2f85-sec_ac.txt`.
The solution is similar to the one suggested above: try again this particular download with the cache turned off:
from pypath.utils import mapping
from pypath.share import curl
with curl.cache_off():
mapping.get_mapper().primary_uniprot(['P00533'])
I hope this helps, please let me know if you're still experiencing any issue.
Best,
Denes
I used the method you mentioned above.I don't know if other people's problems have been solved, but my problems still exist.
<class 'TypeError'>, 'NoneType' object is not iterable, <traceback object at 0x2b65030c4040> Traceback (most recent call last): File "
", line 1, in File "/work/home/nierq01/anaconda3/envs/progeny/lib/python3.10/site-packages/decoupler/omnip.py", line 139, in get_progeny p = translate_net( File "/work/home/nierq01/anaconda3/envs/progeny/lib/python3.10/site-packages/decoupler/omnip.py", line 432, in translate_net _target_organism = taxonomy.ensure_ncbi_tax_id(target_organism) File "/work/home/nierq01/anaconda3/envs/progeny/lib/python3.10/site-packages/pypath/utils/taxonomy.py", line 367, in ensure_ncbi_tax_id taxid_from_common_name(taxon_id) or File "/work/home/nierq01/anaconda3/envs/progeny/lib/python3.10/site-packages/pypath/utils/taxonomy.py", line 283, in taxid_from_common_name common_to_ncbi = get_db('common') File "/work/home/nierq01/anaconda3/envs/progeny/lib/python3.10/site-packages/pypath/utils/taxonomy.py", line 435, in get_db init_db(key) File "/work/home/nierq01/anaconda3/envs/progeny/lib/python3.10/site-packages/pypath/utils/taxonomy.py", line 450, in init_db ncbi_data = uniprot_input.uniprot_ncbi_taxids_2() File "/work/home/nierq01/anaconda3/envs/progeny/lib/python3.10/site-packages/pypath/inputs/uniprot.py", line 1301, in uniprot_ncbi_taxids_2 for line in c.result: TypeError: 'NoneType' object is not iterable
Hi @nierq, I'm sorry to hear this error is still with you. I've just run the above function successfully, suggesting that this is a download issue similar to the others above. These kind of errors are not uncommon, but having them so often suggests a really bad connection or some other issue. To know more we need curl debug logs:
from pypath.share import curl
from pypath.inputs import uniprot
import pypath
with curl.cache_off(), curl.debug_on():
taxons = uniprot.uniprot_ncbi_taxids_2()
pypath.log()
If the download is not successful, the above log could give useful information. This file is not too large (2.3 MB), repeated failure of downloading it suggests some fundamental issue. Can you download it from the CLI?
curl -LOvvv https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/docs/speclist.txt
Can you open the URL above in your browser?
I also encountered a problem:
{
"name": "TypeError",
"message": "object of type 'NoneType' has no len()",
"stack": "---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[32], line 1
----> 1 msigdb = dc.get_resource('MSigDB', organism = \"mouse\")
2 msigdb
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/decoupler/omnip.py:360, in get_resource(name, organism, genesymbol_resource, **kwargs)
358 df.columns = list(df.columns)
359 df = df.reset_index()
--> 360 df = _annotation_identifiers(df, organism, genesymbol_resource)
361 df = df.drop(columns=['record_id', 'uniprot', 'entity_type', 'source'])
362 df = op._misc.dtypes.auto_dtype(df)
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/decoupler/omnip.py:919, in _annotation_identifiers(net, organism, genesymbol_resource)
906 def _annotation_identifiers(
907 net: pd.DataFrame,
908 organism: str | int,
(...)
914 ) = None,
915 ) -> pd.DataFrame:
917 if not _is_human(organism):
--> 919 net = translate_net(
920 net,
921 columns='uniprot',
922 id_type='uniprot',
923 source_organism=9606,
924 target_organism=organism,
925 )
927 if genesymbol_resource is False:
929 net['genesymbol'] = net['uniprot']
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/decoupler/omnip.py:732, in translate_net(net, columns, source_organism, target_organism, id_type, unique_by, **kwargs)
729 return net
731 _check_if_pypath()
--> 732 from pypath.utils import orthology
733 from pypath.share import common
734 from pypath.utils import taxonomy
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/pypath/utils/__init__.py:20
1 #!/usr/bin/env python
2 # -*- coding: utf-8 -*-
3
(...)
17 # Website: https://pypath.omnipathdb.org/
18 #
---> 20 from . import orthology as homology
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/pypath/utils/orthology.py:45
42 import timeloop
43 import pandas as pd
---> 45 import pypath.utils.mapping as mapping
46 import pypath.share.common as common
47 import pypath_common._constants as _const
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/pypath/utils/mapping.py:69
67 import pypath.share.curl as curl
68 import pypath.inputs as inputs
---> 69 import pypath.inputs.uniprot as uniprot_input
70 import pypath.inputs.uniprot_db as uniprot_db
71 import pypath.inputs.pro as pro_input
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/pypath/inputs/uniprot.py:41
39 import pypath.share.common as common
40 import pypath_common._constants as _const
---> 41 import pypath.utils.taxonomy as taxonomy
42 from pypath.inputs.uniprot_idmapping import idtypes as idmapping_idtypes
44 _logger = session_mod.Logger(name = 'uniprot_input')
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/pypath/utils/taxonomy.py:85
46 # XXX: Shouldn't we keep all functions and variables separated
47 # (together among them)?
48 taxids = {
49 9606: 'human',
50 10090: 'mouse',
(...)
77 9544: 'rhesus macaque',
78 }
80 taxids2 = dict(
81 (
82 t.taxon_id,
83 t.common_name.lower()
84 )
---> 85 for t in ensembl_input.ensembl_organisms()
86 )
88 taxa = common.swap_dict_simple(taxids)
89 taxa2 = common.swap_dict_simple(taxids2)
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/pypath/inputs/ensembl.py:52, in ensembl_organisms()
49 with warnings.catch_warnings():
51 warnings.simplefilter('ignore', bs4.XMLParsedAsHTMLWarning)
---> 52 soup = bs4.BeautifulSoup(c.result, 'html.parser')
54 for r in soup.find('table').find_all('tr'):
56 if not record:
File ~/miniconda3/envs/decoupler/lib/python3.12/site-packages/bs4/__init__.py:315, in BeautifulSoup.__init__(self, markup, features, builder, parse_only, from_encoding, exclude_encodings, element_classes, **kwargs)
313 if hasattr(markup, 'read'): # It's a file-type object.
314 markup = markup.read()
--> 315 elif len(markup) <= 256 and (
316 (isinstance(markup, bytes) and not b'<' in markup)
317 or (isinstance(markup, str) and not '<' in markup)
318 ):
319 # Issue warnings for a couple beginner problems
320 # involving passing non-markup to Beautiful Soup.
321 # Beautiful Soup will still parse the input as markup,
322 # since that is sometimes the intended behavior.
323 if not self._markup_is_url(markup):
324 self._markup_resembles_filename(markup)
TypeError: object of type 'NoneType' has no len()"
}
This latter is a download failure, which is either accidental, or is specific to your network, system or environment.
Describe the bug Tried downloading the progeny mouse model but I get a TypeError: Nonetype is not iterable. Human works fine.
To Reproduce
Expected behavior Expected to download the mouse marker genes.
System
Additional context