search

PathwayMerger / PathMe

Harmonizing pathway databases using Biological Expression Language (BEL)
https://pathme.scai.fraunhofer.de/
Apache License 2.0
17 stars 2 forks source link

Error related to sqlalchemy in middle of BEL processing #28

Closed mtvector closed 4 years ago

mtvector commented 4 years ago

Hi there, thanks for the help previously. Unfortunately I made slight progress and found new errors. I noticed in your config that sqlalchemy is supposed to install the newest version, and that may be the source of the problem. Hopefully it's just another version thing. You mentioned previously that you're working on many large changes that will be released soon, if that's true, I'm willing to wait for that and don't worry about addressing these issues in specific here! Anyway, here's what I'm facing: After downloading wikipathways, running python3 -m pathme wikipathways bel

I get partway through and then:

Exporting WikiPathways to BEL in /wynton/home/ye/mschmitz1/.pathme/wikipathways/bel: 48%|████▊ | 242/509 [09:45<25:28, 5.72s/it]Traceback (most recent call last): File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context self.dialect.do_execute( File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute cursor.execute(statement, parameters) sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/__main__.py", line 8, in <module>
    main()
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/cli.py", line 75, in bel
    wikipathways_to_pickles(resource_files, resource_folder, hgnc_manager, export_folder)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/rdf_sparql.py", line 301, in wikipathways_to_pickles
    bel_graph = wikipathways_to_bel(pathway_path, hgnc_manager)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/rdf_sparql.py", line 275, in wikipathways_to_bel
    return rdf_wikipathways_to_bel(rdf_graph, hgnc_manager)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/rdf_sparql.py", line 264, in rdf_wikipathways_to_bel
    return convert_to_bel(nodes, complexes, interactions, metadata, hgnc_manager)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/convert_to_bel.py", line 55, in convert_to_bel
    nodes = {
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/convert_to_bel.py", line 56, in <dictcomp>
    node_id: node_to_bel(node, hgnc_manager, pathway_id)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/convert_to_bel.py", line 93, in node_to_bel
    namespace, name, identifier = get_valid_gene_identifier(node_ids_dict, hgnc_manager, pathway_id)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/utils.py", line 120, in get_valid_gene_identifier
    return _validate_query(hgnc_manager, hgnc_entry, hgnc_symbol, HGNC)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/utils.py", line 83, in _validate_query
    return _get_update_alias_symbol(hgnc_manager, original_identifier, HGNC)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/pathme/wikipathways/utils.py", line 59, in _get_update_alias_symbol
    query_result = hgnc_manager.get_hgnc_from_alias_symbol(original_identifier)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/bio2bel_hgnc/manager.py", line 232, in get_hgnc_from_alias_symbol
    query_result = self.session.query(AliasSymbol).filter(AliasSymbol.alias_symbol == alias_symbol).all()
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3244, in all
    return list(self)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3403, in __iter__
    return self._execute_and_instances(context)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3428, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 984, in execute
    return meth(self, multiparams, params)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_clauseelement
    ret = self._execute_context(
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1287, in _execute_context
    self._handle_dbapi_exception(
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _handle_dbapi_exception
    util.raise_(
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
    self.dialect.do_execute(
  File "/wynton/home/ye/mschmitz1/utils/miniconda3/envs/pathways2/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding parameter 0 - probably unsupported type.
[SQL: SELECT pyhgnc_aliassymbol.id AS pyhgnc_aliassymbol_id, pyhgnc_aliassymbol.alias_symbol AS pyhgnc_aliassymbol_alias_symbol, pyhgnc_aliassymbol.is_previous_symbol AS pyhgnc_aliassymbol_is_previous_symbol, pyhgnc_aliassymbol.hgnc_id AS pyhgnc_aliassymbol_hgnc_id 
FROM pyhgnc_aliassymbol 
WHERE pyhgnc_aliassymbol.alias_symbol = ?]
[parameters: ({'Survivin', 'survivin'},)]
(Background on this error at: http://sqlalche.me/e/rvf5)

And it runs into this error at the same spot every time...

Here are the packages that are installed

attrs                19.3.0             
bel-repository       0.1.0              
bel-resources        0.0.3              
bio2bel              0.2.1              
bio2bel-chebi        0.2.1              
bio2bel-hgnc         0.2.2              
bio2bel-kegg         0.2.5              
bio2bel-reactome     0.2.3              
bio2bel-wikipathways 0.2.3              
certifi              2019.11.28         
chardet              3.0.4              
Click                7.0                
click-plugins        1.1.1              
compath-utils        0.2.1              
configparser         4.0.2              
dataclasses-json     0.4.2              
decorator            4.4.2              
easy-config          0.2.0              
enum34               1.1.10             
flasgger             0.9.4              
Flask                1.1.1              
Flask-Admin          1.5.5              
Flask-Cors           3.0.8              
idna                 2.9                
ijson                2.6.1              
isodate              0.6.0              
itsdangerous         1.1.0              
Jinja2               2.11.1             
jsonschema           3.2.0              
MarkupSafe           1.1.1              
marshmallow          3.5.1              
marshmallow-enum     1.5.1              
mistune              0.8.4              
multisplitby         0.0.1              
mypy-extensions      0.4.3              
ndex2                3.3.1              
networkx             2.4                
numpy                1.16.3             
pandas               0.24.2             
passlib              1.7.2              
pathme               0.1.12             
pip                  20.0.2             
pybel                0.13.2             
pybel-cx             0.1.3              
pybel-tools          0.7.5              
PyHGNC               0.2.4              
PyMySQL              0.9.3              
pyparsing            2.4.6              
pyrsistent           0.16.0             
pysolr               3.8.1              
python-dateutil      2.8.1              
pytz                 2019.3             
PyYAML               5.3.1              
rdflib               4.2.2              
requests             2.23.0             
requests-file        1.4.3              
requests-toolbelt    0.9.1              
scipy                1.4.1              
setuptools           46.1.1.post20200323
six                  1.14.0             
SQLAlchemy           1.3.15             
stringcase           1.2.0              
tqdm                 4.31.1             
typing-extensions    3.7.4.1            
typing-inspect       0.5.0              
urllib3              1.25.8             
Werkzeug             1.0.0              
wheel                0.34.2             
WTForms              2.2.1
ddomingof commented 4 years ago

Ok this is a problem on our side with the new dump of wikipathways I updated. Apparently one of the nodes from WikiPathways has 2 names and we pass a set instead of a string to the query

ddomingof commented 4 years ago

While I fix the problem you can download the WikiPathways files here directly: https://drive.google.com/uc?authuser=0&id=1adwTE9cUpMb0lFiN0gbrGLaWfExtFdux&export=download Move them under ~/.pathme/wikipathways/bel if you want to use them with PathMe directly since it expects them to be there

ddomingof commented 4 years ago

The other dumps for KEGG and Reactome are here:

https://drive.google.com/uc?authuser=0&id=1a_6lck28Df9v2E-Dt45-ZARsdGVs-V9s&export=download

https://owncloud.scai.fraunhofer.de/index.php/s/EKkJmrXnGr2rwd6/download

mtvector commented 4 years ago

Sweet, thank you! I'll give these a shot.

ddomingof commented 4 years ago

I have pinned the version of compath_utils now so the previous problem should be solved (it wasnt pinned and we updated the package in last weeks). I identified this problem now. Gonna fix it

ddomingof commented 4 years ago

This pathway (https://www.wikipathways.org/index.php/Pathway:WP1772) had some issues regarding the identifiers (they were hgnc symbols instead of the expected hgnc id) so it was crashing. I dont want to hack it since this is a problem of the resource. I will do a new release fixing the dependency problem and where this pathway wont be exported. Will try to see whether it is fixed in new newest version from this month of WP, if not then I will curate it myself there but it wont be fixed until the next release.

mtvector commented 4 years ago

Good to know! The zips you linked to work, I can load them! Now I need to figure out how to wrangle the BELgraphs... Thanks for your help!

ddomingof commented 4 years ago

Thank to you for your patience and using PathMe. I make now the new release and close the issue.

ddomingof commented 4 years ago

Fixed with 8e1482ea577593f9c85f1085cbd3b9f72ef86e03