search

Knowledge-Graph-Hub / kg-covid-19

An instance of KG Hub to produce a knowledge graph for COVID-19 response.
https://github.com/Knowledge-Graph-Hub/kg-covid-19/wiki
BSD 3-Clause "New" or "Revised" License
78 stars 26 forks source link

migrate to main KGX branch #381

Closed justaddcoffee closed 3 years ago

justaddcoffee commented 3 years ago

Right now we are pinned to Deepak's fork of KGX (see requirements.txt).

It'd be good to switch to the main KGX branch here

Deepak and I made this PR - this should fix this ticket. We are testing this now.

justaddcoffee commented 3 years ago

@deepakunni3, getting close, but still seeing an error in the merge step:

18:48:57  + . venv/bin/activate
18:48:57  + deactivate nondestructive
18:48:57  + [ -n  ]
18:48:57  + [ -n  ]
18:48:57  + [ -n  -o -n  ]
18:48:57  + [ -n  ]
18:48:57  + unset VIRTUAL_ENV
18:48:57  + [ ! nondestructive = nondestructive ]
18:48:57  + VIRTUAL_ENV=/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv
18:48:57  + export VIRTUAL_ENV
18:48:57  + _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
18:48:57  + PATH=/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
18:48:57  + export PATH
18:48:57  + [ -n  ]
18:48:57  + [ -z  ]
18:48:57  + _OLD_VIRTUAL_PS1=$ 
18:48:57  + [ x(venv)  != x ]
18:48:57  + PS1=(venv) $ 
18:48:57  + export PS1
18:48:57  + [ -n  -o -n  ]
18:48:57  + python3.7 run.py merge -y merge_jenkins.yaml
18:48:59  /var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/biolinkml/__init__.py:158: UserWarning: Some URL processing will fail with python 3.7.5 or earlier.  Current version: sys.version_info(major=3, minor=7, micro=5, releaselevel='final', serial=0)
18:48:59    warn(f"Some URL processing will fail with python 3.7.5 or earlier.  Current version: {sys.version_info}")
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'drug-central'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'pharmgkb'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'STRING'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'ttd'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'zhou-host-proteins'
18:48:59  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'drug-central'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'SciBite-CORD-19'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'sars-cov-2-gene-annot'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'intact'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'chembl'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'gene-ontology'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'mondo-ontology'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'chebi'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'hp-ontology'
18:48:59  [KGX][cli_utils.py][               merge] INFO: Spawning process for 'go-cams'
18:48:59  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'pharmgkb'
18:49:01  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'STRING'
19:11:22  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'ttd'
19:11:22  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'zhou-host-proteins'
19:11:22  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'SciBite-CORD-19'
19:24:43  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'sars-cov-2-gene-annot'
19:24:43  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'intact'
19:24:43  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'chembl'
19:24:43  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'gene-ontology'
19:24:53  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'mondo-ontology'
19:24:58  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'chebi'
19:25:20  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'hp-ontology'
19:25:21  [KGX][cli_utils.py][        parse_source] INFO: Processing source 'go-cams'
19:25:22  Traceback (most recent call last):
19:25:22    File "run.py", line 165, in <module>
19:25:22      cli()
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
19:25:22      return self.main(*args, **kwargs)
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
19:25:22      rv = self.invoke(ctx)
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
19:25:22      return _process_result(sub_ctx.command.invoke(sub_ctx))
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
19:25:22      return ctx.invoke(self.callback, **ctx.params)
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
19:25:22      return callback(*args, **kwargs)
19:25:22    File "run.py", line 84, in merge
19:25:22      load_and_merge(yaml, processes)
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/kg_covid_19/merge_utils/merge_kg.py", line 36, in load_and_merge
19:25:22      merged_graph = merge(yaml_file, processes=processes)
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/kgx/cli/cli_utils.py", line 409, in merge
19:25:22      graphs = [r.get() for r in results]
19:25:22    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.7/site-packages/kgx/cli/cli_utils.py", line 409, in <listcomp>
19:25:22      graphs = [r.get() for r in results]
19:25:22    File "/usr/lib/python3.7/multiprocessing/pool.py", line 657, in get
19:25:22      raise self._value
19:25:22  multiprocessing.pool.MaybeEncodingError: Error sending result: '<networkx.classes.multidigraph.MultiDiGraph object at 0x7f2ef0b85f90>'. Reason: 'error("'i' format requires -2147483648 <= number <= 2147483647")'
justaddcoffee commented 3 years ago

Getting closer, but now seeing this error

06:49:42  BRANCH_NAME=switch_to_main_kgx_run_jenkins
06:49:42  + python3.8 run.py transform
06:49:43  Traceback (most recent call last):
06:49:43    File "run.py", line 6, in <module>
06:49:43      from kg_covid_19 import download as kg_download
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/kg_covid_19/__init__.py", line 2, in <module>
06:49:43      from .transform import transform
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/kg_covid_19/transform.py", line 8, in <module>
06:49:43      from kg_covid_19.transform_utils.gocam_transform.gocam_transform import GocamTransform
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/kg_covid_19/transform_utils/gocam_transform/__init__.py", line 1, in <module>
06:49:43      from .gocam_transform import GocamTransform
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/kg_covid_19/transform_utils/gocam_transform/gocam_transform.py", line 6, in <module>
06:49:43      from kgx import RdfTransformer, PandasTransformer # type: ignore
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.8/site-packages/kgx/__init__.py", line 1, in <module>
06:49:43      from kgx.transformers.pandas_transformer import PandasTransformer
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.8/site-packages/kgx/transformers/pandas_transformer.py", line 10, in <module>
06:49:43      from kgx.utils.kgx_utils import generate_edge_key, generate_uuid
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.8/site-packages/kgx/utils/kgx_utils.py", line 6, in <module>
06:49:43      from biolinkml.meta import TypeDefinitionName, ElementName, SlotDefinition, ClassDefinition, TypeDefinition, Element
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.8/site-packages/biolinkml/meta.py", line 27, in <module>
06:49:43      from includes.annotations import Annotation
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.8/site-packages/includes/annotations.py", line 27, in <module>
06:49:43      from includes.extensions import Extension
06:49:43    File "/var/lib/jenkins/workspace/9_switch_to_main_kgx_run_jenkins/gitrepo/venv/lib/python3.8/site-packages/includes/extensions.py", line 47, in <module>
06:49:43      class Extension(YAMLRoot):
06:49:43    File "/usr/lib/python3.8/dataclasses.py", line 1000, in dataclass
06:49:43      return wrap(cls)
06:49:43    File "/usr/lib/python3.8/dataclasses.py", line 992, in wrap
06:49:43      return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
06:49:43    File "/usr/lib/python3.8/dataclasses.py", line 909, in _process_class
06:49:43      _init_fn(flds,
06:49:43  TypeError: dataclasses_init_fn_with_kwargs() missing 1 required positional argument: 'globals'
justaddcoffee commented 3 years ago

Fixed by #419