search

deeppavlov / DeepPavlov

An open source library for deep learning end-to-end dialog systems and chatbots.
https://deeppavlov.ai
Apache License 2.0
6.74k stars 1.15k forks source link

Can't load the coreference model #1344

Closed tutubalinaev closed 2 years ago

tutubalinaev commented 4 years ago

Could you please help with coreference model loading?

DeepPavlov version (you can look it up by running pip show deeppavlov): Name: deeppavlov Version: 0.13.0 Summary: An open source library for building end-to-end dialog systems and training chatbots. Home-page: https://github.com/deepmipt/DeepPavlov Author: Neural Networks and Deep Learning lab, MIPT Author-email: info@deeppavlov.ai License: Apache License, Version 2.0 Location: /usr/local/lib/python3.6/dist-packages Requires: uvicorn, pyopenssl, fastapi, nltk, ruamel.yaml, Cython, h5py, tqdm, filelock, pytelegrambotapi, click, pymorphy2, rusenttokenize, overrides, pytz, scikit-learn, requests, numpy, pymorphy2-dicts-ru, pandas, scipy, aio-pika, pydantic, sacremoses, prometheus-client Required-by:

git branch

Python version: Python 3.6.7

Operating system (ubuntu linux, windows, ...): ubuntu

Issue:

Model can't find deeppavlov/models/coreference_resolution/old_model/coref_kernels.so: cannot open shared object file: No such file or directory

Content or a name of a configuration file:

Command that led to error:

python -m deeppavlov interact deeppavlov/configs/coreference_resolution/old_model_full.json

(same error for python -m deeppavlov interact deeppavlov/configs/coreference_resolution/new_model_full.json)

Error (including full traceback):

[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package perluniprops to /root/nltk_data...
[nltk_data]   Package perluniprops is already up-to-date!
[nltk_data] Downloading package nonbreaking_prefixes to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package nonbreaking_prefixes is already up-to-date!
2020-11-16 19:27:58.248 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 52: [loading fastText embeddings from `/root/.deeppavlov/downloads/embeddings/ft_native_300_ru_wiki_lenta_nltk_wordpunct_tokenize.bin`]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/tlenusik/DeepPavlov/deeppavlov/__main__.py", line 3, in <module>
    main()
  File "/home/tlenusik/DeepPavlov/deeppavlov/deep.py", line 92, in main
    interact_model(pipeline_config_path)
  File "/home/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 75, in interact_model
    model = build_model(config)
  File "/home/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 61, in build_model
    component = from_params(component_config, mode=mode, serialized=component_serialized)
  File "/home/tlenusik/DeepPavlov/deeppavlov/core/common/params.py", line 90, in from_params
    cls = get_model(cls_name)
  File "/home/tlenusik/DeepPavlov/deeppavlov/core/common/registry.py", line 72, in get_model
    return cls_from_str(_REGISTRY[name])
  File "/home/tlenusik/DeepPavlov/deeppavlov/core/common/registry.py", line 41, in cls_from_str
    return getattr(importlib.import_module(module_name), cls_name)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py", line 25, in <module>
    from deeppavlov.models.coreference_resolution.old_model.tf_ops import distance_bins, extract_mentions, get_antecedents, \
  File "/home/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/tf_ops.py", line 7, in <module>
    coref_op_library = tf.load_op_library(str(kernel_path.joinpath("coref_kernels.so")))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/load_library.py", line 60, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/coref_kernels.so: cannot open shared object file: No such file or directory

No such file:
**ls**  deeppavlov/models/coreference_resolution/old_model/
README.md    __pycache__  conll2model_format.py  custom_layers.py  rucor2conll.py               tf_ops.py
__init__.py  build_op.sh  coref_kernels.cc       model.py          speakers_books_component.py
theanhle commented 4 years ago

You need to run setup_all.sh to create the coref_kernels.so file.

tutubalinaev commented 4 years ago

@theanhle where I can find setup_all.sh? I don't see this script in this repository

theanhle commented 4 years ago

Oops! This is an incomplete branch and wasn't merged into the master branch. This model was reproduced from Kenton Lee's model. So, you could find setup_all.sh in the original model.

tutubalinaev commented 4 years ago

@theanhle I found a script build_op.sh in the model's folder, so I used it. Now the model is loaded successfully, yet there is an error with the prediction. Any ideas?

Python code: import json from deeppavlov import build_model

file_name = "coref_example.txt" with open(file_name, "r", encoding='utf8') as f: x = f.read()

PIPELINE_CONFIG_PATH = 'deeppavlov/configs/coreference_resolution/old_model_full.json' with open(PIPELINE_CONFIG_PATH) as f: config = json.load(f) model = build_model(config) model(x)

Error: 2020-11-23 04:57:19.847 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 52: [loading fastText embeddings from /root/.deeppavlov/downloads/embeddings/ft_native_300_ru_wiki_lenta_nltk_wordpunct_tokenize.bin] Warning : load_model does not return WordVectorModel or SupervisedModel any more, but a FastText object which is very similar. /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. 2020-11-23 04:57:32.552 WARNING in 'tensorflow'['tf_logging'] at line 125: From /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py:547: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version. Instructions for updating: seq_dim is deprecated, use seq_axis instead 2020-11-23 04:57:32.552 WARNING in 'tensorflow'['tf_logging'] at line 125: From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:488: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version. Instructions for updating: batch_dim is deprecated, use batch_axis instead /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " 2020-11-23 04:57:34.540606: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-11-23 04:57:34.899111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531 pciBusID: 0000:06:00.0 totalMemory: 22.38GiB freeMemory: 22.22GiB 2020-11-23 04:57:35.123817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531 pciBusID: 0000:07:00.0 totalMemory: 22.38GiB freeMemory: 22.22GiB 2020-11-23 04:57:35.125330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1 2020-11-23 04:57:35.763857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-11-23 04:57:35.763913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2020-11-23 04:57:35.763923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N Y 2020-11-23 04:57:35.763932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: Y N 2020-11-23 04:57:35.764761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21773 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:06:00.0, compute capability: 6.1) 2020-11-23 04:57:35.765357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 21773 MB memory) -> physical GPU (device: 1, name: Tesla P40, pci bus id: 0000:07:00.0, compute capability: 6.1) Traceback (most recent call last): File "coref_test.py", line 12, in model(x) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/chainer.py", line 143, in call return self._compute(*args, param_names=self.in_x, pipe=self.pipe, targets=self.out_params) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/chainer.py", line 155, in _compute raise RuntimeError(f'{expected} are required to compute {targets} but were not found in memory or inputs') RuntimeError: {'sentences', 'clusters', 'speakers', 'doc_key'} are required to compute ['predicted_clusters', 'mention_to_predicted', 'clusters'] but were not found in memory or inputs

yurakuratov commented 4 years ago

Hi!

Could you try to use old_model_full_interact.json config file instead? It takes text as input.

tutubalinaev commented 4 years ago

Hi @yurakuratov! Coref_example.txt is not text, it is a conll file. With old_model_full_interact.json, I have a mismatch between the current graph and the graph from the checkpoint. Ideas?

Command: CUDA_VISIBLE_DEVICES=1 python -m deeppavlov interact deeppavlov/configs/coreference_resolution/old_model_full_interact.json

Log: [nltk_data] Downloading package punkt to /root/nltk_data... [nltk_data] Package punkt is already up-to-date! [nltk_data] Downloading package stopwords to /root/nltk_data... [nltk_data] Package stopwords is already up-to-date! [nltk_data] Downloading package perluniprops to /root/nltk_data... [nltk_data] Package perluniprops is already up-to-date! [nltk_data] Downloading package nonbreaking_prefixes to [nltk_data] /root/nltk_data... [nltk_data] Package nonbreaking_prefixes is already up-to-date! 2020-11-23 09:16:14.292 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 52: [loading fastText embeddings from /root/.deeppavlov/downloads/embeddings/ft_native_300_ru_wiki_lenta_nltk_wordpunct_tokenize.bin] Warning : load_model does not return WordVectorModel or SupervisedModel any more, but a FastText object which is very similar. /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. 2020-11-23 09:16:27.806 WARNING in 'tensorflow'['tf_logging'] at line 125: From /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py:547: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version. Instructions for updating: seq_dim is deprecated, use seq_axis instead 2020-11-23 09:16:27.807 WARNING in 'tensorflow'['tf_logging'] at line 125: From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:488: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version. Instructions for updating: batch_dim is deprecated, use batch_axis instead /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " 2020-11-23 09:16:29.871665: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-11-23 09:16:30.181659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531 pciBusID: 0000:06:00.0 totalMemory: 22.38GiB freeMemory: 22.22GiB 2020-11-23 09:16:30.181699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2020-11-23 09:16:30.510402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-11-23 09:16:30.510453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2020-11-23 09:16:30.510460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2020-11-23 09:16:30.510922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21773 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:06:00.0, compute capability: 6.1) 2020-11-23 09:16:31.118 INFO in 'deeppavlov.core.models.tf_model'['tf_model'] at line 47: [loading model from /root/.deeppavlov/coreference/rucor_checkpoints/old_model_full/best_model.ckpt] 2020-11-23 09:16:31.203 INFO in 'tensorflow'['tf_logging'] at line 115: Restoring parameters from /root/.deeppavlov/coreference/rucor_checkpoints/old_model_full/best_model.ckpt 2020-11-23 09:16:31.340 ERROR in 'deeppavlov.core.common.params'['params'] at line 106: Exception in <class 'deeppavlov.models.coreference_resolution.old_model.model.CorefModel'> Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[{{node save/Assign_14}} = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1546, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

Caused by op 'save/Assign_14', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/main.py", line 3, in main() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/deep.py", line 92, in main interact_model(pipeline_config_path) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 75, in interact_model model = build_model(config) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 61, in build_model component = from_params(component_config, mode=mode, serialized=component_serialized) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/params.py", line 100, in from_params component = cls(dict(config_params, kwargs)) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 74, in call obj.init(*args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(*args, *kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py", line 178, in init self.load() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py", line 50, in load saver = tf.train.Saver(var_list) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1102, in init self.build() File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1114, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build build_save=build_save, build_restore=build_restore) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps assign_ops.append(saveable.restore(saveable_tensors, shapes)) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 119, in restore self.op.get_shape().is_fully_defined()) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/state_ops.py", line 221, in assign validate_shape=validate_shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/params.py", line 100, in from_params component = cls(dict(config_params, kwargs)) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 74, in call obj.init(*args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(*args, *kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py", line 178, in init self.load() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py", line 51, in load saver.restore(self.sess, path) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1582, in restore err, "a mismatch between the current graph and the graph") tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

Caused by op 'save/Assign_14', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/main.py", line 3, in main() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/deep.py", line 92, in main interact_model(pipeline_config_path) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 75, in interact_model model = build_model(config) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 61, in build_model component = from_params(component_config, mode=mode, serialized=component_serialized) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/params.py", line 100, in from_params component = cls(dict(config_params, kwargs)) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 74, in call obj.init(*args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(*args, *kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py", line 178, in init self.load() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py", line 50, in load saver = tf.train.Saver(var_list) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1102, in init self.build() File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1114, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build build_save=build_save, build_restore=build_restore) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps assign_ops.append(saveable.restore(saveable_tensors, shapes)) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 119, in restore self.op.get_shape().is_fully_defined()) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/state_ops.py", line 221, in assign validate_shape=validate_shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[{{node save/Assign_14}} = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1546, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

Caused by op 'save/Assign_14', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/main.py", line 3, in main() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/deep.py", line 92, in main interact_model(pipeline_config_path) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 75, in interact_model model = build_model(config) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 61, in build_model component = from_params(component_config, mode=mode, serialized=component_serialized) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/params.py", line 100, in from_params component = cls(dict(config_params, kwargs)) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 74, in call obj.init(*args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(*args, *kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py", line 178, in init self.load() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py", line 50, in load saver = tf.train.Saver(var_list) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1102, in init self.build() File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1114, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build build_save=build_save, build_restore=build_restore) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps assign_ops.append(saveable.restore(saveable_tensors, shapes)) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 119, in restore self.op.get_shape().is_fully_defined()) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/state_ops.py", line 221, in assign validate_shape=validate_shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/main.py", line 3, in main() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/deep.py", line 92, in main interact_model(pipeline_config_path) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 75, in interact_model model = build_model(config) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 61, in build_model component = from_params(component_config, mode=mode, serialized=component_serialized) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/params.py", line 100, in from_params component = cls(dict(config_params, kwargs)) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 74, in call obj.init(*args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(*args, *kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py", line 178, in init self.load() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py", line 51, in load saver.restore(self.sess, path) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1582, in restore err, "a mismatch between the current graph and the graph") tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

Caused by op 'save/Assign_14', defined at: File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/main.py", line 3, in main() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/deep.py", line 92, in main interact_model(pipeline_config_path) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 75, in interact_model model = build_model(config) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/commands/infer.py", line 61, in build_model component = from_params(component_config, mode=mode, serialized=component_serialized) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/common/params.py", line 100, in from_params component = cls(dict(config_params, kwargs)) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 74, in call obj.init(*args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(*args, *kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py", line 178, in init self.load() File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 27, in _wrapped return func(args, kwargs) File "/Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py", line 50, in load saver = tf.train.Saver(var_list) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1102, in init self.build() File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1114, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build build_save=build_save, build_restore=build_restore) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps assign_ops.append(saveable.restore(saveable_tensors, shapes)) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 119, in restore self.op.get_shape().is_fully_defined()) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/state_ops.py", line 221, in assign validate_shape=validate_shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign use_locking=use_locking, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [0,8] rhs shape= [186,8] [[node save/Assign_14 (defined at /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/core/models/tf_model.py:50) = Assign[T=DT_FLOAT, _class=["loc:@char_embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](char_embeddings, save/RestoreV2/_33)]]

yurakuratov commented 4 years ago

Looks like config files miss characters vocabulary. You need to add

{
    "url": "http://files.deeppavlov.ai/coreference_models/rucor_char.dict",
    "subdir": "{MODELS_PATH}"
}

to download section in configuration file and re-run python -m deeppavlov interact ... -d with -d flag for downloading. Eventually, it should also fix the same problem with old_model_full.json.

tutubalinaev commented 4 years ago

@yurakuratov thank you, It fixed the loading error. Could you please point to output format documentation? I can't figure out that is [((0, 0), (8, 8))] in my example:

LANG=C.UTF-8 LC_ALL=C.UTF-8 CUDA_VISIBLE_DEVICES=1 python -m deeppavlov interact deeppavlov/configs/coreference_resolution/old_model_full_interact.json [nltk_data] Downloading package punkt to /root/nltk_data... [nltk_data] Package punkt is already up-to-date! [nltk_data] Downloading package stopwords to /root/nltk_data... [nltk_data] Package stopwords is already up-to-date! [nltk_data] Downloading package perluniprops to /root/nltk_data... [nltk_data] Package perluniprops is already up-to-date! [nltk_data] Downloading package nonbreaking_prefixes to [nltk_data] /root/nltk_data... [nltk_data] Package nonbreaking_prefixes is already up-to-date! 2020-11-23 11:26:31.750 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /root/.deeppavlov/coreference/rucor_char.dict] 2020-11-23 11:26:31.755 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 52: [loading fastText embeddings from /root/.deeppavlov/downloads/embeddings/ft_native_300_ru_wiki_lenta_nltk_wordpunct_tokenize.bin] Warning : load_model does not return WordVectorModel or SupervisedModel any more, but a FastText object which is very similar. /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. 2020-11-23 11:26:44.733 WARNING in 'tensorflow'['tf_logging'] at line 125: From /Vol0/user/spdmiras/tlenusik/DeepPavlov/deeppavlov/models/coreference_resolution/old_model/model.py:547: calling reverse_sequence (from tensorflow.python.ops.array_ops) with seq_dim is deprecated and will be removed in a future version. Instructions for updating: seq_dim is deprecated, use seq_axis instead 2020-11-23 11:26:44.734 WARNING in 'tensorflow'['tf_logging'] at line 125: From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:488: calling reverse_sequence (from tensorflow.python.ops.array_ops) with batch_dim is deprecated and will be removed in a future version. Instructions for updating: batch_dim is deprecated, use batch_axis instead /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " 2020-11-23 11:26:46.850978: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-11-23 11:26:47.179574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531 pciBusID: 0000:06:00.0 totalMemory: 22.38GiB freeMemory: 22.22GiB 2020-11-23 11:26:47.179632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2020-11-23 11:26:47.504413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-11-23 11:26:47.504466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2020-11-23 11:26:47.504476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2020-11-23 11:26:47.504949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21773 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:06:00.0, compute capability: 6.1) 2020-11-23 11:26:48.138 INFO in 'deeppavlov.core.models.tf_model'['tf_model'] at line 47: [loading model from /root/.deeppavlov/coreference/rucor_checkpoints/old_model_full/best_model.ckpt] 2020-11-23 11:26:48.222 INFO in 'tensorflow'['tf_logging'] at line 115: Restoring parameters from /root/.deeppavlov/coreference/rucor_checkpoints/old_model_full/best_model.ckpt raw_text::Мальчик перестал ходить в школу, то есть он ходил, уезжал, ехал на метро, потом на маршрутном такси и дальше две остановки на троллейбусе (трудный путь).

[((0, 0), (8, 8))]

yurakuratov commented 4 years ago

[((0, 0), (8, 8))] means that there is single coreference chain ((0, 0), (8, 8)) with two mentions (0, 0) and (8, 8). Each mention is described by position of the first and the last token of the mention.

(0, 0) - Мальчик
(8, 8) - он

ru_sent_tokenizer is used for sentence tokenization and nltk_tokenizer for tokenization. You can add tokenized_sentences to chainer's out to get tokenized text.