TypeError: DefaultTrainer: SequenceLabelingModel: TransformerEmbedder: __init__() got an unexpected keyword argument 'allowed_methods'

[LuoYuanzhen]

I am trying to train the KPE baseline model. I just following the tutorials in src/keyphrase_extraction/readme, and after I typed: python scripts/train.py -c examples/ICASSP2023_MUG_track4/configs/bert_crf_sbert.yaml and I got this error:

2023-01-03 07:39:49,387 - INFO - modelscope - downloading http://www.modelscope.cn/api/v1/models/damo/nlp_structbert_backbone_base_std/repo?Revision=v1.0.0&FilePath=config.json to /home/oppoer/.cache/modelscope/hub/temp/tmpjdt0obap/tmpvpaz66i8
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/modelscope/utils/registry.py", line 212, in build_from_cfg
    return obj_cls(**args)
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/modules/embedders/transformer_embedder.py", line 83, in __init__
    self.transformer_model, self.from_hf = get_transformer(
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/modules/embedders/transformer_embedder.py", line 354, in get_transformer
    MsModel.from_pretrained(
  File "/opt/conda/lib/python3.8/site-packages/modelscope/models/base/base_model.py", line 114, in from_pretrained
    local_model_dir = snapshot_download(
  File "/opt/conda/lib/python3.8/site-packages/modelscope/hub/snapshot_download.py", line 128, in snapshot_download
    http_get_file(
  File "/opt/conda/lib/python3.8/site-packages/modelscope/hub/file_download.py", line 211, in http_get_file
    retry = Retry(
TypeError: __init__() got an unexpected keyword argument 'allowed_methods'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/modelscope/utils/registry.py", line 210, in build_from_cfg
    return obj_cls._instantiate(**args)
  File "/opt/conda/lib/python3.8/site-packages/modelscope/models/base/base_model.py", line 62, in _instantiate
    return cls(**kwargs)
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/models/sequence_labeling_model.py", line 59, in __init__
    self.embedder = Embedder.from_config(embedder)
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/modules/embedders/base.py", line 64, in from_config
    return build_embedder(cfg, default_args=kwargs)
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/modules/embedders/base.py", line 24, in build_embedder
    return build_from_cfg(cfg, EMBEDDERS, group_key='default', default_args=default_args)
  File "/opt/conda/lib/python3.8/site-packages/modelscope/utils/registry.py", line 215, in build_from_cfg
    raise type(e)(f'{obj_cls.__name__}: {e}')
TypeError: TransformerEmbedder: __init__() got an unexpected keyword argument 'allowed_methods'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/modelscope/utils/registry.py", line 212, in build_from_cfg
    return obj_cls(**args)
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/training/default_trainer.py", line 67, in __init__
    super().__init__(
  File "/opt/conda/lib/python3.8/site-packages/modelscope/trainers/trainer.py", line 138, in __init__
    self.model = self.build_model()
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/training/default_trainer.py", line 102, in build_model
    return Model.from_config(self.cfg)
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/models/base.py", line 78, in from_config
    model = build_model(model_config, default_args=kwargs)
  File "/opt/conda/lib/python3.8/site-packages/modelscope/models/builder.py", line 30, in build_model
    return build_from_cfg(
  File "/opt/conda/lib/python3.8/site-packages/modelscope/utils/registry.py", line 215, in build_from_cfg
    raise type(e)(f'{obj_cls.__name__}: {e}')
TypeError: SequenceLabelingModel: TransformerEmbedder: __init__() got an unexpected keyword argument 'allowed_methods'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/train.py", line 51, in <module>
    main(args)
  File "scripts/train.py", line 17, in main
    train_model(
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/commands/train.py", line 129, in train_model
    trainer = build_trainer_from_partial_objects(
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/commands/train.py", line 177, in build_trainer_from_partial_objects
    trainer = build_trainer(
  File "/home/notebook/code/personal/S9051103/projects/adaseq/adaseq/training/default_trainer.py", line 176, in build_trainer
    trainer = ms_build_trainer(name, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/modelscope/trainers/builder.py", line 20, in build_trainer
    return build_from_cfg(cfg, TRAINERS, default_args=default_args)
  File "/opt/conda/lib/python3.8/site-packages/modelscope/utils/registry.py", line 215, in build_from_cfg
    raise type(e)(f'{obj_cls.__name__}: {e}')
TypeError: DefaultTrainer: SequenceLabelingModel: TransformerEmbedder: __init__() got an unexpected keyword argument 'allowed_methods'

What did I miss? Some package versions that do not match? Hoping for your response, thank you!

[izhx]

I'm sorry we cannot reproduct this error. Could you provide some detailed envionment info? E.g, a outputs of `pip list.

[LuoYuanzhen]

I'm sorry we cannot reproduct this error. Could you provide some detailed envionment info? E.g, a outputs of pip list. the outputs ofpip list`:

Package                          Version
-------------------------------- -------------------
absl-py                          0.11.0
accelerate                       0.14.0
addict                           2.4.0
aiohttp                          3.8.3
aiosignal                        1.3.1
alabaster                        0.7.12
aliyun-python-sdk-core           2.13.36
aliyun-python-sdk-kms            2.16.0
apex                             0.1
appdirs                          1.4.4
argon2-cffi                      20.1.0
ascii-graph                      1.5.1
async-generator                  1.10
async-timeout                    4.0.2
attrs                            20.3.0
audioread                        2.1.9
Babel                            2.9.0
backcall                         0.2.0
beautifulsoup4                   4.9.3
bleach                           3.2.1
blis                             0.4.1
boto3                            1.16.25
botocore                         1.19.25
brotlipy                         0.7.0
catalogue                        1.0.0
certifi                          2020.11.8
cffi                             1.14.3
chardet                          3.0.4
charset-normalizer               2.1.1
click                            7.1.2
codecov                          2.1.10
conda                            4.9.2
conda-build                      3.20.5
conda-package-handling           1.7.2
coverage                         5.3
crcmod                           1.7
cryptography                     3.2.1
cxxfilt                          0.2.2
cycler                           0.10.0
cymem                            2.0.4
Cython                           0.28.4
DataProperty                     0.50.0
datasets                         2.7.0
decorator                        4.4.2
deepspeed                        0.5.10
defusedxml                       0.6.0
dill                             0.3.6
DLLogger                         0.1.0
docutils                         0.16
easydict                         1.10
einops                           0.6.0
entrypoints                      0.3
filelock                         3.8.0
flake8                           3.7.9
Flask                            1.1.2
frozenlist                       1.3.3
fsspec                           2022.11.0
future                           0.18.2
gast                             0.5.3
glob2                            0.7
graphsurgeon                     0.4.5
grpcio                           1.33.2
h5py                             3.1.0
hjson                            3.0.2
html2text                        2020.1.16
huggingface-hub                  0.11.1
hypothesis                       4.50.8
idna                             2.10
imageio                          2.9.0
imagesize                        1.2.0
importlib-metadata               2.0.0
inflect                          5.0.2
iniconfig                        1.1.1
ipdb                             0.13.4
ipykernel                        5.3.4
ipython                          7.19.0
ipython-genutils                 0.2.0
itsdangerous                     1.1.0
jedi                             0.17.2
Jinja2                           2.11.2
jmespath                         0.10.0
joblib                           0.17.0
json5                            0.9.5
jsonplus                         0.8.0
jsonschema                       3.0.2
jupyter-client                   6.1.7
jupyter-core                     4.7.0
jupyter-tensorboard              0.2.0
jupyterlab                       1.2.14
jupyterlab-pygments              0.1.2
jupyterlab-server                1.2.0
jupytext                         1.7.1
kiwisolver                       1.3.1
libarchive-c                     2.9
librosa                          0.8.0
llvmlite                         0.34.0
lmdb                             1.0.0
Markdown                         3.3.3
markdown-it-py                   0.5.6
MarkupSafe                       1.1.1
maskrcnn-benchmark               0.1
matplotlib                       3.3.3
mbstrdecoder                     1.0.0
mccabe                           0.6.1
mistune                          0.8.4
mlperf-compliance                0.0.10
mock                             4.0.2
modelscope                       1.1.2
mpi4py                           3.1.3
msgfy                            0.1.0
multidict                        6.0.2
multiprocess                     0.70.14
murmurhash                       1.0.2
nbclient                         0.5.1
nbconvert                        6.0.7
nbformat                         5.0.8
nest-asyncio                     1.4.3
networkx                         2.0
ninja                            1.10.2.3
nltk                             3.5
nodejs                           0.1.1
notebook                         6.1.5
numba                            0.51.2
numpy                            1.19.2
nvidia-dali-cuda110              0.28.0
nvidia-pyprof                    3.7.0
nvidia-tensorboard               1.15.0+nv20.12
nvidia-tensorboard-plugin-dlprof 0.10
onnx                             1.7.0
onnxruntime                      1.5.2
opencv-python                    4.4.0.46
optional-django                  0.1.0
oss2                             2.16.0
packaging                        20.4
pandas                           1.1.4
pandocfilters                    1.4.3
parso                            0.7.0
pathvalidate                     2.3.0
pexpect                          4.8.0
pickleshare                      0.7.5
Pillow                           9.3.0
Pillow-SIMD                      7.0.0.post3
pip                              20.2.4
pkginfo                          1.6.1
plac                             0.9.6
pluggy                           0.13.1
polygraphy                       0.20.13
pooch                            1.2.0
preshed                          3.0.2
progressbar                      2.5
prometheus-client                0.9.0
prompt-toolkit                   3.0.8
protobuf                         3.14.0
psutil                           5.7.2
ptyprocess                       0.6.0
py                               1.9.0
py-cpuinfo                       8.0.0
pyarrow                          10.0.0
pybind11                         2.6.1
pycocotools                      2.0+nv0.4.0
pycodestyle                      2.5.0
pycosat                          0.6.3
pycparser                        2.20
pycryptodome                     3.15.0
pydot                            1.4.1
pyflakes                         2.1.1
Pygments                         2.7.2
pynvml                           8.0.4
pyOpenSSL                        19.1.0
pyparsing                        2.4.7
pyrsistent                       0.17.3
PySocks                          1.7.1
pytablewriter                    0.47.0
pytest                           6.1.2
pytest-cov                       2.10.1
pytest-pythonpath                0.7.3
python-dateutil                  2.8.1
python-hostlist                  1.21
python-nvd3                      0.15.0
python-slugify                   4.0.1
pytorch-transformers             1.1.0
pytz                             2020.1
PyWavelets                       1.1.1
PyYAML                           5.3.1
pyzmq                            20.0.0
regex                            2020.11.13
requests                         2.24.0
resampy                          0.2.2
responses                        0.18.0
revtok                           0.0.3
ruamel-yaml                      0.15.87
s3transfer                       0.3.3
sacrebleu                        1.2.10
sacremoses                       0.0.35
scikit-image                     0.15.0
scikit-learn                     0.23.2
scipy                            1.5.2
Send2Trash                       1.5.0
sentencepiece                    0.1.94
seqeval                          1.2.2
setuptools                       50.3.1.post20201107
simplejson                       3.18.0
six                              1.15.0
snowballstemmer                  2.0.0
sortedcontainers                 2.4.0
SoundFile                        0.10.3.post1
soupsieve                        2.0.1
sox                              1.4.1
spacy                            2.3.2
Sphinx                           3.3.1
sphinx-rtd-theme                 0.5.0
sphinxcontrib-applehelp          1.0.2
sphinxcontrib-devhelp            1.0.2
sphinxcontrib-htmlhelp           1.0.3
sphinxcontrib-jsmath             1.0.1
sphinxcontrib-qthelp             1.0.3
sphinxcontrib-serializinghtml    1.1.4
srsly                            1.0.2
stanfordcorenlp                  3.9.1.1
subword-nmt                      0.3.3
tabledata                        1.1.3
tabulate                         0.8.7
tensorboard                      1.15.9999+nv
tensorrt                         7.2.2.1
terminado                        0.9.1
testpath                         0.4.4
text-unidecode                   1.3
thinc                            7.4.1
threadpoolctl                    2.1.0
titan-plugins                    0.0.2
tokenizers                       0.13.2
toml                             0.10.2
torch                            1.12.1
torchtext                        0.9.0a0
torchvision                      0.9.0a0
tornado                          6.1
tqdm                             4.64.1
traitlets                        5.0.5
transformers                     4.25.1
triton                           1.0.0
typepy                           1.1.1
typing-extensions                3.7.4.3
uff                              0.6.9
Unidecode                        1.1.1
urllib3                          1.25.11
wasabi                           0.8.0
wcwidth                          0.2.5
webencodings                     0.5.1
Werkzeug                         1.0.1
wheel                            0.35.1
wrapt                            1.10.11
xxhash                           3.1.0
yacs                             0.1.8
yapf                             0.32.0
yarl                             1.8.1
zipp                             3.4.0

[LuoYuanzhen]

It seems that the problem is caused by the python package urllib3: when I upgrade my urllib3==1.25.11 to urllib3==1.26, the error didn't occur again. The problem solved! Thanks for your time!

[izhx]

I guess it is a bug casued by the Incompatible libs.

modelscope init Retry with the argument allowed_methods, https://github.com/modelscope/modelscope/blob/master/modelscope/hub/file_download.py#L214

but your urllib3 version is too old, it doesn‘t has this argument name. https://github.com/urllib3/urllib3/blob/1.25.11/src/urllib3/util/retry.py#L162

You could create a new empty conda env and install adaseq & modelscope.

[LuoYuanzhen]

I guess it is a bug casued by the Incompatible libs.

modelscope init Retry with the argument allowed_methods, https://github.com/modelscope/modelscope/blob/master/modelscope/hub/file_download.py#L214

but your urllib3 version is too old, it doesn‘t has this argument name. https://github.com/urllib3/urllib3/blob/1.25.11/src/urllib3/util/retry.py#L162

You could create a new empty conda env and install adaseq & modelscope.

Yes!! Thanks again!