Combining LiLT with XML RoBERTa large fails

[piegu]

Hi @NielsRogge,

Many thanks for your notebook CreateLiLT+_XLM_RoBERTa_base.ipynb.

I try to use it for combining LiLT with XML RoBERTa large but it failed.

Here the code:

python gen_weight_roberta_like.py \
     --lilt /content/drive/MyDrive/LiLT/Original_checkpoints/lilt-only-base/pytorch_model.bin \
     --text /content/LiLT/xlm-roberta-large/pytorch_model.bin \
     --config /content/LiLT/xlm-roberta-large/config.json \
     --out lilt-xlm-roberta-large

... and the error message:

Traceback (most recent call last):
  File "gen_weight_roberta_like.py", line 4, in <module>
    import transformers
  File "/usr/local/lib/python3.8/dist-packages/transformers/__init__.py", line 43, in <module>
    from . import dependency_versions_check
  File "/usr/local/lib/python3.8/dist-packages/transformers/dependency_versions_check.py", line 41, in <module>
    require_version_core(deps[pkg])
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/versions.py", line 101, in require_version_core
    return require_version(requirement, hint)
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/versions.py", line 92, in require_version
    if want_ver is not None and not ops[op](version.parse(got_ver), version.parse(want_ver)):
  File "/usr/local/lib/python3.8/dist-packages/packaging/version.py", line 52, in parse
    return Version(version)
  File "/usr/local/lib/python3.8/dist-packages/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '0.10.1,<0.11'

Could you tell me what to do or update your notebook? Thank you.

[NielsRogge]

Yes, that's expected, as LiLT only has 12 layers and RoBERTa-large has 24 layers. Hence, it's only possible to combine LiLT with base-sized checkpoints.

[ashutoshbhushan]

Thanks!