search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
134.02k stars 26.8k forks source link

Longformer model with weight(model.encoder.embed_positions.weight) error #11301

Closed BinchaoPeng closed 3 years ago

BinchaoPeng commented 3 years ago 
RuntimeError: Error(s) in loading state_dict for BartModel:
    size mismatch for model.encoder.embed_positions.weight: copying a param with shape torch.Size([16386, 768]) from checkpoint, the shape in current model is torch.Size([1026, 768]).

I use longformer model called longformer-encdec-base-16384 which is downloaded in https://github.com/allenai/longformer，and use huggingface to load the model，when transformers’ version is 3.1.0， the code can run, but when it is 4.4.2，the error happened.

MeanWhile,when I use the model to proposal pairs of sentences，I found it that the returned token_type_ids values are just zero without one. how ever,in the model's special_tokens_map.json, it has defined cls_token and sep_token.

Finally, I sincerely hope you would reply me soon. Thanks！

cronoik commented 3 years ago

What code are you running that leads to that error?

BinchaoPeng commented 3 years ago

God, someone finally replied to me，thanks！

code

from transformers import AutoModel, AutoTokenizer, pipeline
import torch

model_name = 'pre-model/' + 'longformer-encdec-base-16384'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
classifier = pipeline('feature-extraction', model=model, tokenizer=tokenizer)

# encoded_inputs = tokenizer(["ATGCATGCNACT"], ["ATGCATGCNACT"], return_token_type_ids=True, return_tensors='pt')
encoded_inputs = tokenizer(["ATGCATGCNACT", "ATGCATG", "ACTGGTCATGCAC"], return_tensors='pt',
                           padding=True)
print(encoded_inputs)
# feature = model(input_ids=encoded_inputs['input_ids'], attention_mask=encoded_inputs['attention_mask'],
#                 return_netsors='pt')
feature = model(**encoded_inputs,
                return_netsors='pt')
print(feature[0])
print(type(feature[0]))
# feature = torch.as_tensor(feature)
# print(feature.shape)
print("***" * 48)

feature = classifier(["ATG", "ATGCATG", "ACTGGTCATGCAC"])
print(type(feature))
feature = torch.as_tensor(feature)
print(feature)
print(feature.shape)
print("***" * 48)

env info

can work: env0

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
absl-py                   0.12.0                   pypi_0    pypi
astunparse                1.6.3                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
biopython                 1.78                     pypi_0    pypi
blas                      1.0                         mkl    defaults
boto3                     1.17.48                  pypi_0    pypi
botocore                  1.20.48                  pypi_0    pypi
brotlipy                  0.7.0           py36h27cfd23_1003    defaults
ca-certificates           2021.1.19            h06a4308_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cachetools                4.2.1                    pypi_0    pypi
certifi                   2020.12.5        py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cffi                      1.14.5           py36h261ae71_0    defaults
chardet                   4.0.0           py36h06a4308_1003    defaults
click                     7.1.2              pyhd3eb1b0_0    defaults
cryptography              3.4.7            py36hd23ed53_0    defaults
cudatoolkit               10.0.130                      0    defaults
dataclasses               0.8                pyh4f3eec9_6    defaults
dill                      0.3.3                    pypi_0    pypi
filelock                  3.0.12             pyhd3eb1b0_1    defaults
freetype                  2.10.4               h5ab3b9f_0    defaults
future                    0.18.2                   pypi_0    pypi
google-auth               1.28.1                   pypi_0    pypi
google-auth-oauthlib      0.4.4                    pypi_0    pypi
grpcio                    1.37.0                   pypi_0    pypi
idna                      2.10               pyhd3eb1b0_0    defaults
imageio                   2.9.0                    pypi_0    pypi
importlib-metadata        3.10.0                   pypi_0    pypi
intel-openmp              2020.2                      254    defaults
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.0.1              pyhd3eb1b0_0    defaults
jpeg                      9b                   h024ee3a_2    defaults
lcms2                     2.12                 h3be6417_0    defaults
ld_impl_linux-64          2.33.1               h53a641e_7    defaults
libffi                    3.3                  he6710b0_2    defaults
libgcc-ng                 9.1.0                hdf63c60_0    defaults
libpng                    1.6.37               hbc83047_0    defaults
libprotobuf               3.14.0               h8c45485_0    defaults
libstdcxx-ng              9.1.0                hdf63c60_0    defaults
libtiff                   4.1.0                h2733197_1    defaults
longformer                0.1                      pypi_0    pypi
lz4-c                     1.9.3                h2531618_0    defaults
markdown                  3.3.4                    pypi_0    pypi
mkl                       2020.2                      256    defaults
mkl-service               2.3.0            py36he8ac12f_0    defaults
mkl_fft                   1.3.0            py36h54f3939_0    defaults
mkl_random                1.1.1            py36h0573a6f_0    defaults
ncurses                   6.2                  he6710b0_1    defaults
ninja                     1.10.2           py36hff7bd54_0    defaults
nlp                       0.4.0                    pypi_0    pypi
nltk                      3.6.1                    pypi_0    pypi
numpy                     1.19.5                   pypi_0    pypi
numpy-base                1.19.2           py36hfa32c7d_0    defaults
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                     py36_0    defaults
openssl                   1.1.1k               h27cfd23_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
packaging                 20.9               pyhd3eb1b0_0    defaults
pandas                    1.1.5                    pypi_0    pypi
patsy                     0.5.1                    pypi_0    pypi
pillow                    8.2.0            py36he98fc37_0    defaults
pip                       21.0.1           py36h06a4308_0    defaults
protobuf                  3.15.8                   pypi_0    pypi
pyahocorasick             1.4.2                    pypi_0    pypi
pyarrow                   3.0.0                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pybedtools                0.8.2                    pypi_0    pypi
pycparser                 2.20                       py_2    defaults
pyopenssl                 20.0.1             pyhd3eb1b0_1    defaults
pyparsing                 2.4.7              pyhd3eb1b0_0    defaults
pysam                     0.16.0.1                 pypi_0    pypi
pysocks                   1.7.1            py36h06a4308_0    defaults
python                    3.6.13               hdb3f193_0    defaults
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.6                     1_cp36m    huggingface
pytorch-lightning         0.8.5                    pypi_0    pypi
pytorch-transformers      1.2.0                    pypi_0    pypi
pytz                      2021.1                   pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
readline                  8.1                  h27cfd23_0    defaults
regex                     2021.4.4         py36h27cfd23_0    defaults
requests                  2.25.1             pyhd3eb1b0_0    defaults
requests-oauthlib         1.3.0                    pypi_0    pypi
rouge-score               0.0.4                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
s3transfer                0.3.6                    pypi_0    pypi
sacremoses                0.0.44                   pypi_0    pypi
scikit-learn              0.24.1                   pypi_0    pypi
scipy                     1.5.4                    pypi_0    pypi
sentencepiece             0.1.91                   pypi_0    pypi
seqeval                   1.2.2                    pypi_0    pypi
setuptools                52.0.0           py36h06a4308_0    defaults
six                       1.15.0           py36h06a4308_0    defaults
sqlite                    3.35.4               hdfb4753_0    defaults
statsmodels               0.12.2                   pypi_0    pypi
tensorboard               2.4.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorboardx              2.2                      pypi_0    pypi
test-tube                 0.7.5                    pypi_0    pypi
threadpoolctl             2.1.0                    pypi_0    pypi
tk                        8.6.10               hbc83047_0    defaults
tokenizers                0.5.0                    pypi_0    pypi
torch                     1.6.0                    pypi_0    pypi
torchvision               0.5.0                py36_cu100    pytorch
tqdm                      4.60.0                   pypi_0    pypi
transformers              3.1.0                    pypi_0    pypi
typing-extensions         3.7.4.3                  pypi_0    pypi
urllib3                   1.26.4             pyhd3eb1b0_0    defaults
werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3eb1b0_0    defaults
xxhash                    2.0.2                    pypi_0    pypi
xz                        5.2.5                h7b6447c_0    defaults
zipp                      3.4.1                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_3    defaults
zstd                      1.4.9                haebb681_0    defaults

can not work

env1：tf2-pt-keras

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
_tflow_select             2.1.0                       gpu    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
absl-py                   0.11.0           py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
aiohttp                   3.6.3            py36h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
apex                      0.1                      pypi_0    pypi
argon2-cffi               20.1.0           py36h7b6447c_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
astor                     0.8.1                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
astunparse                1.6.3                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
async-timeout             3.0.1                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
async_generator           1.10             py36h28b3542_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
attrs                     20.3.0             pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
backcall                  0.2.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
bert-serving-client       1.10.0                   pypi_0    pypi
bert-serving-server       1.10.0                   pypi_0    pypi
blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
bleach                    3.2.1                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
blinker                   1.4                      py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
brotlipy                  0.7.0           py36h27cfd23_1003    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
c-ares                    1.16.1               h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ca-certificates           2021.4.13            h06a4308_1    defaults
cachetools                4.1.1                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
certifi                   2020.12.5        py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cffi                      1.14.3           py36h261ae71_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
chardet                   3.0.4           py36h06a4308_1003    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
click                     7.1.2                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cryptography              3.2.1            py36h3c74f83_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cudatoolkit               10.1.243             h6bb024c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cudnn                     7.6.5                cuda10.1_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cupti                     10.1.168                      0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cycler                    0.10.0                   py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
dataclasses               0.7                      py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
dbus                      1.13.18              hb2f20db_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
decorator                 4.4.2                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
defusedxml                0.6.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
entrypoints               0.3                      py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
expat                     2.2.10               he6710b0_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
filelock                  3.0.12             pyhd3eb1b0_1    defaults
fontconfig                2.13.0               h9420a91_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
freetype                  2.10.4               h5ab3b9f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
gast                      0.2.2                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
glib                      2.66.1               h92f7085_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
google-auth               1.23.0             pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
google-auth-oauthlib      0.4.2              pyhd3eb1b0_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
google-pasta              0.2.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
gputil                    1.4.0                    pypi_0    pypi
grpcio                    1.31.0           py36hf8bcb03_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
gst-plugins-base          1.14.0               hbbd80ab_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
gstreamer                 1.14.0               hb31296c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
h5py                      2.10.0           py36hd6299e0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
hdf5                      1.10.6               hb1b8bf9_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
icu                       58.2                 he6710b0_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
idna                      2.10                       py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
idna_ssl                  1.1.0                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
importlib-metadata        2.0.0                      py_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
importlib_metadata        2.0.0                         1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
intel-openmp              2020.2                      254    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ipykernel                 5.3.4            py36h5ca1d4c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ipython                   7.12.0           py36h5ca1d4c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
ipython_genutils          0.2.0              pyhd3eb1b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ipywidgets                7.6.0              pyhd3eb1b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jedi                      0.10.2                   py36_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
jinja2                    2.11.2                     py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
joblib                    0.17.0                     py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jpeg                      9b                   h024ee3a_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jsonschema                3.2.0                      py_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jupyter                   1.0.0                    py36_7    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jupyter_client            6.1.7                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jupyter_console           6.2.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jupyter_core              4.7.0            py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jupyterlab_pygments       0.1.2                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
keras                     2.3.1                         0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
keras-applications        1.0.8                      py_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
keras-base                2.3.1                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
keras-preprocessing       1.1.0                      py_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
kiwisolver                1.3.0            py36h2531618_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
krb5                      1.18.2               h173b8e3_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
lcms2                     2.11                 h396b838_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ld_impl_linux-64          2.33.1               h53a641e_7    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libcurl                   7.71.1               h20c2e04_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libedit                   3.1.20191231         h14c3975_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libffi                    3.3                  he6710b0_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgcc-ng                 9.1.0                hdf63c60_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgfortran-ng            7.3.0                hdf63c60_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libpng                    1.6.37               hbc83047_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libprotobuf               3.13.0.1             hd408876_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libsodium                 1.0.18               h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libssh2                   1.9.0                h1ba5d50_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libstdcxx-ng              9.1.0                hdf63c60_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libtiff                   4.1.0                h2733197_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libuuid                   1.0.3                h1bed415_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libuv                     1.40.0               h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libxcb                    1.14                 h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libxml2                   2.9.10               hb55368b_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
lz4-c                     1.9.2                heb0550a_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
markdown                  3.3.3            py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
markupsafe                1.1.1            py36h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
matplotlib                3.3.2                         0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
matplotlib-base           3.3.2            py36h817c723_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mistune                   0.8.4            py36h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl                       2020.2                      256    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl-service               2.3.0            py36he904b0f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl_fft                   1.2.0            py36h23d657b_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl_random                1.1.1            py36h0573a6f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
multidict                 4.7.6            py36h7b6447c_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
nbclient                  0.5.1                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
nbconvert                 6.0.7                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
nbformat                  5.0.8                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ncurses                   6.2                  he6710b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
nest-asyncio              1.4.3              pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ninja                     1.10.1           py36hfd86e86_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
notebook                  6.1.6            py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
numpy                     1.19.2           py36h54aff64_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
numpy-base                1.19.2           py36hfa32c7d_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
oauthlib                  3.1.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
olefile                   0.46                     py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
openssl                   1.1.1k               h27cfd23_0    defaults
opt_einsum                3.1.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
packaging                 20.8               pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pandas                    1.1.3            py36he6710b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pandoc                    2.11                 hb0f4dca_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pandocfilters             1.4.3            py36h06a4308_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pcre                      8.44                 he6710b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pexpect                   4.8.0              pyhd3eb1b0_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pickleshare               0.7.5           pyhd3eb1b0_1003    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pillow                    8.0.1            py36he98fc37_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pip                       20.2.4           py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
prometheus_client         0.9.0              pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
prompt-toolkit            3.0.8                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
prompt_toolkit            3.0.8                         0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
protobuf                  3.13.0.1         py36he6710b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ptyprocess                0.6.0              pyhd3eb1b0_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyasn1                    0.4.8                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyasn1-modules            0.2.8                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pycparser                 2.20                       py_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pygments                  2.7.3              pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyjwt                     1.7.1                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyopenssl                 19.1.0             pyhd3eb1b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyparsing                 2.4.7                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyqt                      5.9.2            py36h05f1152_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyrsistent                0.17.3           py36h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pysocks                   1.7.1            py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python                    3.6.12               hcff3b4d_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python-dateutil           2.8.1                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python_abi                3.6                     1_cp36m    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
pytorch                   1.7.0           py3.6_cuda10.1.243_cudnn7.6.3_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
pytz                      2020.1                     py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyyaml                    5.3.1            py36h7b6447c_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyzmq                     20.0.0           py36h2531618_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
qt                        5.9.7                h5867ecd_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
qtconsole                 4.7.7                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
qtpy                      1.9.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
readline                  8.0                  h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
regex                     2021.4.4         py36h27cfd23_0    defaults
requests                  2.24.0                     py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
requests-oauthlib         1.3.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
rsa                       4.6                        py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
sacremoses                0.0.44                   pypi_0    pypi
scikit-learn              0.23.2           py36h0573a6f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
scipy                     1.5.2            py36h0b6359f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
seaborn                   0.11.1             pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
send2trash                1.5.0              pyhd3eb1b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
setuptools                50.3.1           py36h06a4308_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
sip                       4.19.8           py36hf484d3e_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
six                       1.15.0           py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
sqlite                    3.33.0               h62c20be_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorboard               2.3.0              pyh4dce500_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorboard-plugin-wit    1.6.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorflow                2.1.0           gpu_py36h2e5cdaa_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorflow-base           2.1.0           gpu_py36h6c5654b_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorflow-estimator      2.1.0              pyhd54b08b_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorflow-gpu            2.1.0                h0d30ee6_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
termcolor                 1.1.0                    py36_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
terminado                 0.9.1                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
testpath                  0.4.4                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
threadpoolctl             2.1.0              pyh5ca1d4c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tk                        8.6.10               hbc83047_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tokenizers                0.10.2                   pypi_0    pypi
torchaudio                0.7.0                      py36    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
torchvision               0.1.8                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
tornado                   6.0.4            py36h7b6447c_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tqdm                      4.60.0                   pypi_0    pypi
traitlets                 4.3.3                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
transformers              4.4.2                      py_0    huggingface
typing_extensions         3.7.4.3                    py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
urllib3                   1.25.11                    py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
wcwidth                   0.2.5                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
webencodings              0.5.1                    py36_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
werkzeug                  1.0.1                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
wheel                     0.35.1             pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
widgetsnbextension        3.5.1                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
wrapt                     1.12.1           py36h7b6447c_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
xz                        5.2.5                h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
yaml                      0.2.5                h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
yarl                      1.6.2            py36h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zeromq                    4.3.3                he6710b0_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zipp                      3.4.0              pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zlib                      1.2.11               h7b6447c_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zstd                      1.4.5                h9ceee32_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

env2: copied from env0 but not worked

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
absl-py                   0.12.0                   pypi_0    pypi
astunparse                1.6.3                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
boto3                     1.17.53                  pypi_0    pypi
botocore                  1.20.53                  pypi_0    pypi
brotlipy                  0.7.0           py36h27cfd23_1003    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ca-certificates           2021.4.13            h06a4308_1  
cachetools                4.2.1                    pypi_0    pypi
certifi                   2020.12.5        py36h06a4308_0  
cffi                      1.14.5           py36h261ae71_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
chardet                   4.0.0           py36h06a4308_1003    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
click                     7.1.2              pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cryptography              3.4.7            py36hd23ed53_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cudatoolkit               10.0.130                      0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
dataclasses               0.8                pyh4f3eec9_6    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
dill                      0.3.3                    pypi_0    pypi
filelock                  3.0.12             pyhd3eb1b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
freetype                  2.10.4               h5ab3b9f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
future                    0.18.2                   pypi_0    pypi
google-auth               1.29.0                   pypi_0    pypi
google-auth-oauthlib      0.4.4                    pypi_0    pypi
grpcio                    1.37.0                   pypi_0    pypi
idna                      2.10               pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
imageio                   2.9.0                    pypi_0    pypi
importlib-metadata        2.0.0                      py_1    anaconda
intel-openmp              2020.2                      254    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.0.1              pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jpeg                      9b                   h024ee3a_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
lcms2                     2.12                 h3be6417_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ld_impl_linux-64          2.33.1               h53a641e_7    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libffi                    3.3                  he6710b0_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgcc-ng                 9.1.0                hdf63c60_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libpng                    1.6.37               hbc83047_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libprotobuf               3.14.0               h8c45485_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libstdcxx-ng              9.1.0                hdf63c60_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libtiff                   4.1.0                h2733197_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
longformer                0.1                      pypi_0    pypi
lz4-c                     1.9.3                h2531618_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
markdown                  3.3.4                    pypi_0    pypi
mkl                       2020.2                      256    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl-service               2.3.0            py36he8ac12f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl_fft                   1.3.0            py36h54f3939_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl_random                1.1.1            py36h0573a6f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ncurses                   6.2                  he6710b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ninja                     1.10.2           py36hff7bd54_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
nlp                       0.4.0                    pypi_0    pypi
nltk                      3.6.1                    pypi_0    pypi
numpy                     1.19.2           py36h54aff64_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
numpy-base                1.19.2           py36hfa32c7d_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                     py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
openssl                   1.1.1k               h27cfd23_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
packaging                 20.9               pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pandas                    1.1.5                    pypi_0    pypi
pillow                    8.2.0            py36he98fc37_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pip                       21.0.1           py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
protobuf                  3.15.8                   pypi_0    pypi
pyarrow                   3.0.0                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycparser                 2.20                       py_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyopenssl                 20.0.1             pyhd3eb1b0_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pyparsing                 2.4.7              pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pysocks                   1.7.1            py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python                    3.6.13               hdb3f193_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.6                     1_cp36m    huggingface
pytorch-lightning         0.8.5                    pypi_0    pypi
pytorch-transformers      1.2.0                    pypi_0    pypi
pytz                      2021.1                   pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
readline                  8.1                  h27cfd23_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
regex                     2021.4.4         py36h27cfd23_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
requests                  2.25.1             pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
requests-oauthlib         1.3.0                    pypi_0    pypi
rouge-score               0.0.4                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
s3transfer                0.3.7                    pypi_0    pypi
sacremoses                0.0.44                   pypi_0    pypi
sentencepiece             0.1.95                   pypi_0    pypi
setuptools                52.0.0           py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
six                       1.15.0           py36h06a4308_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
sqlite                    3.35.4               hdfb4753_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorboard               2.4.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorboardx              2.2                      pypi_0    pypi
test-tube                 0.7.5                    pypi_0    pypi
tk                        8.6.10               hbc83047_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tokenizers                0.8.1rc2                 pypi_0    pypi
torch                     1.6.0                    pypi_0    pypi
torchvision               0.5.0                py36_cu100    pytorch
tqdm                      4.60.0                   pypi_0    pypi
transformers              3.1.0                    pypi_0    pypi
urllib3                   1.26.4             pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3eb1b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
xxhash                    2.0.2                    pypi_0    pypi
xz                        5.2.5                h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zipp                      3.4.1              pyhd3eb1b0_0  
zlib                      1.2.11               h7b6447c_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zstd                      1.4.9                haebb681_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
BinchaoPeng commented 3 years ago

next step i wan to use gene seqs to pretrain longformer, but i has been seemly dead in step 0 ....

cronoik commented 3 years ago

Can you please post the output of:

type(model)

of your working environment? In case it is showing something with ....BartModel, can you please show us the definition of the class BertEncoder? You can locate it in the directory of:

import transformers
print(transformers.__file__)
BinchaoPeng commented 3 years ago

Can you please post the output of:

type(model)

of your working environment? In case it is showing something with ....BartModel, can you please show us the definition of the class BertEncoder? You can locate it in the directory of:

import transformers
print(transformers.__file__)

code

from transformers import AutoModel, AutoTokenizer  # , pipeline
import transformers
print(transformers.__file__)

model_name = 'pre-model/' + 'longformer-encdec-base-16384'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
# classifier = pipeline('feature-extraction', model=model, tokenizer=tokenizer)

print(type(model))

env0:

/home/pbc/anaconda3/envs/dnabert/lib/python3.6/site-packages/transformers/__init__.py
Some weights of the model checkpoint at pre-model/longformer-encdec-base-16384 were not used when initializing BartModel: ['model.encoder.layers.0.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.0.self_attn.output.weight', 'model.encoder.layers.0.self_attn.output.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.1.self_attn.output.weight', 'model.encoder.layers.1.self_attn.output.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.2.self_attn.output.weight', 'model.encoder.layers.2.self_attn.output.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.3.self_attn.output.weight', 'model.encoder.layers.3.self_attn.output.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.4.self_attn.output.weight', 'model.encoder.layers.4.self_attn.output.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.5.self_attn.output.weight', 'model.encoder.layers.5.self_attn.output.bias']
- This IS expected if you are initializing BartModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BartModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BartModel were not initialized from the model checkpoint at pre-model/longformer-encdec-base-16384 and are newly initialized: ['model.encoder.layers.0.self_attn.k_proj.weight', 'model.encoder.layers.0.self_attn.k_proj.bias', 'model.encoder.layers.0.self_attn.v_proj.weight', 'model.encoder.layers.0.self_attn.v_proj.bias', 'model.encoder.layers.0.self_attn.q_proj.weight', 'model.encoder.layers.0.self_attn.q_proj.bias', 'model.encoder.layers.0.self_attn.out_proj.weight', 'model.encoder.layers.0.self_attn.out_proj.bias', 'model.encoder.layers.1.self_attn.k_proj.weight', 'model.encoder.layers.1.self_attn.k_proj.bias', 'model.encoder.layers.1.self_attn.v_proj.weight', 'model.encoder.layers.1.self_attn.v_proj.bias', 'model.encoder.layers.1.self_attn.q_proj.weight', 'model.encoder.layers.1.self_attn.q_proj.bias', 'model.encoder.layers.1.self_attn.out_proj.weight', 'model.encoder.layers.1.self_attn.out_proj.bias', 'model.encoder.layers.2.self_attn.k_proj.weight', 'model.encoder.layers.2.self_attn.k_proj.bias', 'model.encoder.layers.2.self_attn.v_proj.weight', 'model.encoder.layers.2.self_attn.v_proj.bias', 'model.encoder.layers.2.self_attn.q_proj.weight', 'model.encoder.layers.2.self_attn.q_proj.bias', 'model.encoder.layers.2.self_attn.out_proj.weight', 'model.encoder.layers.2.self_attn.out_proj.bias', 'model.encoder.layers.3.self_attn.k_proj.weight', 'model.encoder.layers.3.self_attn.k_proj.bias', 'model.encoder.layers.3.self_attn.v_proj.weight', 'model.encoder.layers.3.self_attn.v_proj.bias', 'model.encoder.layers.3.self_attn.q_proj.weight', 'model.encoder.layers.3.self_attn.q_proj.bias', 'model.encoder.layers.3.self_attn.out_proj.weight', 'model.encoder.layers.3.self_attn.out_proj.bias', 'model.encoder.layers.4.self_attn.k_proj.weight', 'model.encoder.layers.4.self_attn.k_proj.bias', 'model.encoder.layers.4.self_attn.v_proj.weight', 'model.encoder.layers.4.self_attn.v_proj.bias', 'model.encoder.layers.4.self_attn.q_proj.weight', 'model.encoder.layers.4.self_attn.q_proj.bias', 'model.encoder.layers.4.self_attn.out_proj.weight', 'model.encoder.layers.4.self_attn.out_proj.bias', 'model.encoder.layers.5.self_attn.k_proj.weight', 'model.encoder.layers.5.self_attn.k_proj.bias', 'model.encoder.layers.5.self_attn.v_proj.weight', 'model.encoder.layers.5.self_attn.v_proj.bias', 'model.encoder.layers.5.self_attn.q_proj.weight', 'model.encoder.layers.5.self_attn.q_proj.bias', 'model.encoder.layers.5.self_attn.out_proj.weight', 'model.encoder.layers.5.self_attn.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
<class 'transformers.modeling_bart.BartModel'>

env1:tf2-pt-keras

/home/pbc/anaconda3/envs/tf2_pt_kr2/lib/python3.6/site-packages/transformers-4.4.2-py3.8.egg/transformers/__init__.py
Some weights of the model checkpoint at pre-model/longformer-encdec-base-16384 were not used when initializing BartModel: ['model.encoder.layers.0.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.0.self_attn.output.weight', 'model.encoder.layers.0.self_attn.output.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.1.self_attn.output.weight', 'model.encoder.layers.1.self_attn.output.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.2.self_attn.output.weight', 'model.encoder.layers.2.self_attn.output.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.3.self_attn.output.weight', 'model.encoder.layers.3.self_attn.output.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.4.self_attn.output.weight', 'model.encoder.layers.4.self_attn.output.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.5.self_attn.output.weight', 'model.encoder.layers.5.self_attn.output.bias']
- This IS expected if you are initializing BartModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BartModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BartModel were not initialized from the model checkpoint at pre-model/longformer-encdec-base-16384 and are newly initialized: ['model.encoder.layers.0.self_attn.k_proj.weight', 'model.encoder.layers.0.self_attn.k_proj.bias', 'model.encoder.layers.0.self_attn.v_proj.weight', 'model.encoder.layers.0.self_attn.v_proj.bias', 'model.encoder.layers.0.self_attn.q_proj.weight', 'model.encoder.layers.0.self_attn.q_proj.bias', 'model.encoder.layers.0.self_attn.out_proj.weight', 'model.encoder.layers.0.self_attn.out_proj.bias', 'model.encoder.layers.1.self_attn.k_proj.weight', 'model.encoder.layers.1.self_attn.k_proj.bias', 'model.encoder.layers.1.self_attn.v_proj.weight', 'model.encoder.layers.1.self_attn.v_proj.bias', 'model.encoder.layers.1.self_attn.q_proj.weight', 'model.encoder.layers.1.self_attn.q_proj.bias', 'model.encoder.layers.1.self_attn.out_proj.weight', 'model.encoder.layers.1.self_attn.out_proj.bias', 'model.encoder.layers.2.self_attn.k_proj.weight', 'model.encoder.layers.2.self_attn.k_proj.bias', 'model.encoder.layers.2.self_attn.v_proj.weight', 'model.encoder.layers.2.self_attn.v_proj.bias', 'model.encoder.layers.2.self_attn.q_proj.weight', 'model.encoder.layers.2.self_attn.q_proj.bias', 'model.encoder.layers.2.self_attn.out_proj.weight', 'model.encoder.layers.2.self_attn.out_proj.bias', 'model.encoder.layers.3.self_attn.k_proj.weight', 'model.encoder.layers.3.self_attn.k_proj.bias', 'model.encoder.layers.3.self_attn.v_proj.weight', 'model.encoder.layers.3.self_attn.v_proj.bias', 'model.encoder.layers.3.self_attn.q_proj.weight', 'model.encoder.layers.3.self_attn.q_proj.bias', 'model.encoder.layers.3.self_attn.out_proj.weight', 'model.encoder.layers.3.self_attn.out_proj.bias', 'model.encoder.layers.4.self_attn.k_proj.weight', 'model.encoder.layers.4.self_attn.k_proj.bias', 'model.encoder.layers.4.self_attn.v_proj.weight', 'model.encoder.layers.4.self_attn.v_proj.bias', 'model.encoder.layers.4.self_attn.q_proj.weight', 'model.encoder.layers.4.self_attn.q_proj.bias', 'model.encoder.layers.4.self_attn.out_proj.weight', 'model.encoder.layers.4.self_attn.out_proj.bias', 'model.encoder.layers.5.self_attn.k_proj.weight', 'model.encoder.layers.5.self_attn.k_proj.bias', 'model.encoder.layers.5.self_attn.v_proj.weight', 'model.encoder.layers.5.self_attn.v_proj.bias', 'model.encoder.layers.5.self_attn.q_proj.weight', 'model.encoder.layers.5.self_attn.q_proj.bias', 'model.encoder.layers.5.self_attn.out_proj.weight', 'model.encoder.layers.5.self_attn.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "/home/pbc/anaconda3/envs/tf2_pt_kr2/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-b1f8935f1cfa>", line 1, in <module>
    runfile('/home/pbc/Documents/PycharmProjects/myEPI/src/github.py', wdir='/home/pbc/Documents/PycharmProjects/myEPI/src')
  File "/home/pbc/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/202.7660.27/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/pbc/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/202.7660.27/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/pbc/Documents/PycharmProjects/myEPI/src/github.py", line 8, in <module>
    model = AutoModel.from_pretrained(model_name)
  File "/home/pbc/anaconda3/envs/tf2_pt_kr2/lib/python3.6/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/auto/modeling_auto.py", line 815, in from_pretrained
    pretrained_model_name_or_path, *model_args, config=config, **kwargs
  File "/home/pbc/anaconda3/envs/tf2_pt_kr2/lib/python3.6/site-packages/transformers-4.4.2-py3.8.egg/transformers/modeling_utils.py", line 1183, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for BartModel:
    size mismatch for model.encoder.embed_positions.weight: copying a param with shape torch.Size([16386, 768]) from checkpoint, the shape in current model is torch.Size([1026, 768]).

env2: copied from env0 but not worked

home/pbc/anaconda3/envs/dnabert/lib/python3.6/site-packages/transformers/__init__.py
Some weights of the model checkpoint at pre-model/longformer-encdec-base-16384 were not used when initializing BartModel: ['model.encoder.layers.0.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.0.self_attn.output.weight', 'model.encoder.layers.0.self_attn.output.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.1.self_attn.output.weight', 'model.encoder.layers.1.self_attn.output.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.2.self_attn.output.weight', 'model.encoder.layers.2.self_attn.output.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.3.self_attn.output.weight', 'model.encoder.layers.3.self_attn.output.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.4.self_attn.output.weight', 'model.encoder.layers.4.self_attn.output.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.5.self_attn.output.weight', 'model.encoder.layers.5.self_attn.output.bias']
- This IS expected if you are initializing BartModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BartModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BartModel were not initialized from the model checkpoint at pre-model/longformer-encdec-base-16384 and are newly initialized: ['model.encoder.layers.0.self_attn.k_proj.weight', 'model.encoder.layers.0.self_attn.k_proj.bias', 'model.encoder.layers.0.self_attn.v_proj.weight', 'model.encoder.layers.0.self_attn.v_proj.bias', 'model.encoder.layers.0.self_attn.q_proj.weight', 'model.encoder.layers.0.self_attn.q_proj.bias', 'model.encoder.layers.0.self_attn.out_proj.weight', 'model.encoder.layers.0.self_attn.out_proj.bias', 'model.encoder.layers.1.self_attn.k_proj.weight', 'model.encoder.layers.1.self_attn.k_proj.bias', 'model.encoder.layers.1.self_attn.v_proj.weight', 'model.encoder.layers.1.self_attn.v_proj.bias', 'model.encoder.layers.1.self_attn.q_proj.weight', 'model.encoder.layers.1.self_attn.q_proj.bias', 'model.encoder.layers.1.self_attn.out_proj.weight', 'model.encoder.layers.1.self_attn.out_proj.bias', 'model.encoder.layers.2.self_attn.k_proj.weight', 'model.encoder.layers.2.self_attn.k_proj.bias', 'model.encoder.layers.2.self_attn.v_proj.weight', 'model.encoder.layers.2.self_attn.v_proj.bias', 'model.encoder.layers.2.self_attn.q_proj.weight', 'model.encoder.layers.2.self_attn.q_proj.bias', 'model.encoder.layers.2.self_attn.out_proj.weight', 'model.encoder.layers.2.self_attn.out_proj.bias', 'model.encoder.layers.3.self_attn.k_proj.weight', 'model.encoder.layers.3.self_attn.k_proj.bias', 'model.encoder.layers.3.self_attn.v_proj.weight', 'model.encoder.layers.3.self_attn.v_proj.bias', 'model.encoder.layers.3.self_attn.q_proj.weight', 'model.encoder.layers.3.self_attn.q_proj.bias', 'model.encoder.layers.3.self_attn.out_proj.weight', 'model.encoder.layers.3.self_attn.out_proj.bias', 'model.encoder.layers.4.self_attn.k_proj.weight', 'model.encoder.layers.4.self_attn.k_proj.bias', 'model.encoder.layers.4.self_attn.v_proj.weight', 'model.encoder.layers.4.self_attn.v_proj.bias', 'model.encoder.layers.4.self_attn.q_proj.weight', 'model.encoder.layers.4.self_attn.q_proj.bias', 'model.encoder.layers.4.self_attn.out_proj.weight', 'model.encoder.layers.4.self_attn.out_proj.bias', 'model.encoder.layers.5.self_attn.k_proj.weight', 'model.encoder.layers.5.self_attn.k_proj.bias', 'model.encoder.layers.5.self_attn.v_proj.weight', 'model.encoder.layers.5.self_attn.v_proj.bias', 'model.encoder.layers.5.self_attn.q_proj.weight', 'model.encoder.layers.5.self_attn.q_proj.bias', 'model.encoder.layers.5.self_attn.out_proj.weight', 'model.encoder.layers.5.self_attn.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/pbc/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.7148.72/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/pbc/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.7148.72/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/pbc/PycharmProjects/bert/github.py", line 7, in <module>
    model = AutoModel.from_pretrained(model_name)
  File "/home/pbc/anaconda3/envs/dnabert/lib/python3.6/site-packages/transformers/modeling_auto.py", line 523, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/home/pbc/anaconda3/envs/dnabert/lib/python3.6/site-packages/transformers/modeling_utils.py", line 972, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for BartModel:
    size mismatch for model.encoder.embed_positions.weight: copying a param with shape torch.Size([16386, 768]) from checkpoint, the shape in current model is torch.Size([1026, 768]).

I found transformers.file all are different

cronoik commented 3 years ago

Now please check this directory /home/pbc/anaconda3/envs/dnabert/lib/python3.6/site-packages/transformers/ and locate the file called modeling_bart.py. Post the BertEncoder class definition here.

You should also pay attention to the weights that were not used from the pre-trained weights: ['model.encoder.layers.0.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.0.self_attn.output.weight', 'model.encoder.layers.0.self_attn.output.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.1.self_attn.output.weight', 'model.encoder.layers.1.self_attn.output.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.2.self_attn.output.weight', 'model.encoder.layers.2.self_attn.output.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.3.self_attn.output.weight', 'model.encoder.layers.3.self_attn.output.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.4.self_attn.output.weight', 'model.encoder.layers.4.self_attn.output.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.5.self_attn.output.weight', 'model.encoder.layers.5.self_attn.output.bias']

Are you sure that this model (including config and weights) can be used with the transformers AutoModel class? Currently, it looks to me that someone has built his own model with the transformers library (which is not supposed to work with the AutoClasses).

BinchaoPeng commented 3 years ago

Now please check this directory /home/pbc/anaconda3/envs/dnabert/lib/python3.6/site-packages/transformers/ and locate the file called modeling_bart.py. Post the BertEncoder class definition here.

You should also pay attention to the weights that were not used from the pre-trained weights: ['model.encoder.layers.0.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.0.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.0.self_attn.output.weight', 'model.encoder.layers.0.self_attn.output.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.1.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.1.self_attn.output.weight', 'model.encoder.layers.1.self_attn.output.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.2.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.2.self_attn.output.weight', 'model.encoder.layers.2.self_attn.output.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.3.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.3.self_attn.output.weight', 'model.encoder.layers.3.self_attn.output.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.4.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.4.self_attn.output.weight', 'model.encoder.layers.4.self_attn.output.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.query_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.key_global.bias', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.weight', 'model.encoder.layers.5.self_attn.longformer_self_attn.value_global.bias', 'model.encoder.layers.5.self_attn.output.weight', 'model.encoder.layers.5.self_attn.output.bias']

Are you sure that this model (including config and weights) can be used with the transformers AutoModel class? Currently, it looks to me that someone has built his own model with the transformers library (which is not supposed to work with the AutoClasses).

I am not sure, when I start to use bert model by the same way , it works, So I make it again in longformer, then many errors occured one by one

cronoik commented 3 years ago

That is a different thing. What you have here longformer-encdec-base-16384 is something that is provided by someone that is not supposed to work with the provided AutoClasses by hugging face. Please check the code of this someone and see what this person did. I think this is the repository you should check out: https://github.com/allenai/ms2 or maybe this code snippet: https://github.com/allenai/longformer/issues/154

BinchaoPeng commented 3 years ago

That is a different thing. What you have here longformer-encdec-base-16384 is something that is provided by someone that is not supposed to work with the provided AutoClasses by hugging face. Please check the code of this someone and see what this person did. I think this is the repository you should check out: https://github.com/allenai/ms2 or maybe this code snippet: allenai/longformer#154

yeah, you means that I should install env with allenai/longformer and not huggingface, at start I read allenai/longformer's readme, i just found that it may from huggingface and don't look for any things about how to use its longformer model by python code.

I have seen allenai/longformer#154, and I will try it through Imitating her code.

And another question is, if I want to use hugging face env to load model, that means I should download in https://huggingface.co/?

As for ms2, I will view it soon, Thanks!

Finally,thank you very much! you save me!Thanks!ORZ

cronoik commented 3 years ago

Yes, the allenai/longformer is the framework you should use for longformer-encdec-base-16384.

And another question is, if I want to use hugging face env to load model, that means I should download in https://huggingface.co/?

Yes, you can check the pre-trained models here: https://huggingface.co/models

BinchaoPeng commented 3 years ago

Okay, and I am curious that how do you find allenai/longformer#154 and https://github.com/allenai/ms2. If I have the skill, I can save myself quickly,haha.

cronoik commented 3 years ago

Use a search engine of your choice and look for longformer-encdec-base-16384 ;-)

BinchaoPeng commented 3 years ago

longformer-encdec-base-16384

OK，thank you very much!

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.