search

152334H / DL-Art-School

TorToiSe fine-tuning with DLAS
GNU Affero General Public License v3.0
218 stars 114 forks source link

Error on ljspeech train starting in google colab: Unexpected key(s) in state_dict: "gpt.h.0.attn.bias", ... #86

Open pivolan opened 1 year ago

pivolan commented 1 year ago 
23-10-17 08:04:17.947 - INFO: Loading model for [../experiments/autoregressive.pth]
Traceback (most recent call last):
  File "/content/DL-Art-School/codes/train.py", line 398, in <module>
    trainer.init(args.opt, opt, args.launcher)
  File "/content/DL-Art-School/codes/train.py", line 146, in init
    self.model = ExtensibleTrainer(opt)
  File "/content/DL-Art-School/codes/trainer/ExtensibleTrainer.py", line 192, in __init__
    self.load()  # load networks from save states as needed
  File "/content/DL-Art-School/codes/trainer/ExtensibleTrainer.py", line 539, in load
    self.load_network(load_path, net, self.opt['path']['strict_load'], opt_get(self.opt, ['path', f'pretrain_base_path_{name}']))
  File "/content/DL-Art-School/codes/trainer/base_model.py", line 131, in load_network
    network.load_state_dict(load_net_clean, strict=strict)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UnifiedVoice:
    Unexpected key(s) in state_dict: "gpt.h.0.attn.bias", "gpt.h.0.attn.masked_bias", "gpt.h.1.attn.bias", "gpt.h.1.attn.masked_bias"

pip freeze:

absl-py==1.4.0
aiohttp==3.8.6
aiosignal==1.3.1
alabaster==0.7.13
albumentations==1.3.1
altair==4.2.2
antlr4-python3-runtime==4.9.3
anyio==3.7.1
appdirs==1.4.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
array-record==0.4.1
arviz==0.15.1
astropy==5.3.4
astunparse==1.6.3
async-timeout==4.0.3
attrs==23.1.0
audio2numpy==0.1.2
audioread==3.0.1
autograd==1.6.2
axial-positional-embedding==0.2.1
Babel==2.13.0
backcall==0.2.0
bcrypt==4.0.1
beartype==0.16.3
beautifulsoup4==4.11.2
bitsandbytes==0.41.1
bleach==6.1.0
blinker==1.4
blis==0.7.11
blosc2==2.0.0
bokeh==3.2.2
bqplot==0.12.40
branca==0.6.0
build==1.0.3
CacheControl==0.13.1
cachetools==5.3.1
catalogue==2.0.10
certifi==2023.7.22
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.0
chex==0.1.7
click==8.1.7
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==2.2.1
cmake==3.27.6
cmdstanpy==1.2.0
colorcet==3.0.1
colorlover==0.3.0
colour==0.1.5
CoLT5-attention==0.10.15
community==1.0.0b1
confection==0.1.3
cons==0.4.6
contextlib2==21.6.0
contourpy==1.1.1
cryptography==41.0.4
cufflinks==0.17.3
cupy-cuda11x==11.0.0
customtkinter==5.2.0
cvxopt==1.3.2
cvxpy==1.3.2
cycler==0.12.1
cymem==2.0.8
Cython==3.0.3
darkdetect==0.8.0
dask==2023.8.1
datascience==0.17.6
db-dtypes==1.1.1
dbus-python==1.2.18
debugpy==1.6.6
decorator==4.4.2
deepspeed==0.11.1
defusedxml==0.7.1
distributed==2023.8.1
distro==1.7.0
dlib==19.24.2
dm-tree==0.1.8
docutils==0.18.1
dopamine-rl==4.0.6
duckdb==0.8.1
earthengine-api==0.1.374
easydict==1.10
ecos==2.0.12
editdistance==0.6.2
eerepr==0.0.4
einops==0.7.0
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.6.0/en_core_web_sm-3.6.0-py3-none-any.whl#sha256=83276fc78a70045627144786b52e1f2728ad5e29e5e43916ec37ea9c26a11212
entrypoints==0.4
et-xmlfile==1.1.0
etils==1.5.0
etuples==0.3.9
exceptiongroup==1.1.3
fastai==2.7.12
fastcore==1.5.29
fastdownload==0.0.7
fastjsonschema==2.18.1
fastprogress==1.0.3
fastrlock==0.8.2
ffmpeg==1.4
filelock==3.12.4
Fiona==1.9.4.post1
firebase-admin==5.3.0
Flask==2.2.5
flatbuffers==23.5.26
flax==0.7.4
folium==0.14.0
fonttools==4.43.1
frozendict==2.3.8
frozenlist==1.4.0
fsspec==2023.6.0
ftfy==6.1.1
future==0.18.3
g-mlp-pytorch==0.1.5
gast==0.4.0
gcsfs==2023.6.0
GDAL==3.4.3
gdown==4.6.6
geemap==0.28.2
gensim==4.3.2
geocoder==1.38.1
geographiclib==2.0
geopandas==0.13.2
geopy==2.3.0
gin-config==0.5.0
glob2==0.7
google==2.0.3
google-api-core==2.11.1
google-api-python-client==2.84.0
google-auth==2.17.3
google-auth-httplib2==0.1.1
google-auth-oauthlib==1.0.0
google-cloud-bigquery==3.10.0
google-cloud-bigquery-connection==1.12.1
google-cloud-bigquery-storage==2.22.0
google-cloud-core==2.3.3
google-cloud-datastore==2.15.2
google-cloud-firestore==2.11.1
google-cloud-functions==1.13.3
google-cloud-language==2.9.1
google-cloud-storage==2.8.0
google-cloud-translate==3.11.3
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz#sha256=ba811295bb3b718bfa3fdc6d2467b4aedead25e00cecc3b1d17bdc9ba9d2cd1d
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.6.0
googleapis-common-protos==1.60.0
googledrivedownloader==0.4
graphviz==0.20.1
greenlet==3.0.0
grpc-google-iam-v1==0.12.6
grpcio==1.59.0
grpcio-status==1.48.2
gsa-pytorch==0.2.2
gspread==3.4.2
gspread-dataframe==3.3.1
gym==0.25.2
gym-notices==0.0.8
h5netcdf==1.2.0
h5py==3.9.0
hjson==3.1.0
holidays==0.34
holoviews==1.17.1
html5lib==1.1
httpimport==1.3.1
httplib2==0.22.0
huggingface-hub==0.17.3
humanize==4.7.0
hyperopt==0.2.7
idna==3.4
imageio==2.31.5
imageio-ffmpeg==0.4.9
imagesize==1.4.1
imbalanced-learn==0.10.1
imgaug==0.4.0
importlib-metadata==6.8.0
importlib-resources==6.1.0
imutils==0.5.4
inflect==7.0.0
iniconfig==2.0.0
intel-openmp==2023.2.0
ipyevents==2.0.2
ipyfilechooser==0.6.0
ipykernel==5.5.6
ipyleaflet==0.17.4
ipython==7.34.0
ipython-genutils==0.2.0
ipython-sql==0.5.0
ipytree==0.2.2
ipywidgets==7.7.1
itsdangerous==2.1.2
jax==0.4.16
jaxlib @ https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.4.16+cuda11.cudnn86-cp310-cp310-manylinux2014_x86_64.whl#sha256=78b3a9acfda4bfaae8a1dc112995d56454020f5c02dba4d24c40c906332efd4a
jeepney==0.7.1
jieba==0.42.1
Jinja2==3.1.2
jiwer==3.0.3
joblib==1.3.2
jsonpickle==3.0.2
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
jupyter-client==6.1.12
jupyter-console==6.1.0
jupyter-server==1.24.0
jupyter_core==5.4.0
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.9
kaggle==1.5.16
keras==2.13.1
keyring==23.5.0
kiwisolver==1.4.5
kornia==0.7.0
lambda-networks==0.4.0
langcodes==3.3.0
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
lazy_loader==0.3
libclang==16.0.6
librosa==0.10.1
lightgbm==4.0.0
linear-attention-transformer==0.19.1
linformer==0.2.1
linkify-it-py==2.0.2
lion-pytorch==0.0.7
lit==17.0.2
llvmlite==0.39.1
local-attention==1.8.6
locket==1.0.0
logical-unification==0.4.6
lxml==4.9.3
malloy==2023.1056
Markdown==3.5
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.7.1
matplotlib-inline==0.1.6
matplotlib-venn==0.11.9
mdit-py-plugins==0.4.0
mdurl==0.1.2
miniKanren==1.0.3
missingno==0.5.2
mistune==0.8.4
mizani==0.9.3
mkl==2023.2.0
ml-dtypes==0.3.1
mlxtend==0.22.0
more-itertools==10.1.0
moviepy==1.0.3
mpmath==1.3.0
msgpack==1.0.7
multidict==6.0.4
multipledispatch==1.0.0
multitasking==0.0.11
munch==4.0.0
mup==1.0.0
murmurhash==1.0.10
music21==9.1.0
natsort==8.4.0
nbclassic==1.0.0
nbclient==0.8.0
nbconvert==6.5.4
nbformat==5.9.2
nest-asyncio==1.5.8
networkx==3.1
nibabel==4.0.2
ninja==1.11.1.1
nltk==3.8.1
notebook==6.5.5
notebook_shim==0.2.3
numba==0.56.4
numexpr==2.8.7
numpy==1.23.5
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.18.1
nvidia-nvjitlink-cu12==12.2.140
nvidia-nvtx-cu12==12.1.105
oauth2client==4.1.3
oauthlib==3.2.2
omegaconf==2.3.0
opencv-contrib-python==4.8.0.76
opencv-python==4.8.0.76
opencv-python-headless==4.8.1.78
openpyxl==3.1.2
opt-einsum==3.3.0
optax==0.1.7
orbax-checkpoint==0.4.1
orjson==3.9.9
osqp==0.6.2.post8
packaging==23.2
pandas==1.5.3
pandas-datareader==0.10.0
pandas-gbq==0.17.9
pandas-stubs==1.5.3.230304
pandocfilters==1.5.0
panel==1.2.3
param==1.13.0
paramiko==3.3.1
parso==0.8.3
partd==1.4.1
pathlib==1.0.1
pathy==0.10.2
patsy==0.5.3
peewee==3.16.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.4.0
pip-tools==6.13.0
platformdirs==3.11.0
plotly==5.15.0
plotnine==0.12.3
pluggy==1.3.0
polars==0.17.3
pooch==1.7.0
portpicker==1.5.2
prefetch-generator==1.0.3
preshed==3.0.9
prettytable==3.9.0
product-key-memory==0.2.10
proglog==0.1.10
progressbar2==4.2.0
prometheus-client==0.17.1
promise==2.3
prompt-toolkit==3.0.39
prophet==1.1.5
proto-plus==1.22.3
protobuf==3.20.3
psutil==5.9.5
psycopg2==2.9.9
ptyprocess==0.7.0
py-cpuinfo==9.0.0
py4j==0.10.9.7
pyarrow==9.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycocotools==2.0.7
pycparser==2.21
pyct==0.5.0
pydantic==1.10.13
pydata-google-auth==1.8.2
pydot==1.4.2
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
PyDrive2==1.6.3
pyerfa==2.0.0.3
pygame==2.5.2
Pygments==2.16.1
PyGObject==3.42.1
PyJWT==2.3.0
pymc==5.7.2
pymystem3==0.2.0
PyNaCl==1.5.0
PyOpenGL==3.1.7
pyOpenSSL==23.2.0
pyparsing==3.1.1
pyperclip==1.8.2
pyproj==3.6.1
pyproject_hooks==1.0.0
pyshp==2.3.1
PySocks==1.7.1
pytensor==2.14.2
pytest==7.4.2
python-apt==0.0.0
python-box==7.1.1
python-dateutil==2.8.2
python-louvain==0.16
python-slugify==8.0.1
python-utils==3.8.1
pytorch-fid==0.3.0
pytorch-ssim==0.1
pytz==2023.3.post1
pyviz_comms==3.0.0
PyWavelets==1.4.1
pyworld==0.3.4
PyYAML==6.0.1
pyzmq==23.2.1
qdldl==0.1.7.post0
qudida==0.0.4
rapidfuzz==3.4.0
ratelim==0.1.6
referencing==0.30.2
regex==2023.6.3
requests==2.31.0
requests-oauthlib==1.3.1
requirements-parser==0.5.0
rich==13.6.0
rotary-embedding-torch==0.3.2
rpds-py==0.10.4
rpy2==3.4.2
rsa==4.9
ruamel.yaml==0.17.35
ruamel.yaml.clib==0.2.8
safetensors==0.4.0
scikit-image==0.19.3
scikit-learn==1.2.2
scipy==1.11.3
scooby==0.7.4
scp==0.14.5
scs==3.2.3
seaborn==0.12.2
SecretStorage==3.3.1
Send2Trash==1.8.2
shapely==2.0.1
six==1.16.0
sklearn-pandas==2.2.0
smart-open==6.4.0
sniffio==1.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soundfile==0.12.1
soupsieve==2.5
soxr==0.3.7
spacy==3.6.1
spacy-legacy==3.0.12
spacy-loggers==1.0.5
Sphinx==5.0.2
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-serializinghtml==1.1.9
SQLAlchemy==2.0.21
sqlparse==0.4.4
srsly==2.4.8
stanio==0.3.0
statsmodels==0.14.0
sympy==1.12
tables==3.8.0
tabulate==0.9.0
tb-nightly==2.15.0a20231016
tbb==2021.10.0
tblib==2.0.0
tenacity==8.2.3
tensorboard==2.13.0
tensorboard-data-server==0.7.1
tensorflow==2.13.0
tensorflow-datasets==4.9.3
tensorflow-estimator==2.13.0
tensorflow-gcs-config==2.13.0
tensorflow-hub==0.15.0
tensorflow-io-gcs-filesystem==0.34.0
tensorflow-metadata==1.14.0
tensorflow-probability==0.20.1
tensorstore==0.1.45
termcolor==2.3.0
terminado==0.17.1
text-unidecode==1.3
textblob==0.17.1
tf-slim==1.1.0
tgt==1.4.4
thinc==8.1.12
threadpoolctl==3.2.0
tifffile==2023.9.26
tinycss2==1.2.1
tokenizers==0.14.1
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
torch==2.1.0
torchaudio==2.1.0
torchdata==0.6.1
torchsummary==1.5.1
torchtext==0.15.2
torchvision==0.16.0
tornado==6.3.2
tqdm==4.66.1
traitlets==5.7.1
traittypes==0.2.1
transformers==4.34.0
triton==2.1.0
tweepy==4.13.0
typer==0.9.0
types-pytz==2023.3.1.1
types-setuptools==68.2.0.0
typing_extensions==4.5.0
tzlocal==5.1
uc-micro-py==1.0.2
Unidecode==1.3.7
uritemplate==4.1.1
urllib3==2.0.6
vector-quantize-pytorch==1.9.14
vega-datasets==0.9.0
wadllib==1.3.6
wasabi==1.1.2
wcwidth==0.2.8
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.4
Werkzeug==3.0.0
widgetsnbextension==3.6.6
wordcloud==1.9.2
wrapt==1.15.0
x-clip==0.14.4
x-transformers==1.0.4
xarray==2023.7.0
xarray-einstats==0.6.0
xgboost==2.0.0
xlrd==2.0.1
xyzservices==2023.10.0
yarl==1.9.2
yellowbrick==1.5
yfinance==0.2.31
zict==3.0.0
zipp==3.17.0

nvidia-smi:

Tue Oct 17 08:09:34 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   42C    P8    11W /  70W |      0MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
pawanhv commented 12 months ago

I get the same error

SamuelEnzi commented 4 months ago

It might be a bit late but setting line 131 in \codes\trainer\base_model.py from network.load_state_dict(load_net_clean, strict=strict) to network.load_state_dict(load_net_clean, strict=False)

Fixed it for me. I dont know why tho