Closed danyaljj closed 2 years ago
$ python3 to_hf_weights.py --input-ckpt gs://danielk-files/gpt-j-checkpoints_slim/step_318500 --config configs/6B_roto_256.json --output-path gs://danielk-files/danielk-files/gpt-j-checkpoints_slim_hf/step_318500 --dtype fp32 --cpu Traceback (most recent call last): File "to_hf_weights.py", line 21, in <module> import torch File "/usr/local/lib/python3.8/dist-packages/torch/__init__.py", line 198, in <module> _load_global_deps() File "/usr/local/lib/python3.8/dist-packages/torch/__init__.py", line 151, in _load_global_deps ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL) File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__ self._handle = _dlopen(self._name, mode) OSError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory
Here is the environment:
Package Version ---------------------------- -------------------- absl-py 1.0.0 aiohttp 3.8.1 aiohttp-cors 0.7.0 aioredis 2.0.1 aiosignal 1.2.0 anyio 3.5.0 asgiref 3.5.0 astunparse 1.6.3 async-timeout 4.0.2 attrs 19.3.0 Automat 0.8.0 bcrypt 3.2.0 best-download 0.0.9 black 22.1.0 blessings 1.7 blinker 1.4 cachetools 4.2.4 certifi 2021.10.8 cffi 1.15.0 chardet 3.0.4 charset-normalizer 2.0.11 chex 0.1.1 clang 5.0 click 8.0.4 cloud-init 21.4 cloud-tpu-client 0.10 cloudpickle 1.3.0 colorama 0.4.3 colorful 0.5.4 command-not-found 0.3 configobj 5.0.6 constantly 15.1.0 cryptography 2.8 Cython 0.29.14 DataProperty 0.55.0 datasets 1.15.1 dbus-python 1.2.16 Deprecated 1.2.13 dill 0.3.4 distlib 0.3.4 distro 1.4.0 distro-info 0.23ubuntu1 dm-haiku 0.0.5 dm-tree 0.1.6 docker-pycreds 0.4.0 dyNET38 2.1 einops 0.3.2 entrypoints 0.3 fabric 2.6.0 fastapi 0.75.0 filelock 3.4.2 Flask 1.1.4 flatbuffers 1.12 frozenlist 1.3.0 fsspec 2022.2.0 ftfy 6.1.1 func-timeout 4.3.5 future 0.18.2 gast 0.4.0 gitdb 4.0.9 GitPython 3.1.27 google-api-core 1.31.5 google-api-python-client 1.8.0 google-auth 1.35.0 google-auth-httplib2 0.1.0 google-auth-oauthlib 0.4.6 google-cloud-core 1.7.2 google-cloud-storage 1.36.2 google-crc32c 1.3.0 google-pasta 0.2.0 google-resumable-media 1.3.3 googleapis-common-protos 1.54.0 gpustat 0.6.0 grpcio 1.43.0 h11 0.13.0 h5py 3.1.0 httplib2 0.20.2 huggingface-hub 0.4.0 hyperlink 19.0.0 idna 2.10 importlib-metadata 4.10.1 incremental 16.10.1 iniconfig 1.1.1 intel-openmp 2022.0.2 invoke 1.7.0 itsdangerous 1.1.0 jax 0.2.12 jaxlib 0.1.68 jieba 0.42.1 Jinja2 2.10.1 jmp 0.0.2 joblib 1.1.0 jsonlines 2.0.0 jsonpatch 1.22 jsonpointer 2.0 jsonpointer 2.0 jsonschema 3.2.0 keras 2.6.0 Keras-Applications 1.0.8 Keras-Preprocessing 1.1.2 keyring 18.0.1 language-selector 0.1 launchpadlib 1.10.13 lazr.restfulclient 0.14.2 lazr.uri 1.0.3 libclang 13.0.0 lm-dataformat 0.0.20 lm-eval 0.2.0 Markdown 3.3.6 MarkupSafe 1.1.0 mbstrdecoder 1.1.0 mkl 2022.0.2 mkl-include 2022.0.2 mock 4.0.3 more-itertools 4.2.0 msgfy 0.2.0 msgpack 1.0.3 multidict 6.0.2 multiprocess 0.70.12.2 mypy-extensions 0.4.3 nagisa 0.2.7 netifaces 0.10.4 nltk 3.7 numexpr 2.7.2 numpy 1.19.5 nvidia-ml-py3 7.352.0 oauth2client 4.1.3 oauthlib 3.1.0 openai 0.6.4 opencensus 0.8.0 opencensus-context 0.1.2 opt-einsum 3.3.0 optax 0.0.9 packaging 21.3 pandas 1.4.1 paramiko 2.10.3 pathlib2 2.3.7.post1 pathspec 0.9.0 pathtools 0.1.2 pathvalidate 2.5.0 pathy 0.6.1 pexpect 4.6.0 Pillow 9.0.0 pip 20.0.2 platformdirs 2.4.1 pluggy 0.13.1 portalocker 2.4.0 prometheus-client 0.13.1 promise 2.3 protobuf 3.19.4 psutil 5.9.0 py 1.11.0 py-spy 0.3.11 pyarrow 7.0.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybind11 2.6.2 pycountry 20.7.3 pycparser 2.21 pydantic 1.9.0 PyGObject 3.36.0 PyHamcrest 1.9.0 PyJWT 1.7.1 pymacaroons 0.13.0 PyNaCl 1.3.0 pyOpenSSL 19.0.0 pyparsing 3.0.7 pyrsistent 0.15.5 pyserial 3.4 pytablewriter 0.58.0 pytest 6.2.3 python-apt 2.0.0+ubuntu0.20.4.6 python-dateutil 2.8.2 python-debian 0.1.36ubuntu1 pytz 2021.3 PyYAML 5.4.1 ray 1.4.1 redis 4.1.4 regex 2022.3.15 rehash 1.0.0 requests 2.25.1 requests-oauthlib 1.3.1 requests-unixsocket 0.2.0 rouge-score 0.0.4 rsa 4.8 sacrebleu 1.5.0 sacremoses 0.0.49 scikit-learn 1.0.2 scipy 1.8.0 SecretStorage 2.3.1 sentry-sdk 1.5.8 service-identity 18.1.0 setproctitle 1.2.2 setuptools 60.7.0 shortuuid 1.0.8 simplejson 3.16.0 six 1.16.0 smart-open 5.2.1 smmap 5.0.0 sniffio 1.2.0 sos 4.1 sqlitedict 1.6.0 ssh-import-id 5.10 starlette 0.17.1 systemd-python 234 tabledata 1.3.0 tabulate 0.8.9 tbb 2021.5.1 tcolorpy 0.1.2 tensorboard 2.6.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorflow 2.8.0 tensorflow-cpu 2.6.3 tensorflow-estimator 2.6.0 tensorflow-hub 0.12.0 tensorflow-io-gcs-filesystem 0.23.1 tensorflow-text 2.8.0rc0 termcolor 1.1.0 threadpoolctl 3.1.0 tokenizers 0.11.6 toml 0.10.2 tomli 2.0.1 toolz 0.11.2 torch 1.11.0a0+git06838ce torch-xla 1.11+3413bd1 torchvision 0.12.0a0+563d9ca tqdm 4.63.0 tqdm-multiprocess 0.0.11 transformers 4.17.0 Twisted 18.9.0 typepy 1.3.0 typer 0.4.0 typing-extensions 4.0.1 ubuntu-advantage-tools 27.5 ufw 0.36 ujson 5.1.0 unattended-upgrades 0.1 uritemplate 3.0.1 urllib3 1.26.8 uvicorn 0.17.6 virtualenv 20.13.0 wadllib 1.3.3 wandb 0.12.11 wcwidth 0.2.5 Werkzeug 1.0.1 wheel 0.37.1 wrapt 1.12.1 xxhash 3.0.0 yarl 1.7.2 yaspin 2.1.0 zipp 1.0.0 zope.interface 4.7.1 zstandard 0.15.2
and
$ python3 --version Python 3.8.10 $ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04"
Update: This should do the trick:
export LD_LIBRARY_PATH=/usr/local/lib
More details.
Here is the environment:
and