Models fail to load

[peter-ciccolo]

I'm getting the following error in the final step of the quickstart:

unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file

Input line: ./main -m models/bigcode/gpt_bigcode-santacoder-ggml.bin -p "def fibonnaci(" --top_k 0 --top_p 0.95 --temp 0.2

Output:

main: seed = 1687068338
starcoder_model_load: loading model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49280
starcoder_model_load: n_ctx   = 2048
starcoder_model_load: n_embd  = 2048
starcoder_model_load: n_head  = 16
starcoder_model_load: n_layer = 24
starcoder_model_load: ftype   = 1003
starcoder_model_load: qntvr   = 1
starcoder_model_load: ggml ctx size = 1794.97 MB
starcoder_model_load: memory size =   768.00 MB, n_mem = 49152
starcoder_model_load: unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file
main: failed to load model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'

Notable differences from the sample output:

• starcoder_model_load: ftype = 1 in my output vs starcoder_model_load: ftype = 3 (quanitzed models were produced with ./quantize models/bigcode/gpt_bigcode-santacoder-ggml.bin models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin 3; non-quanitzed model fails with a similar error)
• starcoder_model_load: qntvr = 1 in my output vs. no info on qntvr in the sample output

Other notes:

• this is running on a 2019 Intel MBP, not an M1
• conda list is reproduced below in case I'm somehow missing a dependency

  # Name                    Version                   Build  Channel
  accelerate                0.20.3             pyhd8ed1ab_0    conda-forge
  blas                      1.0                         mkl
  brotlipy                  0.7.0           py310hca72f7f_1002
  bzip2                     1.0.8                h1de35cc_0
  ca-certificates           2023.5.7             h8857fd0_0    conda-forge
  certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
  cffi                      1.15.1          py310h6c40b1e_3
  charset-normalizer        2.0.4              pyhd3eb1b0_0
  click                     8.0.4           py310hecd8cb5_0
  cryptography              39.0.1          py310hf6deb26_2
  dataclasses               0.8                pyh6d0b6a4_7
  filelock                  3.9.0           py310hecd8cb5_0
  future                    0.18.3          py310hecd8cb5_0
  huggingface_hub           0.15.1                     py_0    huggingface
  idna                      3.4             py310hecd8cb5_0
  importlib-metadata        6.0.0           py310hecd8cb5_0
  importlib_metadata        6.0.0                hd3eb1b0_0
  intel-openmp              2023.1.0         ha357a0b_43547
  joblib                    1.2.0           py310hecd8cb5_0
  libcxx                    14.0.6               h9765a3e_0
  libffi                    3.4.4                hecd8cb5_0
  libgfortran               5.0.0           11_3_0_hecd8cb5_28
  libgfortran5              11.3.0              h9dfd629_28
  libopenblas               0.3.21               h54e7dc3_0
  libprotobuf               3.20.3               hfff2838_0
  libuv                     1.44.2               h6c40b1e_0
  llvm-openmp               14.0.6               h0dcd299_0
  mkl                       2023.1.0         h59209a4_43558
  mkl-service               2.4.0           py310h6c40b1e_1
  mkl_fft                   1.3.6           py310h3ea8b11_1
  mkl_random                1.2.2           py310h3ea8b11_1
  ncurses                   6.4                  hcec6c5f_0
  ninja                     1.10.2               hecd8cb5_5
  ninja-base                1.10.2               haf03e11_5
  numpy                     1.24.3          py310h827a554_1
  numpy-base                1.24.3          py310ha186be2_1
  openssl                   3.1.1                h8a1eda9_1    conda-forge
  packaging                 23.0            py310hecd8cb5_0
  pip                       23.1.2          py310hecd8cb5_0
  protobuf                  3.20.3          py310hcec6c5f_0
  psutil                    5.9.5           py310h90acd4f_0    conda-forge
  pycparser                 2.21               pyhd3eb1b0_0
  pyopenssl                 23.0.0          py310hecd8cb5_0
  pysocks                   1.7.1           py310hecd8cb5_0
  python                    3.10.11              h5ee71fb_3
  python_abi                3.10                    2_cp310    conda-forge
  pytorch                   1.13.1          cpu_py310h9e40b02_0
  pyyaml                    6.0             py310h6c40b1e_1
  readline                  8.2                  hca72f7f_0
  regex                     2022.7.9        py310hca72f7f_0
  requests                  2.29.0          py310hecd8cb5_0
  sacremoses                master                     py_0    huggingface
  setuptools                67.8.0          py310hecd8cb5_0
  six                       1.16.0             pyhd3eb1b0_1
  sqlite                    3.41.2               h6c40b1e_0
  tbb                       2021.8.0             ha357a0b_0
  tk                        8.6.12               h5d9f67b_0
  tokenizers                0.11.4          py310h8776b5c_1
  tqdm                      4.65.0          py310h20db666_0
  transformers              4.28.1                     py_0    huggingface
  typing-extensions         4.6.3           py310hecd8cb5_0
  typing_extensions         4.6.3           py310hecd8cb5_0
  tzdata                    2023c                h04d1e81_0
  urllib3                   1.26.16         py310hecd8cb5_0
  wheel                     0.38.4          py310hecd8cb5_0
  xz                        5.4.2                h6c40b1e_0
  yaml                      0.2.5                haf1e3a3_0
  zipp                      3.11.0          py310hecd8cb5_0
  zlib                      1.2.13               h4dc903c_0