search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
135.81k stars 27.19k forks source link

RWKV4neo #20737

Closed ArEnSc closed 1 year ago

ArEnSc commented 1 year ago

Model description

RWKV - Receptance Weighted Key Value

RWKV is a Sequence to Sequence Model that takes the best features of Generative PreTraining (GPT) and Recurrent Nueral Networks (RNN) that performs Language Modelling (LM). This is used to generate text Auto Regressive manner (AR).

This is a hybrid model.

It has Transformer Level Performance without the quadratic attention mechanism. It borrows ideas from Attention Free Transformers, meaning the attention is a linear in complexity. Allowing for infinite context through the hidden state in RWKV_RNN.

There are two models for RWKV, they are refered to as modes.

RWKV_RNN: This mode is designed for running inference quickly. RWKV_GPT: This mode is for training or fine tuning your model quickly.

In the first pass we will be implementing RWKV_RNN Although we can weight share to have RWKV_GPT generate the inital context for RWKV_RNN.

Open source status

More from the Research and Development Repository: https://github.com/BlinkDL/RWKV-LM

ArEnSc commented 1 year ago

@ArthurZucker @younesbelkada https://github.com/huggingface/transformers/issues/17230#issuecomment-1346779059

younesbelkada commented 1 year ago

This is super cool !! 🔥 Really looking forward to it Also as discussed with @leondz , we'll probably also include a blogpost explaining this new architecture! Happy to help for the implementation and the blogpost!

ArEnSc commented 1 year ago

Ok I am gonna just set this up on my linux machine m1 setup isn't ready I spent 2 hours on this gonna try again tomorrow sorry D:

ArEnSc commented 1 year ago

Ok I am gonna just set this up on my linux machine m1 setup isn't ready I spent 2 hours on this gonna try again tomorrow sorry D:

scratch that, it also didn't work, mainly because of my limited harddisk space. Gonna retry mac...

ArEnSc commented 1 year ago 
Building wheels for collected packages: transformers, onnx
  Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.26.0.dev0-0.editable-py3-none-any.whl size=31899 sha256=d4b123bfb17b8f11ab811da95d253a066a98638c492d76f4d00afa509dfc6d71
  Stored in directory: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-ephem-wheel-cache-w5ucgfda/wheels/52/2c/02/9b0e2ee52910e61c69011870086c52ab4eaaa554c34005f48f
  Building wheel for onnx (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [76 lines of output]

      You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      running bdist_wheel
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']
      -- The C compiler identification is unknown
      -- The CXX compiler identification is unknown
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - failed
      -- Check for working C compiler: /usr/bin/cc
      -- Check for working C compiler: /usr/bin/cc - broken
      CMake Error at /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/cmake/data/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
        The C compiler

          "/usr/bin/cc"

        is not able to compile a simple test program.

        It fails with the following output:

          Change Dir: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeScratch/TryCompile-cd2Kpc

          Run Build Command(s):/usr/bin/make -f Makefile cmTC_a6349/fast &&
          You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.

        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:17 (project)

      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for onnx
  Running setup.py clean for onnx
Successfully built transformers
Failed to build onnx
Installing collected packages: onnx, tf2onnx, nbformat, matplotlib, markdown, jaxlib, jax, huggingface-hub, gql, google-auth, GitPython, Flask, docker, cryptography, cliff, botocore, arrow, alembic, aiohttp, accelerate, transformers, timm, s3transfer, pyOpenSSL, optuna, librosa, kubernetes, jinja2-time, ipython, google-auth-oauthlib, dash, csvw, chex, APScheduler, tensorboard, optax, hf-doc-builder, datasets, dash-bootstrap-components, cookiecutter, clldutils, boto3, tensorflow-macos, sigopt, segments, flax, evaluate, codecarbon, phonemizer
  Attempting uninstall: onnx
    Found existing installation: onnx 1.13.0
    Uninstalling onnx-1.13.0:
      Successfully uninstalled onnx-1.13.0
  Running setup.py install for onnx ... error
  error: subprocess-exited-with-error

  × Running setup.py install for onnx did not run successfully.
  │ exit code: 1
  ╰─> [78 lines of output]

      You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      running install
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']
      -- The C compiler identification is unknown
      -- The CXX compiler identification is unknown
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - failed
      -- Check for working C compiler: /usr/bin/cc
      -- Check for working C compiler: /usr/bin/cc - broken
      CMake Error at /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/cmake/data/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
        The C compiler

          "/usr/bin/cc"

        is not able to compile a simple test program.

        It fails with the following output:

          Change Dir: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeScratch/TryCompile-WGiugk

          Run Build Command(s):/usr/bin/make -f Makefile cmTC_3370b/fast &&
          You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.

        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:17 (project)

      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
          return orig.install.run(self)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/install.py", line 546, in run
          self.run_command('build')
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of onnx
  Moving to /Users/michaelchung/Code/transformers/.env/bin/backend-test-tools
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/backend-test-tools
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-model
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/check-model
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-node
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/check-node
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx-1.13.0.dist-info/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx-1.13.0.dist-info
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> onnx

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Fails at building wheels for transformers and onnx I installed onnx manually using pip3 

absl-py==1.3.0
aiosignal==1.3.1
appdirs==1.4.4
appnope==0.1.3
asttokens==2.2.1
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.1.0
audioread==3.0.0
autopage==0.5.1
Babel==2.11.0
backcall==0.2.0
backoff==1.11.1
beautifulsoup4==4.11.1
binaryornot==0.4.4
black==22.3.0
cachetools==5.2.0
certifi==2022.12.7
cffi==1.15.1
chardet==5.1.0
charset-normalizer==2.1.1
click==8.1.3
cmaes==0.9.0
cmake==3.25.0
cmd2==2.4.2
colorama==0.4.6
coloredlogs==15.0.1
colorlog==6.7.0
commonmark==0.9.1
contourpy==1.0.6
cycler==0.11.0
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
decorator==5.1.1
dill==0.3.4
distlib==0.3.6
dlinfo==1.2.1
dm-tree==0.1.7
exceptiongroup==1.0.4
execnet==1.9.0
executing==1.2.0
faiss-cpu==1.7.3
fastjsonschema==2.16.2
filelock==3.8.2
fire==0.5.0
flake8==6.0.0
flatbuffers==2.0.7
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2022.11.0
fugashi==1.1.2a6
gast==0.4.0
gitdb==4.0.10
google-pasta==0.2.0
graphql-core==3.2.3
grpcio==1.51.1
h5py==3.7.0
humanfriendly==10.0
hypothesis==6.61.0
idna==3.4
importlib-metadata==4.13.0
iniconfig==1.1.1
ipadic==1.0.0
isodate==0.6.1
isort==5.11.2
itsdangerous==2.1.2
jedi==0.18.2
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==4.17.3
jupyter_core==5.1.0
kenlm==0.1
keras==2.11.0
keras-nlp==0.3.1
kiwisolver==1.4.4
language-tags==1.1.0
libclang==14.0.6
llvmlite==0.39.1
lxml==4.9.2
Mako==1.2.4
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
mccabe==0.7.0
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.3
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nltk==3.8
numba==0.56.4
numpy==1.23.5
oauthlib==3.2.2
onnx==1.13.0
onnxconverter-common==1.13.0
onnxruntime==1.13.1
onnxruntime-tools==1.7.0
opt-einsum==3.3.0
packaging==22.0
pandas==1.5.2
parameterized==0.8.1
parso==0.8.3
pathspec==0.10.3
pbr==5.11.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
Pint==0.16.1
plac==1.3.5
platformdirs==2.6.0
plotly==5.11.0
pluggy==1.0.0
pooch==1.6.0
portalocker==2.0.0
poyo==0.5.0
prettytable==3.5.0
prompt-toolkit==3.0.36
protobuf==3.19.6
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
py3nvml==0.2.7
pyarrow==10.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.10.0
pycparser==2.21
pyctcdecode==0.4.0
pyflakes==3.0.1
Pygments==2.13.0
pygtrie==2.5.0
pyknp==0.6.1
pylatexenc==2.10
pynvml==11.4.1
pyparsing==3.0.9
pyperclip==1.8.2
pypng==0.20220715.0
pyrsistent==0.19.2
pytest==7.2.0
pytest-timeout==2.1.0
pytest-xdist==3.1.0
python-dateutil==2.8.2
python-slugify==7.0.0
pytz==2022.6
pytz-deprecation-shim==0.1.0.post0
PyYAML==5.4.1
ray==2.2.0
rdflib==6.2.0
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.10.1
resampy==0.4.2
responses==0.18.0
rfc3986==1.5.0
rich==11.2.0
rjieba==0.1.11
rouge-score==0.1.2
rsa==4.9
sacrebleu==1.5.1
sacremoses==0.0.53
safetensors==0.2.6
scikit-learn==1.2.0
scipy==1.8.1
sentencepiece==0.1.97
six==1.16.0
smmap==5.0.0
sortedcontainers==2.4.0
soundfile==0.11.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.45
stack-data==0.6.2
stevedore==4.1.1
SudachiDict-core==20221021
SudachiPy==0.6.6
sympy==1.11.1
tabulate==0.9.0
tenacity==8.1.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5.1
tensorflow-estimator==2.11.0
tensorflow-metal==0.7.0
tensorstore==0.1.28
termcolor==2.1.1
text-unidecode==1.3
threadpoolctl==3.1.0
timeout-decorator==0.5.0
tokenizers==0.13.2
tomli==2.0.1
toolz==0.12.0
torch==1.13.0
torchaudio==0.13.0
torchvision==0.14.0
tqdm==4.64.1
traitlets==5.7.1
typing_extensions==4.4.0
tzdata==2022.7
tzlocal==4.2
unidic==1.1.0
unidic-lite==1.0.8
uritemplate==4.1.1
urllib3==1.26.13
virtualenv==20.17.1
wasabi==0.10.1
wcwidth==0.2.5
websocket-client==1.4.2
Werkzeug==2.2.2
wrapt==1.14.1
xmltodict==0.13.0
xxhash==3.1.0
yarl==1.8.2
zipp==3.11.0

I am on 3.9.11

when running conda install -c conda-forge onnxruntime it says all the components were installed

ArEnSc commented 1 year ago

hmmm I accepted the xcode license agreement. It's built transformers I think it fails to build onnx, but whats odd is it uninstalls onnx. Maybe I can do with out it and remove it out of setup.py?

Building wheels for collected packages: transformers, onnx
  Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.26.0.dev0-0.editable-py3-none-any.whl size=31899 sha256=e63397c8685e05971af9d4767e872200eb8393ed081e37582f8d24760daaca08
  Stored in directory: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-ephem-wheel-cache-w75hif9g/wheels/52/2c/02/9b0e2ee52910e61c69011870086c52ab4eaaa554c34005f48f
  Building wheel for onnx (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [67 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      running bdist_wheel
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']
      -- The C compiler identification is AppleClang 14.0.0.14000029
      -- The CXX compiler identification is AppleClang 14.0.0.14000029
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found PythonInterp: /Users/michaelchung/Code/transformers/.env/bin/python3 (found version "3.9.11")
      -- Found PythonLibs: /opt/homebrew/Frameworks/Python.framework/Versions/3.11/lib/libpython3.11.dylib (found version "3.9.11")
      Generated: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/onnx/onnx-ml.proto
      CMake Error at CMakeLists.txt:299 (message):
        Protobuf compiler not found
      Call Stack (most recent call first):
        CMakeLists.txt:330 (relative_protobuf_generate_cpp)

      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for onnx
  Running setup.py clean for onnx
Successfully built transformers
Failed to build onnx
Installing collected packages: onnx, tf2onnx, nbformat, matplotlib, markdown, jaxlib, jax, huggingface-hub, gql, google-auth, GitPython, Flask, docker, cryptography, cliff, botocore, arrow, alembic, aiohttp, accelerate, transformers, timm, s3transfer, pyOpenSSL, optuna, librosa, kubernetes, jinja2-time, ipython, google-auth-oauthlib, dash, csvw, chex, APScheduler, tensorboard, optax, hf-doc-builder, datasets, dash-bootstrap-components, cookiecutter, clldutils, boto3, tensorflow-macos, sigopt, segments, flax, evaluate, codecarbon, phonemizer
  Attempting uninstall: onnx
    Found existing installation: onnx 1.13.0
    Uninstalling onnx-1.13.0:
      Successfully uninstalled onnx-1.13.0
  Running setup.py install for onnx ... error
  error: subprocess-exited-with-error

  × Running setup.py install for onnx did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      running install
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']
      Generated: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/onnx/onnx-ml.proto
      CMake Error at CMakeLists.txt:299 (message):
        Protobuf compiler not found
      Call Stack (most recent call first):
        CMakeLists.txt:330 (relative_protobuf_generate_cpp)

      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
          return orig.install.run(self)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/install.py", line 546, in run
          self.run_command('build')
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of onnx
  Moving to /Users/michaelchung/Code/transformers/.env/bin/backend-test-tools
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/backend-test-tools
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-model
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/check-model
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-node
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/check-node
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx-1.13.0.dist-info/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx-1.13.0.dist-info
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> onnx

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
ArEnSc commented 1 year ago

Alright dev environment installed, the key here was not to use conda but miniforge

ArEnSc commented 1 year ago

Ok draft created

vonjackustc commented 1 year ago

Do you know how to convert .pth model to config.json/pytorch_model.bin for RWKV4neo?

tensorpro commented 1 year ago

I have a conversion script + draft that has consistent logit ordering with the official implementation here:

conversion script: https://github.com/tensorpro/transformers/blob/rwkv_draft/src/transformers/models/rwkv4_neo/convert_rwkv_original_pytorch_checkpoint_to_pytorch.py model in torch: https://github.com/tensorpro/transformers/blob/rwkv_draft/src/transformers/models/rwkv4_neo/modeling_rwkv4_neo.py

I can clean it up and turn it into a PR if that would help?

ArthurZucker commented 1 year ago

Sure, there is also #22797 that should be in a pretty good state! I'm about to review it but feel free to add your touch to it if you feel like it!

tensorpro commented 1 year ago

Oh cool, that one looks awesome!

YovaKem commented 1 year ago

I have a conversion script + draft that has consistent logit ordering with the official implementation here:

conversion script: https://github.com/tensorpro/transformers/blob/main/src/transformers/models/rwkv4_neo/convert_rwkv_original_pytorch_checkpoint_to_pytorch.py model in torch: https://github.com/tensorpro/transformers/blob/main/src/transformers/models/rwkv4_neo/modeling_rwkv4_neo.py

I can clean it up and turn it into a PR if that would help?

@tensorpro I could really use your scripts but I get 404 when I try to access those links. :/

tensorpro commented 1 year ago

Ah sorry, the links broke when I changed the branch I was working in. I edited the comment to point to the right branch

That said, you may want to use the code in #22797 since it will be closer to the official HF version and already supports CUDA accelerated WKV.