Open zyren123 opened 3 weeks ago
额,你这个llavanext吧?和llava是不一样的。(我不确定奥)
额,你这个llavanext吧?和llava是不一样的。(我不确定奥)
llava 和llavanext我都试啦,这个其实就是llava1.5和llava1.6, 1.6主要就是预处理的图像size不太一样。原版的llava我也试了,也是一样的错误才去试llavanext的
额,你这个llavanext吧?和llava是不一样的。(我不确定奥)
请问您的transformers版本是多少哇,是不是这个的原因,我直接down下来notebook来运行qwen4b 一行一行执行也会报同样的错误,感觉是环境的问题。
看了一下,可能是transformers版本不一样,我这里是transformers==4.41.1
你升级一下,然后给一下反馈。
absl-py==2.0.0
accelerate==0.30.1
addict==2.4.0
aiofiles==23.2.1
aiohttp==3.8.6
aiohttp-cors==0.7.0
aioprometheus==23.12.0
aiorwlock==1.4.0
aiosignal==1.3.1
aliyun-python-sdk-core==2.14.0
aliyun-python-sdk-kms==2.16.2
altair==5.1.2
amqp==5.2.0
annotated-types==0.6.0
anyio==3.7.1
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.1.0
Babel==2.13.1
beautifulsoup4==4.12.2
beir==2.0.0
billiard==4.2.0
bitsandbytes==0.42.0
bleach==6.1.0
blessed==1.20.0
blinker==1.7.0
Brotli @ file:///tmp/abs_ecyw11_7ze/croots/recipe/brotli-split_1659616059936/work
C-MTEB==1.1.1
cachetools==5.3.2
cchardet==2.1.7
celery==5.3.6
certifi @ file:///croot/certifi_1690232220950/work/certifi
cffi @ file:///croot/cffi_1670423208954/work
chardet==5.2.0
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
cloudpickle==3.0.0
colorama==0.4.6
coloredlogs==15.0.1
colorful==0.5.6
comm==0.1.4
contourpy==1.2.0
crcmod==1.7
cryptography @ file:///croot/cryptography_1694444244250/work
cupy-cuda12x==12.1.0
cycler==0.12.1
datasets==2.14.6
debugpy==1.8.1
decorator==5.1.1
deepspeed==0.14.0
defusedxml==0.7.1
dill==0.3.7
diskcache==5.6.3
distlib==0.3.8
distro==1.9.0
docopt==0.6.2
einops==0.7.0
elastic-transport==8.13.1
elasticsearch==7.9.1
elasticsearch-dsl==8.11.0
emoji==2.8.0
et-xmlfile==1.1.0
evaluate==0.4.1
exceptiongroup==1.1.3
executing==2.0.1
faiss-cpu==1.7.4
fastapi==0.104.1
fastjsonschema==2.18.1
fastrlock==0.8.2
ffmpy==0.3.2
filelock==3.13.1
Flask==3.0.0
flatbuffers==23.5.26
fonttools==4.47.2
fqdn==1.5.1
frozenlist==1.4.0
fsspec==2023.10.0
gast==0.5.4
gitdb==4.0.11
GitPython==3.1.40
gmpy2 @ file:///tmp/build/80754af9/gmpy2_1645455533097/work
google-api-core==2.16.1
google-auth==2.25.2
google-auth-oauthlib==1.2.0
googleapis-common-protos==1.62.0
gpustat==1.1.1
gradio==4.23.0
gradio_client==0.14.0
grpcio==1.60.0
h11==0.14.0
hjson==3.1.0
hnswlib==0.8.0
httpcore==1.0.1
httptools==0.6.1
httpx==0.25.1
huggingface-hub==0.23.2
humanfriendly==10.0
idna @ file:///croot/idna_1666125576474/work
importlib-metadata==6.8.0
importlib_resources==6.4.0
interegular==0.3.3
ipykernel==6.26.0
ipython==8.17.2
ipython-genutils==0.2.0
ipywidgets==8.1.1
isoduration==20.11.0
itsdangerous==2.1.2
jedi==0.19.1
jieba==0.42.1
Jinja2 @ file:///croot/jinja2_1666908132255/work
jmespath==0.10.0
joblib==1.3.2
json5==0.9.14
jsonlines==4.0.0
jsonpointer==2.4
jsonschema==4.19.2
jsonschema-specifications==2023.7.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.8.0
jupyter-lsp==2.2.0
jupyter_client==8.5.0
jupyter_core==5.5.0
jupyter_server==2.9.1
jupyter_server_terminals==0.4.4
jupyterlab==4.0.7
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.9
jupyterlab_server==2.25.0
kiwisolver==1.4.5
kombu==5.3.4
lark==1.1.9
llvmlite==0.42.0
Markdown==3.5.1
markdown-it-py==3.0.0
MarkupSafe @ file:///opt/conda/conda-bld/markupsafe_1654597864307/work
matplotlib==3.8.2
matplotlib-inline==0.1.6
mdurl==0.1.2
mistune==3.0.2
mkl-fft @ file:///croot/mkl_fft_1695058164594/work
mkl-random @ file:///croot/mkl_random_1695059800811/work
mkl-service==2.4.0
modelscope==1.11.0
mpmath @ file:///croot/mpmath_1690848262763/work
msgpack==1.0.7
mteb==1.1.1
multidict==6.0.4
multiprocess==0.70.15
nbclient==0.8.0
nbconvert==7.10.0
nbformat==5.9.2
nest-asyncio==1.5.8
networkx @ file:///croot/networkx_1690561992265/work
ninja==1.11.1.1
nltk==3.8.1
notebook==7.0.6
notebook_shim==0.2.3
numba==0.59.1
numpy @ file:///croot/numpy_and_numpy_base_1695830428084/work/dist/numpy-1.26.0-cp310-cp310-linux_x86_64.whl#sha256=fc2732718bc9e06a7b702492cb4f5afffe9671083930452d894377bf563464a3
nvidia-cublas-cu11==11.11.3.6
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu11==11.8.87
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu11==11.8.89
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu11==11.8.89
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu11==8.7.0.84
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu11==10.9.0.58
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu11==10.3.0.86
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu11==11.4.1.48
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu11==11.7.5.86
nvidia-cusparse-cu12==12.1.0.106
nvidia-ml-py==12.535.133
nvidia-nccl-cu11==2.19.3
nvidia-nccl-cu12==2.18.1
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu11==11.8.86
nvidia-nvtx-cu12==12.1.105
nvitop==1.3.1
oauthlib==3.2.2
onnx==1.15.0
onnxruntime==1.16.3
onnxruntime-gpu==1.16.3
openai==1.14.3
opencensus==0.11.4
opencensus-context==0.1.3
openpyxl==3.1.2
optimum==1.16.2
orjson==3.9.15
oss2==2.18.4
outlines==0.0.37
overrides==7.4.0
packaging==23.2
pandas==2.1.2
pandocfilters==1.5.0
parso==0.8.3
peft==0.9.0
pexpect==4.8.0
Pillow @ file:///croot/pillow_1696580024257/work
pipreqs==0.4.13
platformdirs==3.11.0
prometheus-client==0.18.0
prompt-toolkit==3.0.39
protobuf==4.23.4
psutil==5.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
py-spy==0.3.14
pyarrow==13.0.0
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pycryptodome==3.20.0
pydantic==2.6.4
pydantic_core==2.16.3
pydeck==0.8.1b0
pydub==0.25.1
Pygments==2.16.1
PyJWT==2.8.0
pynvml==11.5.0
pyOpenSSL @ file:///croot/pyopenssl_1690223430423/work
pyparsing==3.1.1
PySocks @ file:///home/builder/ci_310/pysocks_1640793678128/work
python-dateutil==2.8.2
python-dotenv==1.0.1
python-json-logger==2.0.7
python-multipart==0.0.9
pytrec-eval==0.5
pytz==2023.3.post1
PyYAML @ file:///croot/pyyaml_1698096049011/work
pyzmq==25.1.1
qtconsole==5.4.4
QtPy==2.4.1
quantile-python==1.1
rank-bm25==0.2.2
ray==2.9.1
redis==5.0.1
referencing==0.30.2
regex==2023.10.3
replicate==0.17.0
requests @ file:///croot/requests_1690400202158/work
requests-oauthlib==1.3.1
responses==0.18.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.6.0
rpds-py==0.10.6
rsa==4.9
ruff==0.3.4
safetensors==0.4.2
scikit-learn==1.3.2
scipy==1.11.3
semantic-version==2.10.0
Send2Trash==1.8.2
sentence-transformers==3.0.0
sentencepiece==0.1.99
shellingham==1.5.4
simplejson==3.19.2
six==1.16.0
smart-open==6.4.0
smmap==5.0.1
sniffio==1.3.0
sortedcontainers==2.4.0
soupsieve==2.5
sse-starlette==1.8.1
stack-data==0.6.3
starlette==0.27.0
streamlit==1.30.0
sympy @ file:///croot/sympy_1668202399572/work
tabulate==0.9.0
tenacity==8.2.3
tensorboard==2.16.2
tensorboard-data-server==0.7.2
tensorboardX==2.6.2.2
termcolor==2.3.0
terminado==0.17.1
threadpoolctl==3.2.0
tiktoken==0.5.2
timm==0.9.12
tinycss2==1.2.1
tokenizers==0.19.1
toml==0.10.2
tomli==2.0.1
tomlkit==0.12.0
toolz==0.12.0
torch==2.1.0
torchaudio==2.1.0
torchvision==0.16.0
tornado==6.3.3
tqdm==4.66.1
traitlets==5.13.0
transformers==4.41.1
transformers-stream-generator==0.0.4
triton==2.1.0
typer==0.9.4
types-python-dateutil==2.8.19.14
typing_extensions @ file:///croot/typing_extensions_1690297465030/work
tzdata==2023.3
tzlocal==5.2
uniem==0.3.3
uri-template==1.3.0
urllib3 @ file:///croot/urllib3_1698257533958/work
uvicorn==0.24.0.post1
uvloop==0.19.0
validators==0.22.0
vine==5.1.0
virtualenv==20.25.0
vllm==0.3.3
watchdog==3.0.0
watchfiles==0.21.0
wcwidth==0.2.9
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.4
websockets==11.0.3
Werkzeug==3.0.1
widgetsnbextension==4.0.9
xformers==0.0.23.post1
XlsxWriter==3.1.9
xxhash==3.4.1
yapf==0.40.2
yarg==0.1.9
yarl==1.9.2
zhipuai==2.0.1
zipp==3.17.0
看了一下,可能是transformers版本不一样,我这里是
transformers==4.41.1
你升级一下,然后给一下反馈。absl-py==2.0.0 accelerate==0.30.1 addict==2.4.0 aiofiles==23.2.1 aiohttp==3.8.6 aiohttp-cors==0.7.0 aioprometheus==23.12.0 aiorwlock==1.4.0 aiosignal==1.3.1 aliyun-python-sdk-core==2.14.0 aliyun-python-sdk-kms==2.16.2 altair==5.1.2 amqp==5.2.0 annotated-types==0.6.0 anyio==3.7.1 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 asttokens==2.4.1 async-lru==2.0.4 async-timeout==4.0.3 attrs==23.1.0 Babel==2.13.1 beautifulsoup4==4.12.2 beir==2.0.0 billiard==4.2.0 bitsandbytes==0.42.0 bleach==6.1.0 blessed==1.20.0 blinker==1.7.0 Brotli @ file:///tmp/abs_ecyw11_7ze/croots/recipe/brotli-split_1659616059936/work C-MTEB==1.1.1 cachetools==5.3.2 cchardet==2.1.7 celery==5.3.6 certifi @ file:///croot/certifi_1690232220950/work/certifi cffi @ file:///croot/cffi_1670423208954/work chardet==5.2.0 charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work click==8.1.7 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.3.0 cloudpickle==3.0.0 colorama==0.4.6 coloredlogs==15.0.1 colorful==0.5.6 comm==0.1.4 contourpy==1.2.0 crcmod==1.7 cryptography @ file:///croot/cryptography_1694444244250/work cupy-cuda12x==12.1.0 cycler==0.12.1 datasets==2.14.6 debugpy==1.8.1 decorator==5.1.1 deepspeed==0.14.0 defusedxml==0.7.1 dill==0.3.7 diskcache==5.6.3 distlib==0.3.8 distro==1.9.0 docopt==0.6.2 einops==0.7.0 elastic-transport==8.13.1 elasticsearch==7.9.1 elasticsearch-dsl==8.11.0 emoji==2.8.0 et-xmlfile==1.1.0 evaluate==0.4.1 exceptiongroup==1.1.3 executing==2.0.1 faiss-cpu==1.7.4 fastapi==0.104.1 fastjsonschema==2.18.1 fastrlock==0.8.2 ffmpy==0.3.2 filelock==3.13.1 Flask==3.0.0 flatbuffers==23.5.26 fonttools==4.47.2 fqdn==1.5.1 frozenlist==1.4.0 fsspec==2023.10.0 gast==0.5.4 gitdb==4.0.11 GitPython==3.1.40 gmpy2 @ file:///tmp/build/80754af9/gmpy2_1645455533097/work google-api-core==2.16.1 google-auth==2.25.2 google-auth-oauthlib==1.2.0 googleapis-common-protos==1.62.0 gpustat==1.1.1 gradio==4.23.0 gradio_client==0.14.0 grpcio==1.60.0 h11==0.14.0 hjson==3.1.0 hnswlib==0.8.0 httpcore==1.0.1 httptools==0.6.1 httpx==0.25.1 huggingface-hub==0.23.2 humanfriendly==10.0 idna @ file:///croot/idna_1666125576474/work importlib-metadata==6.8.0 importlib_resources==6.4.0 interegular==0.3.3 ipykernel==6.26.0 ipython==8.17.2 ipython-genutils==0.2.0 ipywidgets==8.1.1 isoduration==20.11.0 itsdangerous==2.1.2 jedi==0.19.1 jieba==0.42.1 Jinja2 @ file:///croot/jinja2_1666908132255/work jmespath==0.10.0 joblib==1.3.2 json5==0.9.14 jsonlines==4.0.0 jsonpointer==2.4 jsonschema==4.19.2 jsonschema-specifications==2023.7.1 jupyter==1.0.0 jupyter-console==6.6.3 jupyter-events==0.8.0 jupyter-lsp==2.2.0 jupyter_client==8.5.0 jupyter_core==5.5.0 jupyter_server==2.9.1 jupyter_server_terminals==0.4.4 jupyterlab==4.0.7 jupyterlab-pygments==0.2.2 jupyterlab-widgets==3.0.9 jupyterlab_server==2.25.0 kiwisolver==1.4.5 kombu==5.3.4 lark==1.1.9 llvmlite==0.42.0 Markdown==3.5.1 markdown-it-py==3.0.0 MarkupSafe @ file:///opt/conda/conda-bld/markupsafe_1654597864307/work matplotlib==3.8.2 matplotlib-inline==0.1.6 mdurl==0.1.2 mistune==3.0.2 mkl-fft @ file:///croot/mkl_fft_1695058164594/work mkl-random @ file:///croot/mkl_random_1695059800811/work mkl-service==2.4.0 modelscope==1.11.0 mpmath @ file:///croot/mpmath_1690848262763/work msgpack==1.0.7 mteb==1.1.1 multidict==6.0.4 multiprocess==0.70.15 nbclient==0.8.0 nbconvert==7.10.0 nbformat==5.9.2 nest-asyncio==1.5.8 networkx @ file:///croot/networkx_1690561992265/work ninja==1.11.1.1 nltk==3.8.1 notebook==7.0.6 notebook_shim==0.2.3 numba==0.59.1 numpy @ file:///croot/numpy_and_numpy_base_1695830428084/work/dist/numpy-1.26.0-cp310-cp310-linux_x86_64.whl#sha256=fc2732718bc9e06a7b702492cb4f5afffe9671083930452d894377bf563464a3 nvidia-cublas-cu11==11.11.3.6 nvidia-cublas-cu12==12.1.3.1 nvidia-cuda-cupti-cu11==11.8.87 nvidia-cuda-cupti-cu12==12.1.105 nvidia-cuda-nvrtc-cu11==11.8.89 nvidia-cuda-nvrtc-cu12==12.1.105 nvidia-cuda-runtime-cu11==11.8.89 nvidia-cuda-runtime-cu12==12.1.105 nvidia-cudnn-cu11==8.7.0.84 nvidia-cudnn-cu12==8.9.2.26 nvidia-cufft-cu11==10.9.0.58 nvidia-cufft-cu12==11.0.2.54 nvidia-curand-cu11==10.3.0.86 nvidia-curand-cu12==10.3.2.106 nvidia-cusolver-cu11==11.4.1.48 nvidia-cusolver-cu12==11.4.5.107 nvidia-cusparse-cu11==11.7.5.86 nvidia-cusparse-cu12==12.1.0.106 nvidia-ml-py==12.535.133 nvidia-nccl-cu11==2.19.3 nvidia-nccl-cu12==2.18.1 nvidia-nvjitlink-cu12==12.3.101 nvidia-nvtx-cu11==11.8.86 nvidia-nvtx-cu12==12.1.105 nvitop==1.3.1 oauthlib==3.2.2 onnx==1.15.0 onnxruntime==1.16.3 onnxruntime-gpu==1.16.3 openai==1.14.3 opencensus==0.11.4 opencensus-context==0.1.3 openpyxl==3.1.2 optimum==1.16.2 orjson==3.9.15 oss2==2.18.4 outlines==0.0.37 overrides==7.4.0 packaging==23.2 pandas==2.1.2 pandocfilters==1.5.0 parso==0.8.3 peft==0.9.0 pexpect==4.8.0 Pillow @ file:///croot/pillow_1696580024257/work pipreqs==0.4.13 platformdirs==3.11.0 prometheus-client==0.18.0 prompt-toolkit==3.0.39 protobuf==4.23.4 psutil==5.9.6 ptyprocess==0.7.0 pure-eval==0.2.2 py-cpuinfo==9.0.0 py-spy==0.3.14 pyarrow==13.0.0 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work pycryptodome==3.20.0 pydantic==2.6.4 pydantic_core==2.16.3 pydeck==0.8.1b0 pydub==0.25.1 Pygments==2.16.1 PyJWT==2.8.0 pynvml==11.5.0 pyOpenSSL @ file:///croot/pyopenssl_1690223430423/work pyparsing==3.1.1 PySocks @ file:///home/builder/ci_310/pysocks_1640793678128/work python-dateutil==2.8.2 python-dotenv==1.0.1 python-json-logger==2.0.7 python-multipart==0.0.9 pytrec-eval==0.5 pytz==2023.3.post1 PyYAML @ file:///croot/pyyaml_1698096049011/work pyzmq==25.1.1 qtconsole==5.4.4 QtPy==2.4.1 quantile-python==1.1 rank-bm25==0.2.2 ray==2.9.1 redis==5.0.1 referencing==0.30.2 regex==2023.10.3 replicate==0.17.0 requests @ file:///croot/requests_1690400202158/work requests-oauthlib==1.3.1 responses==0.18.0 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rich==13.6.0 rpds-py==0.10.6 rsa==4.9 ruff==0.3.4 safetensors==0.4.2 scikit-learn==1.3.2 scipy==1.11.3 semantic-version==2.10.0 Send2Trash==1.8.2 sentence-transformers==3.0.0 sentencepiece==0.1.99 shellingham==1.5.4 simplejson==3.19.2 six==1.16.0 smart-open==6.4.0 smmap==5.0.1 sniffio==1.3.0 sortedcontainers==2.4.0 soupsieve==2.5 sse-starlette==1.8.1 stack-data==0.6.3 starlette==0.27.0 streamlit==1.30.0 sympy @ file:///croot/sympy_1668202399572/work tabulate==0.9.0 tenacity==8.2.3 tensorboard==2.16.2 tensorboard-data-server==0.7.2 tensorboardX==2.6.2.2 termcolor==2.3.0 terminado==0.17.1 threadpoolctl==3.2.0 tiktoken==0.5.2 timm==0.9.12 tinycss2==1.2.1 tokenizers==0.19.1 toml==0.10.2 tomli==2.0.1 tomlkit==0.12.0 toolz==0.12.0 torch==2.1.0 torchaudio==2.1.0 torchvision==0.16.0 tornado==6.3.3 tqdm==4.66.1 traitlets==5.13.0 transformers==4.41.1 transformers-stream-generator==0.0.4 triton==2.1.0 typer==0.9.4 types-python-dateutil==2.8.19.14 typing_extensions @ file:///croot/typing_extensions_1690297465030/work tzdata==2023.3 tzlocal==5.2 uniem==0.3.3 uri-template==1.3.0 urllib3 @ file:///croot/urllib3_1698257533958/work uvicorn==0.24.0.post1 uvloop==0.19.0 validators==0.22.0 vine==5.1.0 virtualenv==20.25.0 vllm==0.3.3 watchdog==3.0.0 watchfiles==0.21.0 wcwidth==0.2.9 webcolors==1.13 webencodings==0.5.1 websocket-client==1.6.4 websockets==11.0.3 Werkzeug==3.0.1 widgetsnbextension==4.0.9 xformers==0.0.23.post1 XlsxWriter==3.1.9 xxhash==3.4.1 yapf==0.40.2 yarg==0.1.9 yarl==1.9.2 zhipuai==2.0.1 zipp==3.17.0
你好,老师,我升级到了4.41.2,然后已经可以了。因为我的环境是之前安装的,所以transformers版本有些滞后会有这个问题。
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\processing_utils.py:465, in ProcessorMixin.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, kwargs) 462 if token is not None: 463 kwargs[\"token\"] = token --> 465 args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, kwargs) 466 processor_dict, kwargs = cls.get_processor_dict(pretrained_model_name_or_path, kwargs) 468 return cls.from_args_and_dict(args, processor_dict, kwargs)
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\processing_utils.py:511, in ProcessorMixin._get_arguments_from_pretrained(cls, pretrained_model_name_or_path, kwargs) 508 else: 509 attribute_class = getattr(transformers_module, class_name) --> 511 args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, kwargs)) 512 return args
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\tokenization_utils_base.py:2089, in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, trust_remote_code, *init_inputs, *kwargs) 2086 else: 2087 logger.info(f\"loading file {file_path} from cache at {resolved_vocab_files[file_id]}\") -> 2089 return cls._from_pretrained( 2090 resolved_vocab_files, 2091 pretrained_model_name_or_path, 2092 init_configuration, 2093 init_inputs, 2094 token=token, 2095 cache_dir=cache_dir, 2096 local_files_only=local_files_only, 2097 _commit_hash=commit_hash, 2098 _is_local=is_local, 2099 trust_remote_code=trust_remote_code, 2100 **kwargs, 2101 )
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\tokenization_utils_base.py:2311, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, token, cache_dir, local_files_only, _commit_hash, _is_local, trust_remote_code, *init_inputs, *kwargs) 2309 # Instantiate the tokenizer. 2310 try: -> 2311 tokenizer = cls(init_inputs, **init_kwargs) 2312 except OSError: 2313 raise OSError( 2314 \"Unable to load vocabulary from file. \" 2315 \"Please check that the provided vocabulary is accessible and not corrupted.\" 2316 )
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\models\llama\tokenization_llama_fast.py:124, in LlamaTokenizerFast.init(self, vocab_file, tokenizer_file, clean_up_tokenization_spaces, unk_token, bos_token, eos_token, add_bos_token, add_eos_token, use_default_system_prompt, add_prefix_space, kwargs) 119 logger.warning_once( 120 \"You set
add_prefix_space
. The tokenizer needs to be converted from the slow tokenizers\" 121 ) 122 kwargs[\"from_slow\"] = True --> 124 super().init( 125 vocab_file=vocab_file, 126 tokenizer_file=tokenizer_file, 127 clean_up_tokenization_spaces=clean_up_tokenization_spaces, 128 unk_token=unk_token, 129 bos_token=bos_token, 130 eos_token=eos_token, 131 add_bos_token=add_bos_token, 132 add_eos_token=add_eos_token, 133 use_default_system_prompt=use_default_system_prompt, 134 kwargs, 135 ) 136 self._add_bos_token = add_bos_token 137 self._add_eos_token = add_eos_tokenFile c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\tokenization_utils_fast.py:117, in PreTrainedTokenizerFast.init(self, *args, *kwargs) 114 fast_tokenizer = convert_slow_tokenizer(slow_tokenizer) 115 elif self.slow_tokenizer_class is not None: 116 # We need to create and convert a slow tokenizer to build the backend --> 117 slow_tokenizer = self.slow_tokenizer_class(args, **kwargs) 118 fast_tokenizer = convert_slow_tokenizer(slow_tokenizer) 119 else:
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\models\llama\tokenization_llama.py:169, in LlamaTokenizer.init(self, vocab_file, unk_token, bos_token, eos_token, pad_token, sp_model_kwargs, add_bos_token, add_eos_token, clean_up_tokenization_spaces, use_default_system_prompt, spaces_between_special_tokens, legacy, add_prefix_space, kwargs) 167 self.add_eos_token = add_eos_token 168 self.use_default_system_prompt = use_default_system_prompt --> 169 self.sp_model = self.get_spm_processor(kwargs.pop(\"from_slow\", False)) 170 self.add_prefix_space = add_prefix_space 172 super().init( 173 bos_token=bos_token, 174 eos_token=eos_token, (...) 185 kwargs, 186 )
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\transformers\models\llama\tokenization_llama.py:196, in LlamaTokenizer.get_spm_processor(self, from_slow) 194 tokenizer = spm.SentencePieceProcessor(**self.sp_model_kwargs) 195 if self.legacy or from_slow: # no dependency on protobuf --> 196 tokenizer.Load(self.vocab_file) 197 return tokenizer 199 with open(self.vocab_file, \"rb\") as f:
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\sentencepiece\init.py:961, in SentencePieceProcessor.Load(self, model_file, model_proto) 959 if model_proto: 960 return self.LoadFromSerializedProto(model_proto) --> 961 return self.LoadFromFile(model_file)
File c:\Users\zhiyuanUSYD\miniconda3\envs\autoGluon\lib\site-packages\sentencepiece\init.py:316, in SentencePieceProcessor.LoadFromFile(self, arg) 315 def LoadFromFile(self, arg): --> 316 return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string" }