search

nod-ai / SHARK-TestSuite

Temporary home of a test suite we are evaluating
Apache License 2.0
2 stars 28 forks source link

"UnicodeEncodeError: 'charmap' codec can't encode character" running llama2-7b-GPTQ #105

Closed ScottTodd closed 4 months ago

ScottTodd commented 6 months ago

I'm trying to get .mlir files from the test cases in e2eshark/pytorch/models/, so I can export them as standalone (no Python) tests ready for use in IREE (via the iree_tests/ subproject.

I ran this on my Windows machine:

D:\dev\projects\SHARK-TestSuite\e2eshark
(.venv) λ python run.py --cachedir D:\dev\scratch\e2eshark \
  -c D:\dev\projects\torch-mlir\build \
  -i D:\dev\projects\iree-build \
  --tests pytorch/models/llama2-7b-GPTQ \
  --mode turbine

with this output (after 10 minutes running with my CPU at 100% and no intermediate output):

Cache Directory: D:\dev\scratch\e2eshark
Starting e2eshark tests. Using 4 processes
Torch MLIR build: D:\dev\projects\torch-mlir\build
IREE build: D:\dev\projects\iree-build
Test run directory: D:\dev\projects\SHARK-TestSuite\e2eshark\test-run
Since --tests or --testsfile was specified, --groups ignored
Framework:pytorch mode=turbine backend=llvm-cpu runfrom=model-run runupto=inference
Test list: ['pytorch/models/llama2-7b-GPTQ']
Test pytorch/models/llama2-7b-GPTQ failed [model-run]

Completed run of e2e shark tests

If using the upload feature, you can find a map of the test name to azure urls for your uploaded artifacts in upload_urls.json

model-run.log ends with:

Prompt: What is nature of our existence?
Response: Traceback (most recent call last):
  File "D:\dev\projects\SHARK-TestSuite\e2eshark\test-run\pytorch\models\llama2-7b-GPTQ\runmodel.py", line 46, in <module>
    print("Response:", tokenizer.decode(model_response[0]))
  File "C:\Program Files\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u2113' in position 197: character maps to <undefined>

The directory in the 'test-run' folder then has no .mlir or other useful debugging artifacts:

D:\dev\projects\SHARK-TestSuite\e2eshark
(.venv) λ ls D:\dev\projects\SHARK-TestSuite\e2eshark\test-run\pytorch\models\llama2-7b-GPTQ
__pycache__/  commands.log  commonutils.py@  model-run.log  runmodel.py  time.pkl
ScottTodd commented 4 months ago

Similar error for whisper-small

python run.py --cachedir D:\dev\scratch\e2eshark --tests pytorch/models/whisper-small --mode turbine

model-run.log

Response: Traceback (most recent call last):
  File "D:\dev\projects\SHARK-TestSuite\e2eshark\test-run\pytorch\models\whisper-small\runmodel.py", line 47, in <module>
    print("Response:", tokenizer.decode(model_response[0]))
  File "C:\Program Files\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 82-83: character maps to <undefined>
Output from pip freeze 
pip freeze
accelerate==0.30.1
aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
auto_gptq==0.7.1
azure-core==1.30.1
azure-storage-blob==12.20.0
brevitas @ git+https://github.com/Xilinx/brevitas.git@6695e8df7f6a2c7715b9ed69c4b78157376bb60b
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
cmake==3.29.3
colorama==0.4.6
coloredlogs==15.0.1
cryptography==42.0.7
datasets==2.19.1
dependencies==2.0.1
diffusers==0.24.0
dill==0.3.8
einops==0.8.0
filelock==3.14.0
flatbuffers==24.3.25
frozenlist==1.4.1
fsspec==2024.3.1
gekko==1.1.1
huggingface-cli==0.1
huggingface-hub==0.23.0
humanfriendly==10.0
idna==3.7
importlib_metadata==7.1.0
intel-openmp==2021.4.0
iree-compiler==20240530.909
iree-runtime==20240530.909
isodate==0.6.1
Jinja2==3.1.4
MarkupSafe==2.1.5
mkl==2021.4.0
ml-dtypes==0.4.0
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
networkx==3.3
ninja==1.11.1.1
numpy==1.26.4
onnx==1.15.0
onnxruntime==1.17.3
optimum==1.19.2
packaging==24.0
pandas==2.2.2
peft==0.10.0
pillow==10.3.0
protobuf==5.27.0rc2
psutil==5.9.8
pyarrow==16.1.0
pyarrow-hotfix==0.6
pybind11==2.12.0
pycparser==2.22
pyreadline3==3.4.1
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2024.5.10
requests==2.31.0
rouge==1.0.1
safetensors==0.4.3
sentencepiece==0.2.0
shark-turbine==2.3.0
simplejson==3.19.2
six==1.16.0
sympy==1.12.1rc1
tabulate==0.9.0
tbb==2021.12.0
tokenizers==0.15.2
torch==2.4.0.dev20240505+cpu
torchvision==0.19.0.dev20240505+cpu
tqdm==4.66.4
transformers==4.37.1
-e git+https://github.com/nod-ai/SHARK-Turbine.git@f919efe78903727d149c997e326f41b54ea1e147#egg=turbine_models&subdirectory=models
typing_extensions==4.11.0
tzdata==2024.1
unfoldNd==0.2.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4
zipp==3.18.1