search

marella / ctransformers

Python bindings for the Transformer models implemented in C/C++ using GGML library.
MIT License
1.77k stars 138 forks source link

Segmentation fault on m1 mac #8

Closed s-kostyaev closed 1 year ago

s-kostyaev commented 1 year ago

Trying simple example on m1 mac:

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained(
    "/path/to/starcoderbase-GGML/starcoderbase-ggml-q4_0.bin",
    model_type="starcoder",
    lib="basic",
)

print(llm("Hi"))

leads to segmentation fault. Model works fine with ggml example code.

marella commented 1 year ago

Hi, ggml recently introduced a breaking change so existing models have to be re-quantized. This error happens when you are using a old model with the new ggml library. If you pull the latest changes from ggml repo or do a fresh clone, you should get the same error with example code as well.

Latest quantized models are available in this repo: https://huggingface.co/NeoDim/starcoderbase-GGML/tree/main If you have already downloaded from this repo, please check if they are the latest as they got updated just 1 day ago.

Please ensure you are using the latest version of this library:

pip install --upgrade ctransformers

and then run:

llm = AutoModelForCausalLM.from_pretrained(
    'NeoDim/starcoderbase-GGML',
    model_file='starcoderbase-ggml-q4_0.bin',
    model_type='starcoder',
)

print(llm('Hi', max_new_tokens=1))

Above example downloads the latest model file from hugging face repo directly. Please let me know if this works. The reason I used max_new_tokens=1 is because currently it is slow on Mac M1 (https://github.com/marella/ctransformers/issues/5#issuecomment-1556263858). If this basic example is working, we can see how to improve the performance.

s-kostyaev commented 1 year ago

Latest quantized models are available in this repo: https://huggingface.co/NeoDim/starcoderbase-GGML/tree/main If you have already downloaded from this repo, please check if they are the latest as they got updated just 1 day ago.

I know, this is mine repo :)

It still crashes with segmentation fault

marella commented 1 year ago

I know, this is mine repo :)

Oh, nice! :)

Can you please try building from source and let me know if it works:

git clone --recurse-submodules https://github.com/marella/ctransformers
cd ctransformers
./scripts/build.sh

The compiled library will be located at build/lib/libctransformers.dylib which can be used as:

llm = AutoModelForCausalLM.from_pretrained(..., lib='/path/to/ctransformers/build/lib/libctransformers.dylib')
s-kostyaev commented 1 year ago

Compiled from source also crashes with segmentation fault.

marella commented 1 year ago

Thanks for checking. Can you please check with a simpler model to verify if it is starcoder specific issue or library issue:

llm = AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml')

Also were you getting the error while loading the model using from_pretrained() or while generating text using llm()?

Also can you please share your macOS and Python versions. Since I don't have a mac, it may take a while to debug this.

s-kostyaev commented 1 year ago

Unfortunately it also segfaults

marella commented 1 year ago

Were you getting the error while loading the model using from_pretrained() or while generating text using llm()?

s-kostyaev commented 1 year ago

Also were you getting the error while loading the model using from_pretrained() or while generating text using llm()?

While generating. Load is fine. Tokenizer also works

marella commented 1 year ago

Thanks. Can you try running the following and let me know where it is throwing the error:

print('eval', llm.eval([123]))

print('sample', llm.sample())
s-kostyaev commented 1 year ago

Sample works fine. Eval leads to segmentation fault

s-kostyaev commented 1 year ago 
% export PYTHONFAULTHANDLER=1
% python modules/test.py     
Fetching 1 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 19239.93it/s]
Fetching 1 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2978.91it/s]
loaded
Fatal Python error: Segmentation fault

Thread 0x00000001711bf000 (most recent call first):
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 324 in wait
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 607 in wait
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00000001e4c2db40 (most recent call first):
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 241 in eval
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 320 in generate
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 362 in _stream
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 453 in __call__
  File "/Users/username/nn/text-generation-webui/modules/test.py", line 11 in <module>

Extension modules: charset_normalizer.md, yaml._yaml (total: 2)
zsh: segmentation fault  python modules/test.py
/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown                     
  warnings.warn('resource_tracker: There appear to be %d '
s-kostyaev commented 1 year ago

I'm pretty new in python world. I hope it can help to debug this issue.

s-kostyaev commented 1 year ago

With pdb:

(Pdb) step
> /opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py(242)eval()
-> batch_size, threads)
(Pdb) step
> /opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py(241)eval()
-> status = self.ctransformers_llm_batch_eval(tokens, n_tokens,
(Pdb) step
Fatal Python error: Segmentation fault

Thread 0x0000000171ac7000 (most recent call first):
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 324 in wait
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 607 in wait
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00000001e4c2db40 (most recent call first):
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 241 in eval
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 320 in generate
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 362 in _stream
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/site-packages/ctransformers/llm.py", line 453 in __call__
  File "/Users/sergeykostyaev/nn/text-generation-webui/modules/test.py", line 11 in <module>
  File "<string>", line 1 in <module>
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/bdb.py", line 597 in run
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/pdb.py", line 1592 in _runscript
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/pdb.py", line 1732 in main
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/pdb.py", line 1759 in <module>
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/runpy.py", line 86 in _run_code
  File "/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/runpy.py", line 196 in _run_module_as_main

Extension modules: charset_normalizer.md, yaml._yaml (total: 2)
zsh: segmentation fault  python -m pdb modules/test.py
/opt/homebrew/anaconda3/envs/textgen/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown                     
  warnings.warn('resource_tracker: There appear to be %d '
s-kostyaev commented 1 year ago

With lldb I can see:

%  lldb `which python3.10`                  
error: module importing failed: Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cpython_lldb'
(lldb) target create "/opt/homebrew/anaconda3/envs/textgen/bin/python3.10"
Current executable set to '/opt/homebrew/anaconda3/envs/textgen/bin/python3.10' (arm64).
(lldb) run modules/test.py
Process 60321 launched: '/opt/homebrew/anaconda3/envs/textgen/bin/python3.10' (arm64)
Fetching 1 files: 100% 1/1 [00:00<00:00, 19599.55it/s]
Fetching 1 files: 100% 1/1 [00:00<00:00, 3890.82it/s]
loaded
Process 60321 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x16edffff8)
    frame #0: 0x00000001897bade0 libsystem_pthread.dylib`___chkstk_darwin + 60
libsystem_pthread.dylib`:
->  0x1897bade0 <+60>: ldur   x11, [x11, #-0x8]
    0x1897bade4 <+64>: mov    x10, sp
    0x1897bade8 <+68>: cmp    x9, #0x1, lsl #12         ; =0x1000 
    0x1897badec <+72>: b.lo   0x1897bae04               ; <+96>
(lldb) up
frame #1: 0x0000000105386810 libctransformers.dylib`ggml_graph_compute + 128
libctransformers.dylib`ggml_graph_compute:
->  0x105386810 <+128>: mov    x9, sp
    0x105386814 <+132>: sub    x23, x9, x8
    0x105386818 <+136>: mov    sp, x23
    0x10538681c <+140>: mov    x25, #0x0
(lldb) up
frame #2: 0x0000000105352960 libctransformers.dylib`gpt2_eval(gpt2_model const&, int, int, std::__1::vector<int, std::__1::allocator<int>> const&, std::__1::vector<float, std::__1::allocator<float>>&, unsigned long&) + 2252
libctransformers.dylib`gpt2_eval:
->  0x105352960 <+2252>: ldp    x24, x22, [sp, #0x20]
    0x105352964 <+2256>: ldp    x20, x8, [x24]
    0x105352968 <+2260>: sub    x8, x8, x20
    0x10535296c <+2264>: asr    x8, x8, #2
(lldb)
marella commented 1 year ago

Thanks for the detailed info. It looks like you are using anaconda and in a different issue (https://github.com/tee-ar-ex/trx-python/issues/23#issuecomment-1113606685 not related to this library) someone pointed out that anaconda could be the cause. So can you please try installing Python from https://www.python.org/downloads/ and see if it works. Once Python is installed, pip can be installed using python3 -m ensurepip --upgrade

s-kostyaev commented 1 year ago

I will try, thanks

s-kostyaev commented 1 year ago

Without anaconda it doesn't segfault. But it's super slow. And parameter threads does nothing. Always 100% cpu. Eleven minutes is not enough to generate single token on m1 max with model "marella/gpt-2-ggml"

marella commented 1 year ago

Can you try building from source and see if it improves.

s-kostyaev commented 1 year ago

Sure. Now try using built from source

s-kostyaev commented 1 year ago

Why only single thread running? It also doesn't segfault on manually built library, but also super slow. I'm not sure how long it will take to generate single token.

s-kostyaev commented 1 year ago

16 minutes starchat-alpha-q4_0 100% cpu no output with max_new_tokens=1 file test.py:

from ctransformers import AutoModelForCausalLM
from ctransformers import AutoConfig

config = AutoConfig.from_pretrained(
    "/Users/sergeykostyaev/nn/text-generation-webui/models/starchat-alpha-ggml-q4_0.bin",
    threads=8,
)

llm = AutoModelForCausalLM.from_pretrained(
    "/Users/sergeykostyaev/nn/text-generation-webui/models/starchat-alpha-ggml-q4_0.bin",
    model_type="starcoder",
    lib="/Users/sergeykostyaev/nn/ctransformers/build/lib/libctransformers.dylib",
    config=config,
)
print("loaded")
print(llm("Hi", max_new_tokens=1, threads=8))

Printed only "loaded"

marella commented 1 year ago

I think there might be some issue in the library itself. Another user also reported same issue (https://github.com/marella/ctransformers/issues/1#issuecomment-1556250969) but I thought it was just running slow. Before the breaking changes in GGML, older version of this library was working on M1 mac but just very slow. Now it appears to be not working.

Can you also try running a LLaMA model which basically uses llama.cpp:

llm = AutoModelForCausalLM.from_pretrained(
    'TheBloke/LLaMa-7B-GGML',
    model_file='llama-7b.ggmlv3.q4_0.bin',
    model_type='llama',
)
s-kostyaev commented 1 year ago

I will try with llama.cpp model

s-kostyaev commented 1 year ago

Also inference on example code from library - https://github.com/ggerganov/ggml/tree/master/examples/starcoder run just fine.

s-kostyaev commented 1 year ago

I think there might be some issue in the library itself. Another user also reported same issue (#1 (comment)) but I thought it was just running slow. Before the breaking changes in GGML, older version of this library was working on M1 mac but just very slow. Now it appears to be not working.

Can you also try running a LLaMA model which basically uses llama.cpp:

llm = AutoModelForCausalLM.from_pretrained(
    'TheBloke/LLaMa-7B-GGML',
    model_file='llama-7b.ggmlv3.q4_0.bin',
    model_type='llama',
)

Looks like for now llamacpp models has the same issue on apple silicon

s-kostyaev commented 1 year ago

16 minutes starchat-alpha-q4_0 100% cpu no output with max_new_tokens=1 file test.py:

from ctransformers import AutoModelForCausalLM
from ctransformers import AutoConfig

config = AutoConfig.from_pretrained(
    "/Users/sergeykostyaev/nn/text-generation-webui/models/starchat-alpha-ggml-q4_0.bin",
    threads=8,
)

llm = AutoModelForCausalLM.from_pretrained(
    "/Users/sergeykostyaev/nn/text-generation-webui/models/starchat-alpha-ggml-q4_0.bin",
    model_type="starcoder",
    lib="/Users/sergeykostyaev/nn/ctransformers/build/lib/libctransformers.dylib",
    config=config,
)
print("loaded")
print(llm("Hi", max_new_tokens=1, threads=8))

Printed only "loaded"

45 minutes - nothing changes

marella commented 1 year ago

Thanks for checking patiently. I will debug this later.

Can you please try one last thing: try installing an older version of this library and see if it works:

pip install ctransformers==0.1.2
llm = AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml')

print(llm('Hi', max_new_tokens=1))
s-kostyaev commented 1 year ago

Thanks for checking patiently. I will debug this later.

Can you please try one last thing: try installing an older version of this library and see if it works:

pip install ctransformers==0.1.2
llm = AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml')

print(llm('Hi', max_new_tokens=1))

Sure.

s-kostyaev commented 1 year ago

Thanks for checking patiently. I will debug this later.

Can you please try one last thing: try installing an older version of this library and see if it works:

pip install ctransformers==0.1.2
llm = AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml')

print(llm('Hi', max_new_tokens=1))

Looks like after downgrade issue still here.

marella commented 1 year ago

Thanks. Tomorrow I will add a main.cc file to repo which can be run directly without Python. It should make it easy to debug the issue.

marella commented 1 year ago

Hi, I added the main.cc file in debug git branch. Please check if it works:

git clone --recurse-submodules https://github.com/marella/ctransformers
cd ctransformers
git checkout debug

./scripts/build.sh
./build/lib/main <model_type> <model_path> # ./build/lib/main gpt2 /path/to/ggml-model.bin

Also please send the output of both ./scripts/build.sh and ./build/lib/main commands.

s-kostyaev commented 1 year ago

Hi. Sure, will test it now.

s-kostyaev commented 1 year ago 
%  ./build/lib/main starcoder ../text-generation-webui/models/starchat-alpha-ggml-q4_0.bin 

model type : 'starcoder'
model path : '../text-generation-webui/models/starchat-alpha-ggml-q4_0.bin'
prompt     : 'Hi'

load ... ✔
tokenize ... ✔
eval ... ✔
sample ... ✔
detokenize ... ✔
delete ... ✔

response : ''
s-kostyaev commented 1 year ago 
%  ./scripts/build.sh
-- CTRANSFORMERS_INSTRUCTIONS: avx2
-- ARM detected
-- Accelerate framework found
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/sergeykostyaev/nn/ctransformers/build
[ 60%] Built target ctransformers
[ 80%] Building CXX object CMakeFiles/main.dir/main.cc.o
[100%] Linking CXX executable lib/main
ld: warning: directory not found for option '-L/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/'
[100%] Built target main
s-kostyaev commented 1 year ago 
%  ./build/lib/main llama ../text-generation-webui/models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin 

model type : 'llama'
model path : '../text-generation-webui/models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin'
prompt     : 'Hi'

load ... ✔
tokenize ... ✔
eval ... ✔
sample ... ✔
detokenize ... ✔
delete ... ✔

response : '!'
s-kostyaev commented 1 year ago 
%  ./build/lib/main starcoder ../text-generation-webui/models/starcoder-ggml-q4_0.bin

model type : 'starcoder'
model path : '../text-generation-webui/models/starcoder-ggml-q4_0.bin'
prompt     : 'Hi'

load ... ✔
tokenize ... ✔
eval ... ✔
sample ... ✔
detokenize ... ✔
delete ... ✔

response : ''
marella commented 1 year ago 
%  ./scripts/build.sh
-- CTRANSFORMERS_INSTRUCTIONS: avx2
-- ARM detected
-- Accelerate framework found
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/sergeykostyaev/nn/ctransformers/build
[ 60%] Built target ctransformers
[ 80%] Building CXX object CMakeFiles/main.dir/main.cc.o
[100%] Linking CXX executable lib/main
ld: warning: directory not found for option '-L/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/'
[100%] Built target main

Thanks. Is this entire output of build script? It should print the line "Found Threads: TRUE" Also not sure why ld: warning: directory not found for option '-L/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/' appears on a arm macos.

s-kostyaev commented 1 year ago

ld: warning: directory not found for option '-L/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/' this is problem in my configuration. Shouldn't be a problem. And yes, this is entire output.

marella commented 1 year ago

I'm suspecting the issue to be with threads library not being found because the errors you posted previously also show threads in error message.

When you build ggml repo, are you seeing a line which says Found Threads: TRUE?

Also can you please try removing the line set(THREADS_PREFER_PTHREAD_FLAG ON) in CMakeLists.txt and try building again and see if threads is appearing.

s-kostyaev commented 1 year ago

After removing line set(THREADS_PREFER_PTHREAD_FLAG ON) from file models/CMakeLists.txt:

%  ./scripts/build.sh         
-- CTRANSFORMERS_INSTRUCTIONS: avx2
-- ARM detected
-- Accelerate framework found
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/sergeykostyaev/nn/ctransformers/build
[ 60%] Built target ctransformers
[100%] Built target main
marella commented 1 year ago 
%  ./build/lib/main llama ../text-generation-webui/models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin 

model type : 'llama'
model path : '../text-generation-webui/models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin'
prompt     : 'Hi'

load ... ✔
tokenize ... ✔
eval ... ✔
sample ... ✔
detokenize ... ✔
delete ... ✔

response : '!'

At least the LLaMA model is giving some output, so the C++ code is working. So the issue might be when loading the library into Python. I will search more about this and get back to you if I find a solution. Thanks for helping with the debugging.

s-kostyaev commented 1 year ago

Maybe this - https://stackoverflow.com/questions/54587052/cmake-on-mac-could-not-find-threads-missing-threads-found

marella commented 1 year ago

Maybe this - https://stackoverflow.com/questions/54587052/cmake-on-mac-could-not-find-threads-missing-threads-found

I also saw this but cmake should fail with an error but it is successfully building. May be it found threads but simply not printing it. When you build ggml repo, are you seeing a line which says Found Threads: TRUE?

s-kostyaev commented 1 year ago

Thank you. Will wait if you find a solution.

s-kostyaev commented 1 year ago

Maybe this - https://stackoverflow.com/questions/54587052/cmake-on-mac-could-not-find-threads-missing-threads-found

I also saw this but cmake should fail with an error but it is successfully building. May be it found threads but simply not printing it. When you build ggml repo, are you seeing a line which says Found Threads: TRUE?

No.

marella commented 1 year ago

Maybe this - https://stackoverflow.com/questions/54587052/cmake-on-mac-could-not-find-threads-missing-threads-found

I also saw this but cmake should fail with an error but it is successfully building. May be it found threads but simply not printing it. When you build ggml repo, are you seeing a line which says Found Threads: TRUE?

No.

Thanks for checking. I think cmake is just not printing that it found threads library, otherwise it wouldn't work all.

bgonzalezfractal commented 1 year ago

Hi @marella, I've been mentioned in #1 and #5. I have been able to run quantized models for starcoder, starchat, llama, whisper and mpt so far. Nonetheless, none of them work in ctransformers:

I get exactly the same error as @s-kostyaev, meaning the llm object keeps running forever without any change, using the models natively works just fine. We've been trying to use ctransformers and langchain but nothing works, any new information?

I have done everything mentioned in this repo as well, building from source doesn't work.

image

works just fine with ggml natively at 79.63 ms/token

marella commented 1 year ago

Hi @bgonzalezfractal, s-kostyaev was helping me debug the issue but I couldn't find the reason/solution to this yet. So far we found that:

I will keep looking for a solution and will let you know on this thread if I find a solution or if I need your help in debugging the issue.

Can you also please run the following and share the output:

git clone --recurse-submodules https://github.com/marella/ctransformers
cd ctransformers
git checkout debug

./scripts/build.sh
./build/lib/main <model_type> <model_path> # example: ./build/lib/main gpt2 /path/to/ggml-model.bin

Please share the output of ./scripts/build.sh and ./build/lib/main commands.

s-kostyaev commented 1 year ago 
 %  ./scripts/build.sh
-- CTRANSFORMERS_INSTRUCTIONS: avx2
-- ARM detected
-- Accelerate framework found
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/sergeykostyaev/nn/ctransformers/build
[ 60%] Built target ctransformers
[ 80%] Building CXX object CMakeFiles/main.dir/main.cc.o
[100%] Linking CXX executable lib/main
ld: warning: directory not found for option '-L/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/'
[100%] Built target main
s-kostyaev commented 1 year ago 
%  ./build/lib/main llama ../LocalAI/models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin 

model type : 'llama'
model path : '../LocalAI/models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin'
prompt     : 'Hi'

load ... ✔
tokenize ... ✔
> [ 1 18567 ]
eval ... ✔
sample ... ✔
> 29892
detokenize ... ✔
> ','
delete ... ✔