Closed hijkzzz closed 5 months ago
Hi @hijkzzz, do you include any tokenizer files in your folder /home/scratch.jianh_inf/.cache/huggingface/hub/models--lightblue--Jamba-v0.1-chat-multilingual/snapshots/38a2d5d2301ba642d1a48be1251a825022f78730
? If so, you can also directly run following command to see which part you got hang.
python /home/scratch.jianh_gpu/projects/RULER/scripts/data/synthetic/niah.py \
--save_dir /home/scratch.jianh_gpu/projects/RULER/jamba/synthetic/131072/data \
--save_name niah_single_1 \
--subset validation \
--tokenizer_path /home/scratch.jianh_inf/.cache/huggingface/hub/models--lightblue--Jamba-v0.1-chat-multilingual/snapshots/38a2d5d2301ba642d1a48be1251a825022f78730 \
--tokenizer_type hf \
--max_seq_length 131072 \
--tokens_to_generate 128 \
--num_samples 500 \
--random_seed 42 \
--type_haystack repeat \
--type_needle_k words \
--type_needle_v numbers \
--num_needle_k 1 \
--num_needle_v 1 \
--num_needle_q 1 \
--template "<|im_start|>system
You are a helpful AI assistant.
<|im_end|>
<|im_start|>user
Some special magic {type_needle_v} are hidden within the following text. Make sure to memorize it. I will quiz you about the {type_needle_v} afterwards.
{context}
What are all the special magic {type_needle_v} for {query} mentioned in the provided text?
<|im_end|>
<|im_start|>assistant
The special magic {type_needle_v} for {query} mentioned in the provided text are"
@hsiehjackson I can load the tokenizer use "AutoTokenizer.from_pretrained" from "/home/scratch.jianh_inf/.cache/huggingface/hub/models--lightblue--Jamba-v0.1-chat-multilingual/snapshots/38a2d5d2301ba642d1a48be1251a825022f78730"
my script
model = AutoModelForCausalLM.from_pretrained("/home/scratch.jianh_inf/.cache/huggingface/hub/models--lightblue--Jamba-v0.1-chat-multilingual/snapshots/38a2d5d2301ba642d1a48be1251a825022f78730",
trust_remote_code=True,
attn_implementation="flash_attention_2",
torch_dtype=torch.bfloat16,
device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("/home/scratch.jianh_inf/.cache/huggingface/hub/models--lightblue--Jamba-v0.1-chat-multilingual/snapshots/38a2d5d2301ba642d1a48be1251a825022f78730")
btw, niah.py
hangs as it started, and I didn't see useful information....
And after running this script, I find that the docker container also hangs Control-C failed
cased by docker container mount
envs:
This script hangs here 12 hours
logs