Jetson AGX Orin上推理glm4v-9b，无法停止

mltloveyy commented 1 month ago

System Info / 系統信息

cuda版本：12.2 transformers版本：4.44.0 python版本：3.10 pytorch：2.4 操作系统：Jetpack 6.0 (Ubuntu 22.04) 硬件信息：Jetson AGX Orin 64G 模型下载：魔搭上下载

Who can help? / 谁可以帮助到您？

No response

Information / 问题信息

[X] The official example scripts / 官方的示例脚本
[ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

从NGC下载torch镜像：nvcr.io/nvidia/pytorch:24.07-py3-igpu
安装依赖transformers、tiktoken
修改模型路径并运行basic_demo/trans_cli_vision_demo.py
输入图像路径并提问
终端显示结果为： Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [05:44<00:00, 22.98s/it] Welcome to the GLM-4-9B CLI chat. Type your messages below. Image Path:/workspace/images/outdoor_family.jpeg

You: What is the little girl holding in her hand
GLM-4V:The young child, who appears to be between three and five years old based on their size relative to an adult's arm span while walking down steps with them as support suggests a protective stance for safety reasons rather than age indication of exactly how many birthdays they have had.

In this image provided by Shutterstock ID 615022317, we see that she has something tucked into both hands. The object looks like it could potentially fit within these two small fingers:

1) A water bottle - It seems practical given its shape which aligns closely enough; 2) An ice cream cone wrapper – If you consider what might typically accompany such treats during outdoor activities or sightseeing trips where families are often found exploring historical sites together (such as castles).

Without any additional context clues from visible text labels surrounding objects held directly around waist height level towards camera view angle perspective viewpoint assumption would suggest either scenario above depending upon setting/location specifics at time taken photo was captured along touristic pathway leading downwards stairway ascending stone wall structure possibly suggesting visitation site related travel destination tourism activity involving leisurely pursuits including hiking adventures amongst scenic landscapes offering cultural heritage appreciation opportunities through exploration historic landmarks architectures structures preserved throughout history periods civilizations epochs past present future generations alike preserving traditions customs beliefs values norms societal constructs social fabric tapestry woven threads continuity lineage ancestry descent family tree roots branching out spreading wings embracing change adaptation resilience adaptability flexibility dynamism innovation creativity entrepreneurship spirit drive ambition motivation determination perseverance tenacity grit fortitude courage bravery valor strength power capability potential possibilities endless universe cosmos infinite expanse boundless horizon limitless sky blue heavens azure canopy celestial sphere stars twinkling shining glowing illuminating brightening enlightening awakening consciousness awareness understanding knowledge wisdom intelligence insight discernment perception cognition thinking reasoning problem-solving decision-making choices options alternatives strategies tactics plans actions deeds behaviors conduct manners etiquette protocol rules regulations laws justice fairness equality balance harmony cooperation teamwork collaboration synergy partnership alliance friendship love compassion empathy kindness generosity benevolence charity philanthropy humanitarian aid assistance relief help hope optimism positivity energy enthusiasm passion dedication commitment loyalty faith trust belief conviction certainty assurance security stability peace tranquility serenity calmness composure collected demeanor poise grace elegance beauty aesthetics artistry craftsmanship skill talent ability expertise proficiency mastery genius brilliance prodigy wunderkind phenom phenomenon marvel miracle wonder awe inspiration enlightenment illumination revelation epiphany breakthrough discovery invention creation originality uniqueness individuality personality character traits qualities attributes features aspects elements components parts pieces fragments segments portions shares stakes interests concerns worries anxieties fears doubts hesitations uncertainties vagueness ambiguity confusion perplexity complexity intricacy detail precision accuracy exactitude correctness rightfulness righteousness moral ethical virtuous goodness decency humanity civilization culture tradition folklore legend myth tale story narrative fable allegory parable proverb maxim aphorism saying quote remark statement declaration proclamation announcement broadcast message communication dialogue conversation discussion debate argument controversy disagreement dissent conflict war violence aggression terrorism brutality cruelty harm injury pain suffering sadness grief loss loneliness despair depression anxiety fear panic terror shock surprise astonishment disbelief bewilderment confusion disorientation dizziness vertigo nausea vomiting diarrhea constipation headache stomachache toothache earache eye ache muscle cramp backache leg ache foot ache wrist ache neck ache shoulder ache jaw ache sinus pressure congestion runny nose sneezing cough sore throat ...

Expected behavior / 期待表现

Jetson AGX Orin上推理glm4v-9b能够正常停止

zRzRzRzRzRzRzR commented 1 month ago

我估计是stoptoken没有检测到，你是BF16吗

mltloveyy commented 1 month ago

是的，并未修改官方脚本basic_demo/trans_cli_vision_demo.py：

model = AutoModel.from_pretrained(
    MODEL_PATH,
    trust_remote_code=True,
    # quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    # attn_implementation="flash_attention_2",  # Use Flash Attention
    torch_dtype=torch.bfloat16,
    device_map="auto",
    low_cpu_mem_usage=True
).eval()

THUDM / GLM-4