magicGo
commented
9 months ago Have you solved this problem? I also met the same problem as you, can you share the solution?
Closed renatz closed 10 months ago
magicGo
commented
9 months ago Have you solved this problem? I also met the same problem as you, can you share the solution?
用fastchat框架加载的微调后的baichuan backbone的checkpoint后，每次自动下载model_baichuan.py并报错如下：
ERROR: Exception in ASGI application Traceback (most recent call last): File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi result = await app( # type: ignore[func-returns-value] File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__ return await self.app(scope, receive, send) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/fastapi/applications.py", line 292, in __call__ await super().__call__(scope, receive, send) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__ await self.middleware_stack(scope, receive, send) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__ raise exc File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__ await self.app(scope, receive, _send) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__ raise exc File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__ await self.app(scope, receive, sender) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__ raise e File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__ await self.app(scope, receive, send) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__ await route.handle(scope, receive, send) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/routing.py", line 66, in app response = await func(request) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/fastapi/routing.py", line 273, in app raw_response = await run_endpoint_function( File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/fastapi/routing.py", line 192, in run_endpoint_function return await run_in_threadpool(dependant.call, **values) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool return await anyio.to_thread.run_sync(func, *args) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, *args) File "/mntcephfs/lab_data/kongchuyi/jy/generate-jy/produce-reconstruct/api.py", line 53, in generate history = produce_history(model, tokenizer, req_history_copy, File "/mntcephfs/lab_data/kongchuyi/jy/generate-jy/produce-reconstruct/llmzoo/deploy/webapp/inference.py", line 275, in produce_history q = chatio.stream_output(output_stream) File "/mntcephfs/lab_data/kongchuyi/jy/generate-jy/produce-reconstruct/llmzoo/deploy/webapp/inference.py", line 232, in stream_output for outputs in output_stream: File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 35, in generator_context response = gen.send(None) File "/mntcephfs/lab_data/kongchuyi/jy/generate-jy/produce-reconstruct/llmzoo/deploy/webapp/inference.py", line 137, in generate_stream out = model(torch.as_tensor([input_ids], device=device), use_cache=True) File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/kongchuyi/.cache/huggingface/modules/transformers_modules/checkpoint-1928/modeling_baichuan.py", line 685, in forward outputs = self.model( File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/kongchuyi/.cache/huggingface/modules/transformers_modules/checkpoint-1928/modeling_baichuan.py", line 461, in forward layer_outputs = decoder_layer( File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/kongchuyi/.cache/huggingface/modules/transformers_modules/checkpoint-1928/modeling_baichuan.py", line 273, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/mntcephfs/lab_data/kongchuyi/miniconda3/envs/bcfc/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/kongchuyi/.cache/huggingface/modules/transformers_modules/checkpoint-1928/modeling_baichuan.py", line 215, in forward query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids) File "/home/kongchuyi/.cache/huggingface/modules/transformers_modules/checkpoint-1928/modeling_baichuan.py", line 149, in apply_rotary_pos_emb sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim] RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)