Open Iven2132 opened 2 weeks ago
Yes, I believe it is caused by the current issues using multiple GPUs. Apologies.
See discussion here https://github.com/cambrian-mllm/cambrian/issues/25 and @penghao-wu's temp fix here https://github.com/cambrian-mllm/cambrian/issues/12#issuecomment-2198610750
Or more simply, you can force 1 GPU for the model worker via CUDA_VISIBLE_DEVICES=0 ...
@ellisbrown How to fix
raceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/site-packages/cambrian/serve/gradio_web_server.py", line 469, in <module>
demo = build_demo(args.embed, concurrency_count=args.concurrency_count)
File "/usr/local/lib/python3.10/site-packages/cambrian/serve/gradio_web_server.py", line 337, in build_demo
gr.Examples(examples=[
File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 58, in create_examples
examples_obj = Examples(
File "/usr/local/lib/python3.10/site-packages/gradio/helpers.py", line 216, in __init__
prediction_value = processing_utils.move_files_to_cache(
File "/usr/local/lib/python3.10/site-packages/gradio/processing_utils.py", line 265, in move_files_to_cache
return client_utils.traverse(data, _move_to_cache, client_utils.is_file_obj)
File "/usr/local/lib/python3.10/site-packages/gradio_client/utils.py", line 919, in traverse
return func(json_obj)
File "/usr/local/lib/python3.10/site-packages/gradio/processing_utils.py", line 257, in _move_to_cache
temp_file_path = move_resource_to_block_cache(payload.path, block)
File "/usr/local/lib/python3.10/site-packages/gradio/processing_utils.py", line 234, in move_resource_to_block_cache
return block.move_resource_to_block_cache(url_or_file_path)
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 258, in move_resource_to_block_cache
temp_file_path = processing_utils.save_file_to_cache(
File "/usr/local/lib/python3.10/site-packages/gradio/processing_utils.py", line 170, in save_file_to_cache
temp_dir = hash_file(file_path)
File "/usr/local/lib/python3.10/site-packages/gradio/processing_utils.py", line 102, in hash_file
with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/site-packages/cambrian/serve/examples/chicken_world.webp'
I am getting "NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE" When I try to run the demo on 2 A100s GPU. Any fix?