search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
35.61k stars 4.38k forks source link

Arena Error - probabilities contain NaN #3294

Open SuperSecureHuman opened 2 months ago

SuperSecureHuman commented 2 months ago

When trying to run arena battle, it errors. The traceback is below

image

2024-04-29 06:11:41 | ERROR | stderr | /workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py:304: RuntimeWarning: invalid value encountered in divide
2024-04-29 06:11:41 | ERROR | stderr |   model_weights = model_weights / total_weight
2024-04-29 06:11:41 | ERROR | stderr | Traceback (most recent call last):
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/queueing.py", line 527, in process_events
2024-04-29 06:11:41 | ERROR | stderr |     response = await route_utils.call_process_api(
2024-04-29 06:11:41 | ERROR | stderr |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/route_utils.py", line 270, in call_process_api
2024-04-29 06:11:41 | ERROR | stderr |     output = await app.get_blocks().process_api(
2024-04-29 06:11:41 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1847, in process_api
2024-04-29 06:11:41 | ERROR | stderr |     result = await self.call_function(
2024-04-29 06:11:41 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1433, in call_function
2024-04-29 06:11:41 | ERROR | stderr |     prediction = await anyio.to_thread.run_sync(
2024-04-29 06:11:41 | ERROR | stderr |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2024-04-29 06:11:41 | ERROR | stderr |     return await get_async_backend().run_sync_in_worker_thread(
2024-04-29 06:11:41 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
2024-04-29 06:11:41 | ERROR | stderr |     return await future
2024-04-29 06:11:41 | ERROR | stderr |            ^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
2024-04-29 06:11:41 | ERROR | stderr |     result = context.run(func, *args)
2024-04-29 06:11:41 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 788, in wrapper
2024-04-29 06:11:41 | ERROR | stderr |     response = f(*args, **kwargs)
2024-04-29 06:11:41 | ERROR | stderr |                ^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py", line 350, in add_text
2024-04-29 06:11:41 | ERROR | stderr |     model_left, model_right = get_battle_pair()
2024-04-29 06:11:41 | ERROR | stderr |                               ^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py", line 305, in get_battle_pair
2024-04-29 06:11:41 | ERROR | stderr |     chosen_idx = np.random.choice(len(models), p=model_weights)
2024-04-29 06:11:41 | ERROR | stderr |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "numpy/random/mtrand.pyx", line 971, in numpy.random.mtrand.RandomState.choice
2024-04-29 06:11:41 | ERROR | stderr | ValueError: probabilities contain NaN
2024-04-29 06:11:43 | INFO | gradio_web_server_multi | bot_response_multi (anony). ip: [REDACTED], [REDACTED]
2024-04-29 06:11:43 | ERROR | stderr | Traceback (most recent call last):
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/queueing.py", line 527, in process_events
2024-04-29 06:11:43 | ERROR | stderr |     response = await route_utils.call_process_api(
2024-04-29 06:11:43 | ERROR | stderr |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/route_utils.py", line 270, in call_process_api
2024-04-29 06:11:43 | ERROR | stderr |     output = await app.get_blocks().process_api(
2024-04-29 06:11:43 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1847, in process_api
2024-04-29 06:11:43 | ERROR | stderr |     result = await self.call_function(
2024-04-29 06:11:43 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1445, in call_function
2024-04-29 06:11:43 | ERROR | stderr |     prediction = await utils.async_iteration(iterator)
2024-04-29 06:11:43 | ERROR | stderr |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 612, in async_iteration
2024-04-29 06:11:43 | ERROR | stderr |     return await iterator.__anext__()
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 605, in __anext__
2024-04-29 06:11:43 | ERROR | stderr |     return await anyio.to_thread.run_sync(
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2024-04-29 06:11:43 | ERROR | stderr |     return await get_async_backend().run_sync_in_worker_thread(
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
2024-04-29 06:11:43 | ERROR | stderr |     return await future
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
2024-04-29 06:11:43 | ERROR | stderr |     result = context.run(func, *args)
2024-04-29 06:11:43 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 588, in run_sync_iterator_async
2024-04-29 06:11:43 | ERROR | stderr |     return next(iterator)
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 771, in gen_wrapper
2024-04-29 06:11:43 | ERROR | stderr |     response = next(iterator)
2024-04-29 06:11:43 | ERROR | stderr |                ^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py", line 436, in bot_response_multi
2024-04-29 06:11:43 | ERROR | stderr |     state0.to_gradio_chatbot(),
2024-04-29 06:11:43 | ERROR | stderr |     ^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr | AttributeError: 'NoneType' object has no attribute 'to_gradio_chatbot'

Arena Side By Side works fine no issues there.

image

Lemme know what are the details you need to debug this. Happy to help!

kjwony commented 1 month ago

Add your model name in "SAMPLING_WEIGHTS" dict in gradio_block_arena_anony.py code.