Top P can be set to 0 in the KoboldAI Client UI which results in an error upon trying to generate anything. It also reports errors in the console when it is running as a horde worker and receives a request with Top P set to 0. I'm assuming this is some kind of divide by 0 error and that Top P should not be allowed to be below 0.01.
Error I see in KoboldAI-Client web UI when setting Top P to 0 and trying to generate anything:
Error
probability tensor contains either `inf`, `nan` or element < 0
Error I see in the console when my horde worker receives a request where Top P is set to 0:
ERROR | KoboldAI-Horde-Bridge.bridge:bridge:202 - Unexpected response received from http://127.0.0.1:5000: {'detail': {'msg': '500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.', 'type': 'generic.error_500'}}. Please check the health of the KAI worker. Retrying in 10 seconds...
INFO | KoboldAI-Horde-Bridge.bridge:bridge:163 - Job received from https://aihorde.net for 512 tokens and 1792 max context. Starting generation...
[2023-04-11 00:02:32,711] ERROR in app: Exception on /api/latest/generate/ [POST]
Traceback (most recent call last):
File "B:\python\lib\site-packages\flask\app.py", line 2528, in wsgi_app
response = self.full_dispatch_request()
File "B:\python\lib\site-packages\flask\app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "B:\python\lib\site-packages\flask\app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "B:\python\lib\site-packages\flask\app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "aiserver.py", line 767, in g
return f(*args, **kwargs)
File "aiserver.py", line 689, in decorated
response = f(schema, *args, **kwargs)
File "aiserver.py", line 672, in decorated
raise e
File "aiserver.py", line 663, in decorated
return f(*args, **kwargs)
File "aiserver.py", line 11004, in post_generate
return _generate_text(body)
File "aiserver.py", line 10869, in _generate_text
genout = apiactionsubmit(body.prompt, use_memory=body.use_memory, use_story=body.use_story, use_world_info=body.use_world_info, use_authors_note=body.use_authors_note)
File "aiserver.py", line 5253, in apiactionsubmit
genout = apiactionsubmit_generate(tokens, minimum, maximum)
File "aiserver.py", line 5137, in apiactionsubmit_generate
_genout, already_generated = tpool.execute(core_generate, txt, minimum, maximum, set())
File "B:\python\lib\site-packages\eventlet\tpool.py", line 132, in execute
six.reraise(c, e, tb)
File "B:\python\lib\site-packages\six.py", line 719, in reraise
raise value
File "B:\python\lib\site-packages\eventlet\tpool.py", line 86, in tworker
rv = meth(*args, **kwargs)
File "aiserver.py", line 5611, in core_generate
result = raw_generate(
File "aiserver.py", line 5839, in raw_generate
batch_encoded = torch_raw_generate(
File "aiserver.py", line 5935, in torch_raw_generate
genout = generator(
File "B:\python\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "B:\python\lib\site-packages\transformers\generation\utils.py", line 1485, in generate
return self.sample(
File "aiserver.py", line 2413, in new_sample
return new_sample.old_sample(self, *args, **kwargs)
File "B:\python\lib\site-packages\transformers\generation\utils.py", line 2560, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
ERROR | KoboldAI-Horde-Bridge.bridge:bridge:202 - Unexpected response received from http://127.0.0.1:5000: {'detail': {'msg': '500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.', 'type': 'generic.error_500'}}. Please check the health of the KAI worker. Retrying in 10 seconds...
ERROR | KoboldAI-Horde-Bridge.bridge:bridge:90 - Exceeded retry count 5 for generation id ###. Aborting generation!
Top P can be set to 0 in the KoboldAI Client UI which results in an error upon trying to generate anything. It also reports errors in the console when it is running as a horde worker and receives a request with Top P set to 0. I'm assuming this is some kind of divide by 0 error and that Top P should not be allowed to be below 0.01.
Error I see in KoboldAI-Client web UI when setting Top P to 0 and trying to generate anything:
Error I see in the console when my horde worker receives a request where Top P is set to 0: