Chat listing multiple issues: fails to load when log file contains "null"; old or broken logs fail to parse

Describe the bug

My past chats widget fail to load because of old or malformed chat content in the logs, and when I fix it to workaround the data issues, there are still more issues loading old chats. Perhaps adding a version number or JSON schema reference in the chat logs can aid in parsing and migration.

There is no indication in the logs for which fail has failed, so to debug, I added
chat.py:559: logger.info(f"Reading chat \"{path}\"") in order to debug and obtain the results below.

Failure cases seen so far; there appear to be many more.

JSON file contains just "null"
- Some of my log files are just the word "null", perhaps due to old bugs. ALL chat logs fail to load for characters that have any such logs due to logic errors in chat.py find_all_histories_with_first_prompts and load_history
JSON file has `internal = []'
- Some logs have an empty array of internal, causing parsing failure.

Old deponent logs get no name

$ cat "/home/klotz/wip/oobabooga/text-generation-webui/logs/chat/Assistant/20240627-14-04-41.json"
{
"internal": [
    [
        "<|BEGIN-VISIBLE-CHAT|>",
        "How can I help you today?"
    ]
],
"visible": [
    [
        "",
        "How can I help you today?"
    ]
]

There is a log file called None.json with the same content as point 4 immediately above.

Is there an existing issue for this?

[X] I have searched the existing issues

Reproduction

use text-generation-webui and accumulate logs, through many versions
update to 6915c5
start text-generation-webui and chat logs widget no longer displays data

Screenshot

This is the failure to list issue:

This is the failure to parse issue, obtained after a few workarounds applied:

Logs

This log is mostly my added logger.info:
...
14:58:16-615990 INFO     Reading chat "logs/chat/Assistant/20240627-14-04-41.json"
14:58:16-620025 INFO     Reading chat "logs/chat/Assistant/20240627-12-55-37.json"
14:58:16-623105 INFO     Reading chat "logs/chat/Assistant/20240627-12-55-31.json"
14:58:16-625805 INFO     Reading chat "logs/chat/Assistant/20240627-12-18-37.json"
Traceback (most recent call last):
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/queueing.py", line 527, in process_events
    response = await route_utils.call_process_api(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 261, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1786, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1338, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 759, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/modules/ui_chat.py", line 300, in <lambda>
    lambda x: gr.update(choices=(histories := chat.find_all_histories_with_first_prompts(x)), value=histories[0][1]), gradio('interface_state'), gradio('unique_id'), show_progress=False).then(
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/klotz/wip/oobabooga/text-generation-webui/modules/chat.py", line 563, in find_all_histories_with_first_prompts
    if 'visible' in data and len(data['visible']) > 0:
       ^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
14:58:16-656005 INFO     Reading chat "logs/chat/Assistant/20240627-14-04-41.json"
14:58:16-656564 INFO     Reading chat "logs/chat/Assistant/20240627-12-55-37.json"
14:58:16-657109 INFO     Reading chat "logs/chat/Assistant/20240627-12-55-31.json"
14:58:16-657644 INFO     Reading chat "logs/chat/Assistant/20240627-12-18-37.json"
Traceback (most recent call last):
    File "/home/klotz/wip/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/queueing.py", line 527, in process_events

System Info

$ uname -a
Linux tensor 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ head -1 /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"

oobabooga / text-generation-webui