jsonl broken, will only read as json

[carelesswhisp]

any time I try to use the JSONL I get this error 03:29:19-716909 INFO Loading JSONL datasets...
Traceback (most recent call last): File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/queueing.py", line 407, in call_prediction output = await route_utils.call_process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 226, in call_process_api output = await app.get_blocks().process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1550, in process_api result = await self.call_function( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1199, in call_function prediction = await utils.async_iteration(iterator) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 519, in async_iteration return await iterator.anext() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 512, in anext return await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 495, in run_sync_iterator_async return next(iterator) ^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 649, in gen_wrapper yield from f(args, **kwargs) File "/media/cher/brains/text-generation-webui/extensions/Training_PRO_wip/script.py", line 466, in check_dataset loaded_JSONLdata = json.load(dataFile) ^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/json/init.py", line 293, in load return loads(fp.read(), ^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/cher/brains/text-generation-webui/installer_files/env/lib/python3.11/json/decoder.py", line 340, in decode raise JSONDecodeError("Extra data", s, end) json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 4268)

it's loading all jsonl as json? so the next lines will always cause an error. this seems to be with every model. I've tried so far GPT2, mistral and lmsys_vicuna

[carelesswhisp]

something that can take normal jsonl like gpt would be great, where I can essentially transcribe a show and have the ai take on the personality of a character but have full context of an episode. such as

{"messages": [{"role": "user", "content": "text text text"}, {"role": "assistant", "content": "text text"}, {"role": "user", "content": "text text"},

[Sohex]

Jsonl 'works', but the extension needs it to be formatted incorrectly. Wrap the whole thing like an array (e.g.[]) and add commas at the end of all but the last line and it'll work.

To clarify, the correct format for jsonl looks like this:

{"messages": [{"role": "system", "content": "prompt"}, {"role": "user", "content": "input"}, {"role": "assistant", "content": "output"}]}
{"messages": [{"role": "system", "content": "prompt"}, {"role": "user", "content": "input"}, {"role": "assistant", "content": "output"}]}
{"messages": [{"role": "system", "content": "prompt"}, {"role": "user", "content": "input"}, {"role": "assistant", "content": "output"}]}
...

Whereas right now Training_PRO expects:

[
{"messages": [{"role": "system", "content": "prompt"}, {"role": "user", "content": "input"}, {"role": "assistant", "content": "output"}]},
{"messages": [{"role": "system", "content": "prompt"}, {"role": "user", "content": "input"}, {"role": "assistant", "content": "output"}]},
{"messages": [{"role": "system", "content": "prompt"}, {"role": "user", "content": "input"}, {"role": "assistant", "content": "output"}]},
...]

[carelesswhisp]

ah gotcha. though i do notice it will now give the error " raise TemplateError(message) jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/..."

meaning it can't take like a script and format it, can we just modify this template?

Edit: just to explain this isn't an issue with the training pro (I think) this has to do with the embedded template in the tokenizer. so what ever model you're using will determine the format. vicuna - v1.1 actually seems to work out of the box.