Finetune Llama-2 error

[ioma8]

Describe the bug

I am trying to finetune Llama-2 with raw textfile data.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

My llama file is this: llama-2-7b-chat.ggmlv3.q4_1.bin

Text generation works. I want to finetune it on custom raw textfile data. I go to the training tab, put all the parametrs, load the dataset file, press start LoRA training. I get error. In console is the following log. Can llama-2 be fineduned this way?

Screenshot

Logs

2023-07-20 14:34:33 INFO:Loading raw text file dataset...
llama_tokenize_with_model: too many tokens
2023-07-20 14:34:42 WARNING:LoRA training has only currently been validated for LLaMA, OPT, GPT-J, and GPT-NeoX models. (Found model type: LlamaCppModel)
2023-07-20 14:34:47 WARNING:It is highly recommended you use `--load-in-8bit` for LoRA training.
2023-07-20 14:34:49 INFO:Loading raw text file dataset...
llama_tokenize_with_model: too many tokens
Traceback (most recent call last):
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/gradio/routes.py", line 427, in run_predict
    output = await app.get_blocks().process_api(
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1067, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/gradio/utils.py", line 336, in async_iteration
    return await iterator.__anext__()
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/gradio/utils.py", line 329, in __anext__
    return await anyio.to_thread.run_sync(
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/Users/jakubkolcar/custom/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/gradio/utils.py", line 312, in run_sync_iterator_async
    return next(iterator)
  File "/Users/jakubkolcar/custom/oobabooga_macos/text-generation-webui/modules/training.py", line 422, in do_train
    text_chunks = [cut_chunk_for_newline(x, newline_favor_len) for x in text_chunks]
  File "/Users/jakubkolcar/custom/oobabooga_macos/text-generation-webui/modules/training.py", line 422, in <listcomp>
    text_chunks = [cut_chunk_for_newline(x, newline_favor_len) for x in text_chunks]
  File "/Users/jakubkolcar/custom/oobabooga_macos/text-generation-webui/modules/training.py", line 719, in cut_chunk_for_newline
    if '\n' not in chunk:
TypeError: a bytes-like object is required, not 'str'

System Info

MacBook Pro M1
32GB RAM

[Ph0rk0z]

I am not aware that llama.cpp is able to be used for finetuning.

[blueberrysnapple]

Hi, I think I may have found the answer to why this is not working over here:

https://old.reddit.com/r/LocalLLaMA/comments/13c3i33/creating_loras_either_with_llamacpp_or_oobabooga/

I'm still not sure on how to go about getting around this error however.

[Ph0rk0z]

Not sure what the answer is until llama.cpp updates to allow for training.

[jiapei100]

Please also take a look at my issue here at privateGPT

I got this ERROR llama_tokenize_with_model: too many tokens as well...

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.