monkeypatch problem

[yfliao]

After finish fintuning, I would like to set up the text-generation-webui server for text generation. But how to install monkeypatch?

Approach #1: pip install monkeypatch didn't work

❯ pip install monkeypatch
Collecting monkeypatch
  Using cached monkeypatch-0.1rc3.zip (7.9 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-v23au8m4/monkeypatch_bfa54471d1d64473aa0694f70af11c9a/setup.py", line 99
          except ImportError, e:
                            ^
      SyntaxError: invalid syntax
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Approach #2, python setup.py install didn't work

changed

    if use_stdeb:
        import platform
        if 'debian' in platform.dist():
            try:
                import stdeb
            except ImportError as e:
                pass

then

❯ pwd
/home/nyculiao/liao/alpaca_lora_4bit/text-generation-webui
❯ python server.py

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/nyculiao/anaconda3/envs/pytorch/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so
/home/nyculiao/anaconda3/envs/pytorch/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/nyculiao/anaconda3/envs/pytorch/lib/libcudart.so.11.0'), PosixPath('/home/nyculiao/anaconda3/envs/pytorch/lib/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)
CUDA SETUP: CUDA runtime path found: /home/nyculiao/anaconda3/envs/pytorch/lib/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 118
/home/nyculiao/anaconda3/envs/pytorch/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
  warn(msg)
CUDA SETUP: Loading binary /home/nyculiao/anaconda3/envs/pytorch/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so...
Traceback (most recent call last):
  File "/home/nyculiao/liao/alpaca_lora_4bit/text-generation-webui/server.py", line 5, in <module>
    import custom_monkey_patch # apply monkey patch
  File "/home/nyculiao/liao/alpaca_lora_4bit/text-generation-webui/custom_monkey_patch.py", line 6, in <module>
    from monkeypatch.peft_tuners_lora_monkey_patch import replace_peft_model_with_gptq_lora_model, Linear4bitLt
ModuleNotFoundError: No module named 'monkeypatch.peft_tuners_lora_monkey_patch'; 'monkeypatch' is not a package
❯

[johnsmith0031]

You should add the path of this repo to you include path

import sys
sys.path.add(your path to the repo)

or

export PYTHONPATH=$PYTHONPATH:your path to the repo

[VegaStarlake]

After my issues building wheels I'm still having a "no module named monkeypatch" error and the test finetune listed on the guide (with dataset and model changed) didn't work.

I tried adding the main repo folder using sys.path.append() but that also didn't work. I found appending webui's sever.py to the "alpaca_lora_4bit/text-generation-webui" where "custom_monkey_patch" is located imports that properly, but then it failed to import autograd_4bit. I appended that folder and now it's failing to load module "alpaca_lora_4bit". So, should I append every folder within the repo or is there another/easier way to fix this?

I also noticed there are two autograd files, I appended both folders but I added the one in a cpu architecture (?) folder first. The order didn't seem to matter but does it?

I also tried adding the repo folder to PYTHONPATH but it's still failing to detect modules. Should these changes be made to the server.py in the alpaca_lora folder or in webui?

[johnsmith0031]

Fixed custom_monkey_patch under pip version.

import time
import torch
import alpaca_lora_4bit.autograd_4bit
from alpaca_lora_4bit.autograd_4bit import load_llama_model_4bit_low_ram, Autograd4bitQuantLinear
from peft import PeftModel
from alpaca_lora_4bit.monkeypatch.peft_tuners_lora_monkey_patch import replace_peft_model_with_int4_lora_model
from alpaca_lora_4bit.models import Linear4bitLt
replace_peft_model_with_int4_lora_model()

[VegaStarlake]

I uninstalled anaconda, deleted all python-related system variables, deleted old textgen files, reinstalled using one-click installer, did not follow the winglian steps but simply

ran start_windows on one-click installer ran update_windows git clone https://github.com/johnsmith0031/alpaca_lora_4bit installed requirements pip install . in the alpaca_lora_4bit folder pip install git+https://github.com/sterlind/GPTQ-for-LLaMa.git@lora_4bit python server.py --listen --lora tloen_alpaca-lora-7b --monkey-patch *I can't remember if I used the "--lora tloen" bit or not

Then it started the finetuning process. But I had a data format error so it couldn't complete. To fix that I needed a new python env so I could automate things. I can't be 100% sure that it was fixed since I couldn't complete the finetune, but I had never seen that text/process before.

From the default env that activates when running "cmd_windows" from ooba, I used conda to create a new env, activated it, installed dependencies for automation (pywin32, python-docx odfpy), fixed the dataset by running the new python program, and now monkey-patch is broken again.

I tried deleting the newly created extra environment folder, tried starting with start_windows vs cmd, tried activating the environment from a normal cmd, and I tried adding the sys.path.append to the repositories folder where this repo was dl'd (a step I did before uninstalling but hadn't done when it was 'working'). No changes have been made to the system variables.

I guess I can just reinstall it all again to test my dataset, but if that works then the only workaround I can find is reinstalling everything every time I need to use python for anything else. Non-textgen programs seem to work fine there but I've not tried finetuning on image or tts models. Based on that, I feel like my issues are related to python or conda environments on win10.

Should I just make a new error thread here and or make one on the ooba main?

[johnsmith0031]

As the main branch is not default branch anymore I think there may be some problem with webui. I'll test it later and see If I can do anything.

[VegaStarlake]

I reinstalled everything again, and this time I used checkout for the winglian branch.

Now it gets further along the fine-tuning process WITHOUT monkey-patch flag enabled, but encounters the error:

“\modules\[training.py](http://training.py/)”, line 505, in do_train lora_model = get_peft_model(shared.model, config) File “\env\lib\site-packages\peft\[mapping.py](http://mapping.py/)”, line 106, in get_peft_model return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name) File “\env\lib\site-packages\peft\peft_model.py”, line 889, in init super().__init__(model, peft_config, adapter_name) File “\env\lib\site-packages\peft\peft_model.py”, line 111, in init self.base_model = PEFT_TYPE_TO_MODEL_MAPPING[peft_config.peft_type]( File “\env\lib\site-packages\peft\tuners\[lora.py](http://lora.py/)”, line 274, in init super().__init__(model, config, adapter_name) File “\env\lib\site-packages\peft\tuners\tuners_utils.py”, line 88, in init self.inject_adapter(self.model, adapter_name) File “\env\lib\site-packages\peft\tuners\tuners_utils.py”, line 222, in inject_adapter raise ValueError( ValueError: Target modules [‘q_proj’, ‘v_proj’] not found in the base model. Please check the target modules and try again.

This could be an issue with the model I'm using, and I'll check that.

I'm posting again because, with monkey-patch enabled in the UI I'm still getting an error immediately without the process starting at all:

Traceback (most recent call last): File "\env\lib\site-packages\gradio\routes.py", line 427, in run_predict output = await app.get_blocks().process_api( File "\env\lib\site-packages\gradio\blocks.py", line 1323, in process_api result = await self.call_function( File "\env\lib\site-packages\gradio\blocks.py", line 1067, in call_function prediction = await utils.async_iteration(iterator) File "\env\lib\site-packages\gradio\utils.py", line 336, in async_iteration return await iterator.__anext__() File "\env\lib\site-packages\gradio\utils.py", line 329, in __anext__ return await anyio.to_thread.run_sync( File "\env\lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "\env\lib\site-packages\anyio\_backends\_asyncio.py", line 2106, in run_sync_in_worker_thread return await future File "\env\lib\site-packages\anyio\_backends\_asyncio.py", line 833, in run result = context.run(func, *args) File "\env\lib\site-packages\gradio\utils.py", line 312, in run_sync_iterator_async return next(iterator) File "\modules\training.py", line 272, in do_train from monkeypatch.peft_tuners_lora_monkey_patch import ( ModuleNotFoundError: No module named 'monkeypatch'

I had to use python again to fix dataset issues, but I was able to run python programs using the activated textgen environment without breaking things. This time I edited the programs I was using so they didn't install additional dependencies that weren't normally part of either this repo or ooba. So, with my extremely limited knowledge it does seem like creating new environments in the textgen files can cause problems somehow. However I have no idea what to try next or what to do since the instructions say to enable monkey-patch but I get further in the process with it disabled.

Do you think I should ask on the webui repo instead?

[johnsmith0031]

It seems that the webui is using the old branch of this repo. I think I may need to make a PR to webui for fixing. https://github.com/oobabooga/text-generation-webui/pull/3853

[VegaStarlake]

Ok, thanks for your patience!

From this thread https://github.com/oobabooga/text-generation-webui/issues/3655

I was able to fine tune using GPTQ and one of the models linked in the thread. This method did not use monkey-patch, but I wanted to mention it in case it can help in anyway. I don't know if this method will work for me so I may still need to work with monkeypatch. I will keep trying if it turns out I need it.