search

Hellisotherpeople / llm_steer-oobabooga

Steer LLM outputs towards a certain topic/subject and enhance response capabilities using activation engineering by adding steering vectors, now in oobabooga text generation webui!
MIT License
42 stars 2 forks source link

RuntimeError: coroutine raised StopIteration #1

Closed Jasperb3 closed 7 months ago

Jasperb3 commented 7 months ago

This extension looks very interesting but when trying to add a steering vector, I get this error. Any ideas how to solve it?

Traceback (most recent call last): File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\anyio_backends_asyncio.py", line 851, in run result = context.run(func, args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\gradio\utils.py", line 661, in wrapper response = f(args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\text-generation-webui\extensions\llm_steer-oobabooga\script.py", line 157, in add_steering_vector shared.steered_model = Steer(shared.model, shared.tokenizer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\llm_steer.py", line 41, in init self.device = torch.device(next(model.parameters()).device) ^^^^^^^^^^^^^^^^^^^^^^^^ StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\gradio\queueing.py", line 407, in call_prediction output = await route_utils.call_process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\gradio\route_utils.py", line 226, in call_process_api output = await app.get_blocks().process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\gradio\blocks.py", line 1550, in process_api result = await self.call_function( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\gradio\blocks.py", line 1185, in call_function prediction = await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\XYZ\miniconda3\envs\textgen\Lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ RuntimeError: coroutine raised StopIteration

Hellisotherpeople commented 7 months ago

Hi, not exactly sure what the issue is here. This extension only works for the transformers backend. Can you confirm that you are using the transformers backend?

What model are you trying to run?

Jasperb3 commented 7 months ago

Hi, thanks for your reply.

I was trying it with an EXL2 model (bartowski_dolphin-2.8-experiment26-7b-exl2_8_0) with the ExLlamav2_HF loader.

Might be worth adding to the readme if it only works with transformers.

Hellisotherpeople commented 7 months ago

Yes, I will add it. Apologies for the confusion.

I would suggest writing an issue asking for support for other backends in llm_steer: https://github.com/Mihaiii/llm_steer

Hellisotherpeople commented 7 months ago

I will close this as I am at the whims of llm_steer for supporting other backends - but I will ask Mihaiii to

marknasr22 commented 2 days ago

Any news? Same issue with llamacpp_HF. I thought HF loads as Transformers?