Using local configuration deepseekcoder.py return Expected all tensors to be on the same device, but found at least two devices

[alexsiu398]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. lavague -i examples/instructions/huggingface.yaml -c examples/configurations/local/deepseek-coder.py launch
2. Accees url or share link
3. enter instructions
4. See error. No code or actions performed. Traceback (most recent call last): File "/LaVague/venv_lv/lib/python3.11/site-packages/gradio/queueing.py", line 501, in call_prediction output = await route_utils.call_process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/LaVague/venv_lv/lib/python3.11/site-packages/gradio/route_utils.py", line 253, in call_process_api output = await app.get_blocks().process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/LaVague/venv_lv/lib/python3.11/site-packages/gradio/blocks.py", line 1695, in process_api result = await self.call_function( ^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/gradio/blocks.py", line 1247, in call_function prediction = await utils.async_iteration(iterator) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/gradio/utils.py", line 516, in async_iteration return await iterator.anext() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/gradio/utils.py", line 509, in anext return await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, args) ^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/gradio/utils.py", line 492, in run_sync_iterator_async return next(iterator) ^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/gradio/utils.py", line 675, in gen_wrapper response = next(iterator) ^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/lavague/command_center.py", line 65, in process_instructions_impl for text in self.actionEngine.get_action_streaming(query, state): File " /LaVague/venv_lv/lib/python3.11/site-packages/lavague/action_engine.py", line 116, in get_action_streaming streaming_response = query_engine.query(query) ^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/base/base_query_engine.py", line 40, in query return self._query(str_or_query_bundle) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/query_engine/retriever_query_engine.py", line 186, in _query nodes = self.retrieve(query_bundle) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/query_engine/retriever_query_engine.py", line 142, in retrieve nodes = self._retriever.retrieve(query_bundle) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/base/base_retriever.py", line 229, in retrieve nodes = self._retrieve(query_bundle) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/lavague/retrievers.py", line 28, in _retrieve return self.html_retriever._retrieve_html(self.html, query_bundle) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/lavague/retrievers.py", line 309, in _retrieve_html results = self._get_results(html, query.query_str) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/lavague/retrievers.py", line 220, in _get_results index = VectorStoreIndex(nodes, embed_model=self.embedder) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 74, in init super().init( File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/indices/base.py", line 94, in init index_struct = self.build_index_from_nodes( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 307, in build_index_from_nodes return self._build_index_from_nodes(nodes, insert_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 279, in _build_index_from_nodes self._add_nodes_to_index( File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 232, in _add_nodes_to_index nodes_batch = self._get_node_with_embedding(nodes_batch, show_progress) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 140, in _get_node_with_embedding id_to_embed_map = embed_nodes( ^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/indices/utils.py", line 138, in embed_nodes new_embeddings = embed_model.get_text_embedding_batch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py", line 255, in get_text_embedding_batch embeddings = self._get_text_embeddings(cur_batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/embeddings/huggingface/base.py", line 204, in _get_text_embeddings return self._embed(texts) ^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/llama_index/embeddings/huggingface/base.py", line 161, in _embed model_output = self._model(encoded_input) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 981, in forward embedding_output = self.embeddings( ^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 207, in forward inputs_embeds = self.word_embeddings(input_ids) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/modules/sparse.py", line 163, in forward return F.embedding( ^^^^^^^^^^^^ File " /LaVague/venv_lv/lib/python3.11/site-packages/torch/nn/functional.py", line 2264, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

Expected behavior No error and actions performed for the instructions

[lyie28]

Hey @alexsiu398 - sorry we never got back to you before. We could not replicate your previous errors.

However, have you tried deepseekcoder with the latest package. The codebase has changed a lot since your last message.

I just tested with deepseekcoder and it at least seems to work with some simple use cases - haven't tested performance with more complex things):

from lavague.drivers.selenium import SeleniumDriver
from lavague.core import ActionEngine, WorldModel
from lavague.core.agents import WebAgent
from lavague.core.logger import AgentLogger
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.huggingface import HuggingFaceLLM

driver = SeleniumDriver(headless=True)

llm = HuggingFaceLLM(model_name="deepseek-ai/deepseek-coder-6.7b-instruct")
embedding = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

action_engine = ActionEngine(driver=driver, llm=llm, embedding=embedding)
world_model = WorldModel()

agent = WebAgent(world_model, action_engine)

agent.get("https://huggingface.co/")
agent.run("what is the top model")

Things to note: 1) You might want to change headless to False if you have an environment with access to the browser so you can visualize what's going on in the browser window - I had set it to True here 2) I still use OpenAI for the multi-modal LLM because I am not aware of an open-source solution yet that works well

[lyie28]

I will close this issue for now since it is an old one now- feel free to ping us on Discord or on your more recent issue to discuss further.