Open tot-ra opened 2 weeks ago
can this be because of docker volume that I try to use?
Hello @tot-ra , thanks for raising this issue. First of all, the forum thread you followed to mount the volume is a bit out-dated. Please follow https://docs.marqo.ai/2.11/Guides/Advanced-Usage/transferring_state/#guide to mount /opt/vespa/var
instead. /opt/vespa/var
contains only the data and configs. This will allow you to transfer data to later version of Marqo if needed.
From the docker compose file you provided, you are using the internal vespa baked into Marqo image. The zookeeper server is embedded in the internal Vespa. I'll need more information (like the marqo log and Vespa log) to pinpoint the issue. In order to access the vespa log, you can either mount a volume to /opt/vespa/logs
as described in the doc, or docker compose exec marqo bash
to check /opt/vespa/logs/vespa/vespa.log
.
My theory is that the mounted volume might have some permission issue that prevented Vespa from starting up. Please provide more information to help me investigate. Thanks a lot.
I added the volume but I keep getting this issue with docker
marqo-1 | httpx.HTTPStatusError: Server error '507 Insufficient Storage' for url 'http://localhost:8080/document/v1/marqo__settings/marqo__settings/docid/collection_768'
marqo-1 | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/507
marqo-1 |
marqo-1 | The above exception was the direct cause of the following exception:
marqo-1 |
marqo-1 | Traceback (most recent call last):
marqo-1 | File "/app/src/marqo/api/route.py", line 20, in marqo_custom_route_handler
marqo-1 | return await original_route_handler(request)
marqo-1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 235, in app
marqo-1 | raw_response = await run_endpoint_function(
marqo-1 | File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
marqo-1 | return await run_in_threadpool(dependant.call, **values)
marqo-1 | File "/usr/local/lib/python3.8/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
marqo-1 | return await anyio.to_thread.run_sync(func, *args)
marqo-1 | File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
marqo-1 | return await get_asynclib().run_sync_in_worker_thread(
marqo-1 | File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
marqo-1 | return await future
marqo-1 | File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
marqo-1 | result = context.run(func, *args)
marqo-1 | File "/app/src/marqo/tensor_search/api.py", line 250, in create_index
marqo-1 | marqo_config.index_management.create_index(settings.to_marqo_index_request(index_name))
marqo-1 | File "/app/src/marqo/core/index_management/index_management.py", line 150, in create_index
marqo-1 | self._save_index_settings(marqo_index)
marqo-1 | File "/app/src/marqo/core/index_management/index_management.py", line 607, in _save_index_settings
marqo-1 | self.vespa_client.feed_document(
marqo-1 | File "/app/src/marqo/vespa/vespa_client.py", line 238, in feed_document
marqo-1 | self._raise_for_status(resp)
marqo-1 | File "/app/src/marqo/vespa/vespa_client.py", line 907, in _raise_for_status
marqo-1 | raise VespaStatusError(message=response.text, cause=e) from e
marqo-1 | marqo.vespa.exceptions.VespaStatusError: 507: {"pathId":"/document/v1/marqo__settings/marqo__settings/docid/collection_768","id":"id:marqo__settings:marqo__settings::collection_768","message":"[UNKNOWN(251009) @ tcp/14ec605af8d2:19112/default]: ReturnCode(NO_SPACE, External feed is blocked due to resource exhaustion: in content cluster 'content_default': disk on node 0 [14ec605af8d2] is 95.5% full (the configured limit is 75.0%). See https://docs.vespa.ai/en/operations/feed-block.html) "}
marqo-1 | INFO: 192.168.65.1:53140 - "POST /indexes/collection_768 HTTP/1.1" 500 Internal Server Error
Hello @tot-ra . The error you encounter is due to a Vespa default setting. Vespa reserves some disk for internal tasks, it will block feeding of documents when the disk usage is more than 75% by default. Because Marqo stores index information as Vespa document now (this will change in future releases), your creation of index will be blocked if the disk usage of your docker machine is more than 75%.
Solution 1: use docker system prune
to clean up some disk space for docker machine.
Solution 2: Since you are mounting a local folder as vespa folder, you might need to check your local disk usage as well
Solution 3: Mount a fix-sized volume to /opt/vespa/var, using following command and compose config
# Create a 2GB volume
docker volume create --driver local \
--opt type=tmpfs \
--opt device=tmpfs \
--opt o=size=2g,uid=1000 \
vespa-var
# in docker-compose.yml file
volumes:
vespa-var:
external: true
services:
marqo:
image: marqoai/marqo:2.11
ports:
- "8882:8882"
volumes:
- vespa-var:/opt/vespa/
Please let me know if this helps. Thanks
Describe the bug A clear and concise description of what the bug is.
To Reproduce
code:
Logs:
Desktop (please complete the following information):