[BUG] "POST /indexes/marqo-simplewiki-demo-all HTTP/1.1" 500 Internal Server Error

[llermaly]

Describe the bug Server error 500 trying to run the SimpleWiki example. Tried with 0.0.5 and 0.0.3 as the demo shows.

gllermaly@ubuntu-s-2vcpu-4gb-amd-nyc1-01:~/marqo/SimpleWiki$ python3 simple_wiki_demo.py loaded data with 188557 entries Traceback (most recent call last): File "/home/gllermaly/.local/lib/python3.10/site-packages/marqo/_httprequests.py", line 131, in __validate request.raise_for_status() File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:8882/indexes/marqo-simplewiki-demo-all

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/gllermaly/marqo/SimpleWiki/simple_wiki_demo.py", line 64, in client.create_index(index_name, model='onnx/all_datasets_v4_MiniLM-L6') File "/home/gllermaly/.local/lib/python3.10/site-packages/marqo/client.py", line 62, in create_index return Index.create( File "/home/gllermaly/.local/lib/python3.10/site-packages/marqo/index.py", line 78, in create return req.post(f"indexes/{index_name}", body={ File "/home/gllermaly/.local/lib/python3.10/site-packages/marqo/_httprequests.py", line 99, in post return self.send_request(requests.post, path, body, content_type) File "/home/gllermaly/.local/lib/python3.10/site-packages/marqo/_httprequests.py", line 77, in send_request return self.__validate(response) File "/home/gllermaly/.local/lib/python3.10/site-packages/marqo/_httprequests.py", line 134, in __validate convert_to_marqo_error_and_raise(response=request, err=err) File "/home/gllermaly/.local/lib/python3.10/site-packages/marqo/_httprequests.py", line 148, in convert_to_marqo_error_and_raise raise MarqoWebError(message=response_msg, code=code, error_type=error_type, marqo.errors.MarqoWebError: MarqoWebError: MarqoWebError Error message: {'message': "HTTPSConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /marqo-simplewiki-demo-all (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f3bcd18cc10>: Failed to establish a new connection: [Errno 111] Connection refused'))", 'code': 'backend_communication_error', 'type': 'internal', 'link': ''} status_code: 500, type: internal, code: backend_communication_error, link:

To Reproduce Steps to reproduce the behavior:

1. Install marqo
2. Run examples/SimpleWiki

Expected behavior Demo should work

Desktop (please complete the following information):

• OS: Ubuntu 22 server DO fresh droplet

[llermaly]

Not sure if this is related but I see this when running docker before the marqo success

dockerd is running OpenSearch not found; running OpenSearch Unable to find image 'marqoai/marqo-os:0.0.2' locally Error response from daemon: No such container: marqo-os Error: failed to start containers: marqo-os 0.0.2: Pulling from marqoai/marqo-os 017c82d6391f: Pull complete d192a8214bfc: Pull complete e3d96ac441d0: Pull complete 58812b24fd36: Pull complete 4f4fb700ef54: Pull complete cd70a404e73a: Pull complete 0d27a856ae42: Pull complete b44bb45da9a5: Pull complete 1340728d1abc: Pull complete Digest: sha256:f5320b8778f4be248d7affe797898711f9857985c502adc6c4346ca825170897 Status: Downloaded newer image for marqoai/marqo-os:0.0.2 fb3ef73b57532b252546628688cb8d5ce4779fbdf5b845ffe0b3174c2a10ac00 time="2022-10-16T19:15:57.844286896Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2022-10-16T19:15:57.844393329Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2022-10-16T19:15:57.844404971Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2022-10-16T19:15:57.845128064Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/fb3ef73b57532b252546628688cb8d5ce4779fbdf5b845ffe0b3174c2a10ac00 pid=1667 runtime=io.containerd.runc.v2 time="2022-10-16T19:15:58.059404481Z" level=error msg="failed to enable controllers ([cpuset cpu io memory hugetlb pids rdma misc])" error="failed to write subtree controllers [cpuset cpu io memory hugetlb pids rdma misc] to \"/sys/fs/cgroup/docker/cgroup.subtree_control\": write /sys/fs/cgroup/docker/cgroup.subtree_control: no such file or directory" time="2022-10-16T19:15:58.061049215Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/docker/fb3ef73b57532b252546628688cb8d5ce4779fbdf5b845ffe0b3174c2a10ac00/memory.events\": no such file or directory"

[jn2clark]

Thanks @llermaly . I will take a look. A couple of questions 1) which commands did you use to start marqo? 2) how long after starting marqo did you try and run the simple-wiki script? the first run takes a little longer as there are a couple of downloads 3) could you run any of the readme examples?

[llermaly]

Hi @jn2clark I started with the ones from the main README :

docker rm -f marqo;
docker pull marqoai/marqo:0.0.5;
docker run --name marqo -it --privileged -p 8882:8882 --add-host host.docker.internal:host-gateway marqoai/marqo:0.0.5

Then installing the python client and ran the python script

python3 simple_wiki_demo.py

What I noticed is marqo-os was not running after doing this. Tried to run the marqo-os container manually but no luck, same error.

I will try removing all the containers/images/volumes and trying again

[llermaly]

Tried spinning up opensearch independently as the M1 guide says and spins up but once I run the demo py script opensearch crashes :

Killing performance analyzer process 34 ./opensearch-docker-entrypoint.sh: line 116: 33 Killed "$@" "${opensearch_opts[@]}" OpenSearch exited with code 137 Performance analyzer exited with code 143

[jn2clark]

Are you running on an arm based machine? If not, you will need to modify the command slighlty: docker rm -f marqo-os; docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" marqoai/marqo-os:0.0.2 Note the non -arm tag

[llermaly]

Yeah, that's what I did. Opensearch comes up good, I can curl localhost:9200 (admin/admin creds) and see the cluster status. But once I run the python script it goes down

[llermaly]

At least running both things separately I can ingest some docs, so the issue is in the python script

[jn2clark]

Thanks. It could be related to the system RAM. What machine are you running it on? You can try and change this line https://github.com/marqo-ai/marqo/blob/mainline/examples/SimpleWiki/simple_wiki_demo.py#L69 to use only a single process and see if that works. responses = client.index(index_name).add_documents(data, device=device, processes=1, batch_size=50)

[llermaly]

No luck reducing processes. This is the machine:

4 GB Memory / 2 Intel vCPUs / 80 GB Disk

I will try bumping up

[jn2clark]

Ah, yeah might be worth moving to 8GB RAM

[llermaly]

Thank you, it is not crashing but loading now.

Do you have any idea why marqos-os is not spinning up with the initial run command?

[jn2clark]

Great! Not sure why the first method did not work. We will take a look and see if we can reproduce. My only other suggestion (might be a long-shot) is to upgrade docker and try again.

[kdewald]

I had similar issues before with this example, caused by two reasons:

• Depending on your batch size, you might be sending a request that's too large to OpenSearch and it gets rejected there.
• (In GPU RAM) The language model itself is 2GB, and needs around 4GB more to parse roughly 100 documents at the time. The underlying model will run out of memory and the thrown exception will crash Marqo.

My suggestion would be to make that demo smaller to run on less powerful hardware.

[jn2clark]

I updated the demo to automatically handle excessively large documents. https://github.com/marqo-ai/marqo/blob/mainline/examples/SimpleWiki/simple_wiki_demo.py#L57 . if you have any other problems, you can re-open!