CUDA out of memroy - Benjamin.

[lipere123]

./exo-cli-3.1-70b.sh hello Go for :

!/bin/bash

/usr/bin/curl --progress-bar --connect-timeout 1800 --max-time 1800 http://edgenode2:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama-3.1-70b", "messages": [{"role": "user", "content": "hello"}], "temperature": 0.7 }'

{"detail": "Error processing prompt (see logs with DEBUG>=2): <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = \"Unexpected <class 'RuntimeError'>: CUDA Error 2, out of memory\"\n\tdebug_error_string = \"UNKNOWN:Error received from peer {created_time:\"2024-10-10T05:40:20.356329187+00:00\", grpc_status:2, grpc_message:\"Unexpected <class \'RuntimeError\'>: CUDA Error 2, out of memory\"}\"\n>"}

[fullofcaffeine]

I'm also intermittently experiencing this, see: https://github.com/exo-explore/exo/issues/235.

[FFAMax]

Seems like unable properly split model into chunks so model can be portionally loaded across several nodes. (llama 3.1 8B unable split across 3x8GB GPUs)