Closed zaventh closed 1 year ago
Hi, Can you PR the issue with Triton 22? You can check the logs of the Triton Server to ensure that you are not running out of memory. It also takes sometime to load the model on the Inference Server the first time.
@zaventh Please go ahead with the PR for Triton 22 (please do not update the image)
The same issue already discussed in #98, but it is not resolved despite that issue being closed with no obvious resolution.
When following the guide here https://docs.coreweave.com/compass/examples/triton-inference-server-fastertransformer#deploying-the-kubernetes-resources
The following error is the resultant:
While running the transform, the disks look like this:
It appears when transforming, it does approach around 115GB of space, not <100GB:
I increased the PVC to 200GB and I receive a subsequent error:
This is with consideration that the following line is present in the example:
https://github.com/coreweave/kubernetes-cloud/blob/master/online-inference/fastertransformer/download-weights-job-gpt-neox.yml#L31
However this line:
https://github.com/coreweave/kubernetes-cloud/blob/master/online-inference/fastertransformer/download-weights-job-gpt-neox.yml#L36 appears to have a typo in it:
Notice the absence of
gpt-neox
in the path.To summarize, the neoX job appears to require approximately 115GB of storage, not 100GB and typos in the command path prevent the script from running successfully. The following is a diff of the changes I have made to successfully pass this step:
Finally, the stage to start the Inference server
Also does not complete successfully as-is with the output reporting only:
I am still debugging this stage, however, and it may be more appropriate as a separate issue as it could be due to upgrading from Triton 22->23.
I am happy to PR the above changes if you are accepting.