Closed pseudotensor closed 3 weeks ago
If I try pytorch backend, I get on startup:
language_model.model.layers.79.input_layernorm.weight: 54%|?2024-07-26 20:45:57,609 - lmdeploy - ERROR - RuntimeError: Internal Triton PTX codegen error:
ptxas fatal : Value 'sm_90a' is not defined for option 'gpu-name'
2024-07-26 20:45:57,609 - lmdeploy - ERROR - <Triton> test failed!
Please ensure it has been installed correctly.
Same for InternVL 1-5 model.
My nvidia-smi if helpful:
Fri Jul 26 20:51:43 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07 Driver Version: 535.161.07 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA H100 80GB HBM3 On | 00000000:0F:00.0 Off | 0 |
| N/A 31C P0 119W / 700W | 67915MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA H100 80GB HBM3 On | 00000000:2D:00.0 Off | 0 |
| N/A 36C P0 120W / 700W | 68135MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA H100 80GB HBM3 On | 00000000:44:00.0 Off | 0 |
| N/A 33C P0 124W / 700W | 68135MiB / 81559MiB | 1% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA H100 80GB HBM3 On | 00000000:5B:00.0 Off | 0 |
| N/A 37C P0 124W / 700W | 67557MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 4 NVIDIA H100 80GB HBM3 On | 00000000:89:00.0 Off | 0 |
| N/A 31C P0 114W / 700W | 71661MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 5 NVIDIA H100 80GB HBM3 On | 00000000:A8:00.0 Off | 0 |
| N/A 36C P0 118W / 700W | 78617MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 6 NVIDIA H100 80GB HBM3 On | 00000000:C0:00.0 Off | 0 |
| N/A 37C P0 123W / 700W | 80371MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 7 NVIDIA H100 80GB HBM3 On | 00000000:D8:00.0 Off | On |
| N/A 32C P0 120W / 700W | 63356MiB / 81559MiB | N/A Default |
| | | Enabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| MIG devices: |
+------------------+--------------------------------+-----------+-----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+================================+===========+=======================|
| 7 1 0 0 | 32519MiB / 40448MiB | 60 0 | 3 0 3 0 3 |
| | 3MiB / 65535MiB | | |
+------------------+--------------------------------+-----------+-----------------------+
| 7 2 0 1 | 30837MiB / 40448MiB | 60 0 | 3 0 3 0 3 |
| | 3MiB / 65535MiB | | |
+------------------+--------------------------------+-----------+-----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2083468 C /opt/py38/bin/python3 67908MiB |
| 1 N/A N/A 2083468 C /opt/py38/bin/python3 68128MiB |
| 2 N/A N/A 2083468 C /opt/py38/bin/python3 68128MiB |
| 3 N/A N/A 2083468 C /opt/py38/bin/python3 67550MiB |
| 4 N/A N/A 1172218 C python3 71642MiB |
| 5 N/A N/A 1171696 C python3 78600MiB |
| 6 N/A N/A 1181117 C /opt/py38/bin/python3 80364MiB |
| 7 1 0 1465874 C python 32484MiB |
| 7 2 0 1475745 C python3 30802MiB |
+---------------------------------------------------------------------------------------+
This is after the model is already on the 0-3 GPUs
But some suggest cuda 11.8 is fine for sm_90: https://discuss.pytorch.org/t/cuda-version-conundrum/185714/3
Ah, even OpenGVLab/InternVL-Chat-V1-5 segfaults.
So lmdeploy is broken somehow, because I'm using the same docker build scripts as I used for already-running cases just fine, only difference is using latest lmdeploy repo hash
I'm also confused by this in readme:
Since v0.3.0, The default prebuilt package is compiled on CUDA 12. However, if CUDA 11+ is required, you can install lmdeploy by
But the docker/Dockerfile still references cu118 and (I guess) uses tritonserver that only as cuda 11.8.
Is this a problem for deploying on H100? It's worked on lmdeploy from (maybe) 2-3 weeks ago, so I guess not, but maybe the pytorch issue is related.
Why can't the docker image use updated triton server image?
https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-24-06.html#rel-24-06
that uses cuda 12.5?
And why is the triton server image used as base image at all? Seems overly complicaetd and you don't even use the triton server. Why not just normal Ubuntu with python 3.10?
The exact same build process but on f6138148aaf30ce019e38c9f4295d00e8ca4d66d works fine, no segfault, so definitely a regression.
@pseudotensor hi, thanks for your feedback. Looks like it only happens in docker image of triton-serve based. https://github.com/InternLM/lmdeploy/pull/1971 can fix it. As for docker image, we will consider provide a docker image of cuda12.x later.
@pseudotensor hi, could you kindly try on this updated dockerfile from https://github.com/InternLM/lmdeploy/pull/2182? Any feedback would be greatly appreciated.
Why can't the docker image use updated triton server image?
https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-24-06.html#rel-24-06
that uses cuda 12.5?
And why is the triton server image used as base image at all? Seems overly complicaetd and you don't even use the triton server. Why not just normal Ubuntu with python 3.10?
The initial version of lmdeploy inherits FasterTransformer and triton inference server. With the development of lmdeploy, we are gradually removing them #1986 The cu12 docker image (#2182) won't be released until full test passes
Still hitting segfaults, unsure same issue: https://github.com/InternLM/lmdeploy/issues/2223
Probably same, so not fixed.
Hi, @pseudotensor we tried A100 (*8) but were not able to reproduce this issue. I was wondering if there is any way that we can access your env and debugging it?
Hi I plan to do the debugging thing: https://github.com/InternLM/lmdeploy/issues/2223#issuecomment-2266335124
Just busy with other stuff.
I'm unable to give access to the machine directly, but we can do a shared debugging session if that's helpful. You can email me at pseudotensor@gmail.com to setup details
Checklist
Describe the bug
Text handling by the model is fine, but any image leads to crash
Reproduction
Just do any single image request as one would for other vision models. This is sufficient to cause crash every time:
I haven't used the latest lmdeploy for other models like internVL 1-5 that work fine with older version, so it's possible those are broken too. I'll try InternVL 1-5 to see if lmdeploy is generally broken.
Environment
Using latest docker w/ extra build for vision stuff on 4*H100
See how docker image is built here: https://github.com/InternLM/lmdeploy/issues/2163
Error traceback