Closed fhamborg closed 1 month ago
Hi @fhamborg 👋
Thanks for reporting. That's an interesting one, I'm unfortunately not able to reproduce on my machine. Also weird that there's no std err output from the shard Shard complete standard error output:
.
Do you @OlivierDehaene have any hunch what could cause this?
Kindly let me know in case there's any other information that I could provide :)
Thank you! :)
Any information on what's going on the GPU side while this crashes would be helpful. Something along the lines of this:
nvidia-smi --query-gpu=name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.gpucurrent,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used,reset_status.reset_required,reset_status.drain_and_reset_recommended,compute_cap,ecc.errors.corrected.volatile.total,mig.mode.current,power.draw.instant,power.limit
Sure, here's the output according to your command executed from just before starting the example script from the readme.md until it finishes (executed 5 times per second). I guess you can see that the GPU is never actually employed, which is inline with what I notice when using nvtop. Note that I verified that CUDA works on the machine - both when running, for example, huggingface transformers with pytorch directly on the machine, as well as in other docker containers.
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.51 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.60 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.48 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.47 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.34 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.41 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.11 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.04 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48665 MiB, 3 MiB, No, [N/A], 8.6, [N/A], [N/A], 26.37 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48665 MiB, 3 MiB, No, [N/A], 8.6, [N/A], [N/A], 26.26 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48665 MiB, 3 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.21 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 26.49 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 26.27 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48665 MiB, 3 MiB, No, [N/A], 8.6, [N/A], [N/A], 25.20 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48665 MiB, 3 MiB, No, [N/A], 8.6, [N/A], [N/A], 26.26 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48665 MiB, 3 MiB, No, [N/A], 8.6, [N/A], [N/A], 20.98 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 26.25 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 26.20 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.21 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.30 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.33 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.28 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.13 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.15 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.17 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.27 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.36 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.14 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.28 W, 300.00 W
NVIDIA RTX A6000, 00000000:01:00.0, 550.54.15, P8, 4, 1, 44, 0 %, 0 %, 49140 MiB, 48668 MiB, 1 MiB, No, [N/A], 8.6, [N/A], [N/A], 18.55 W, 300.00 W
Thanks for the added info 👍
Most definitely not an easy debug. I don't have an RTX A6000 at the moment at my disposal but I think replicating your environment would be the best move. Unfortunately bandwidth-wise I won't be able to look at this for a few weeks. My apologies.
If however you find anything that might help point us in the right direction feel free to update here.
Sure! Do you have any idea how I could get more logging information or other helpful information out of the docker container?
@fhamborg , you can try running with some more logs:
model=HuggingFaceH4/zephyr-7b-beta
# share a volume with the Docker container to avoid downloading weights every run
volume=$PWD/data
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data -e LOG_LEVEL=debug,text_generation_router=debug ghcr.io/huggingface/text-generation-inference:2.0.4 --model-id $model
Thanks! I think there's not really new relevant information:
2024-07-22T08:04:11.562343Z INFO text_generation_launcher: Args {
model_id: "VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct",
revision: None,
validation_workers: 2,
sharded: None,
num_shard: None,
quantize: Some(
BitsandbytesNF4,
),
speculate: None,
dtype: None,
trust_remote_code: false,
max_concurrent_requests: 128,
max_best_of: 2,
max_stop_sequences: 4,
max_top_n_tokens: 5,
max_input_tokens: None,
max_input_length: None,
max_total_tokens: None,
waiting_served_ratio: 0.3,
max_batch_prefill_tokens: None,
max_batch_total_tokens: None,
max_waiting_tokens: 20,
max_batch_size: None,
cuda_graphs: None,
hostname: "808aa2c597ef",
port: 80,
shard_uds_path: "/tmp/text-generation-server",
master_addr: "localhost",
master_port: 29500,
huggingface_hub_cache: Some(
"/data",
),
weights_cache_override: None,
disable_custom_kernels: false,
cuda_memory_fraction: 1.0,
rope_scaling: None,
rope_factor: None,
json_output: false,
otlp_endpoint: None,
otlp_service_name: "text-generation-inference.router",
cors_allow_origin: [],
watermark_gamma: None,
watermark_delta: None,
ngrok: false,
ngrok_authtoken: None,
ngrok_edge: None,
tokenizer_config_path: None,
disable_grammar_support: false,
env: false,
max_client_batch_size: 4,
lora_adapters: None,
}
2024-07-22T08:04:11.562407Z INFO hf_hub: Token file not found "/root/.cache/huggingface/token"
2024-07-22T08:04:11.595793Z DEBUG ureq::stream: connecting to huggingface.co:443 at 18.154.63.62:443
2024-07-22T08:04:11.621153Z DEBUG rustls::client::hs: No cached session for DnsName("huggingface.co")
2024-07-22T08:04:11.621233Z DEBUG rustls::client::hs: Not resuming any session
2024-07-22T08:04:11.636117Z DEBUG rustls::client::hs: Using ciphersuite TLS13_AES_128_GCM_SHA256
2024-07-22T08:04:11.636135Z DEBUG rustls::client::tls13: Not resuming
2024-07-22T08:04:11.636198Z DEBUG rustls::client::tls13: TLS1.3 encrypted extensions: [ServerNameAck]
2024-07-22T08:04:11.636204Z DEBUG rustls::client::hs: ALPN protocol is None
2024-07-22T08:04:11.637402Z DEBUG ureq::stream: created stream: Stream(RustlsStream)
2024-07-22T08:04:11.637410Z DEBUG ureq::unit: sending request GET https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct/resolve/main/config.json
2024-07-22T08:04:11.637415Z DEBUG ureq::unit: writing prelude: GET /VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct/resolve/main/config.json HTTP/1.1
Host: huggingface.co
Accept: */*
User-Agent: unkown/None; hf-hub/0.3.2; rust/unknown
Range: bytes=0-0
2024-07-22T08:04:11.946450Z DEBUG ureq::response: Body entirely buffered (length: 1)
2024-07-22T08:04:11.946468Z DEBUG ureq::pool: adding stream to pool: https|huggingface.co|443 -> Stream(RustlsStream)
2024-07-22T08:04:11.946474Z DEBUG ureq::unit: response 206 to GET https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct/resolve/main/config.json
2024-07-22T08:04:11.947251Z DEBUG ureq::stream: connecting to huggingface.co:443 at 18.154.63.57:443
2024-07-22T08:04:11.959248Z DEBUG rustls::client::hs: Resuming session
2024-07-22T08:04:11.972000Z DEBUG rustls::client::hs: Using ciphersuite TLS13_AES_128_GCM_SHA256
2024-07-22T08:04:11.972011Z DEBUG rustls::client::tls13: Resuming using PSK
2024-07-22T08:04:11.972069Z DEBUG rustls::client::tls13: TLS1.3 encrypted extensions: []
2024-07-22T08:04:11.972076Z DEBUG rustls::client::hs: ALPN protocol is None
2024-07-22T08:04:11.972112Z DEBUG ureq::stream: created stream: Stream(RustlsStream)
2024-07-22T08:04:11.972119Z DEBUG ureq::unit: sending request GET https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct/resolve/main/config.json
2024-07-22T08:04:11.972123Z DEBUG ureq::unit: writing prelude: GET /VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct/resolve/main/config.json HTTP/1.1
Host: huggingface.co
Accept: */*
User-Agent: unkown/None; hf-hub/0.3.2; rust/unknown
accept-encoding: gzip
2024-07-22T08:04:12.278568Z DEBUG ureq::response: Body entirely buffered (length: 719)
2024-07-22T08:04:12.278581Z DEBUG ureq::pool: adding stream to pool: https|huggingface.co|443 -> Stream(RustlsStream)
2024-07-22T08:04:12.278585Z DEBUG ureq::unit: response 200 to GET https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct/resolve/main/config.json
2024-07-22T08:04:12.278762Z DEBUG ureq::stream: dropping stream: Stream(RustlsStream)
2024-07-22T08:04:12.278795Z DEBUG ureq::stream: dropping stream: Stream(RustlsStream)
2024-07-22T08:04:12.278826Z INFO text_generation_launcher: Model supports up to 8192 but tgi will now set its default to 4096 instead. This is to save VRAM by refusing large prompts in order to allow more users on the same hardware. You can increase that size using `--max-batch-prefill-tokens=8242 --max-total-tokens=8192 --max-input-tokens=8191`.
2024-07-22T08:04:12.278829Z INFO text_generation_launcher: Default `max_input_tokens` to 4095
2024-07-22T08:04:12.278831Z INFO text_generation_launcher: Default `max_total_tokens` to 4096
2024-07-22T08:04:12.278832Z INFO text_generation_launcher: Default `max_batch_prefill_tokens` to 4145
2024-07-22T08:04:12.278834Z INFO text_generation_launcher: Bitsandbytes doesn't work with cuda graphs, deactivating them
2024-07-22T08:04:12.278918Z INFO download: text_generation_launcher: Starting check and download process for VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct
2024-07-22T08:04:13.298423Z INFO text_generation_launcher: Detected system cuda
2024-07-22T08:04:14.525622Z INFO text_generation_launcher: Files are already present on the host. Skipping download.
2024-07-22T08:04:14.983473Z INFO download: text_generation_launcher: Successfully downloaded weights for VAGOsolutions/Llama-3-SauerkrautLM-70b-Instruct
2024-07-22T08:04:14.983636Z INFO shard-manager: text_generation_launcher: Starting shard rank=0
2024-07-22T08:04:16.122739Z INFO text_generation_launcher: Detected system cuda
2024-07-22T08:04:17.216548Z DEBUG text_generation_launcher: WARNING 07-22 08:04:17 ray_utils.py:46] Failed to import Ray with ModuleNotFoundError("No module named 'ray'"). For distributed inference, please install Ray with `pip install ray`.
2024-07-22T08:04:17.686637Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
rank=0
2024-07-22T08:04:17.686666Z ERROR shard-manager: text_generation_launcher: Shard process was signaled to shutdown with signal 4 rank=0
Error: ShardCannotStart
2024-07-22T08:04:17.786038Z ERROR text_generation_launcher: Shard 0 failed to start
2024-07-22T08:04:17.786050Z INFO text_generation_launcher: Shutting down shards
The only thing that comes to my mind is
2024-07-22T08:04:17.216548Z DEBUG text_generation_launcher: WARNING 07-22 08:04:17 ray_utils.py:46] Failed to import Ray with ModuleNotFoundError("No module named 'ray'"). For distributed inference, please install Ray with `pip install ray`.
but I guess that shouldn't be an issue here since I don't want to perform distributed inference.
Signal 4 means TGI tries to use illegal instruction. Are you running in a VM? Can you check your Docker Nvidia install and that AVX instructions are enabled as in https://github.com/huggingface/text-generation-inference/issues/908?
Thanks for your reply. Yes, I run this in a VM. Though please note that I can run any other huggingface transformers code directly from within python (e.g. in a conda environment) on that VM.
Here's the output of the docker container when executed with --env
(base) felix@heavy-gpu-02:~/dev/anychat/helpers$ docker run --gpus all ghcr.io/huggingface/text-generation-inference:latest --env
2024-07-22T12:49:51.199087Z INFO text_generation_launcher: Runtime environment:
Target: x86_64-unknown-linux-gnu
Cargo version: 1.79.0
Commit sha: da82c63a4ff9c6b8f3d0901cb955c8db04c9a492
Docker label: sha-da82c63
nvidia-smi:
Mon Jul 22 12:49:51 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A6000 On | 00000000:01:00.0 Off | Off |
| 30% 42C P8 19W / 300W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
xpu-smi:
N/A
And here of nvcc
(base) felix@heavy-gpu-02:~/dev/anychat/helpers$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
Okay apparently the missing avx instructions were the issue here. Before changing anything in the vm manager, I had:
cat /proc/cpuinfo | grep -i avx
which yielded nothing, whereas after enabling avx instructions I got this:
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean flushbyasid pausefilter pfthreshold v_vmsave_vmload vgif umip rdpid arch_capabilities
And now the docker image is running nicely :-) I'm sorry for the confusion, I had wrongly assumed that it wouldn't be an issue related to the vm, because I was able to run other torch / transformer code in the vm directly as well as in docker.
Just out of curiosity, do you have an idea, why this particular docker image requires the avx instructions to be available?
Anyways, thanks a lot for the help!
No worries, glad that the issue got resolved @fhamborg 👍
Can't say directly but we're also doing some rework on separating the backend from the frontend. So in the future there would be e.g. different docker images depending on which backend you use (cpu, cuda etc.). The AVX requirement might come from something unexpected which isn't even necessarily used in your case.
System Info
OS: Debian 6.1.85-1 NVIDIA-SMI 550.54.15
Driver Version: 550.54.15
CUDA Version: 12.4 Card: NVIDIA RTX A6000
Information
Tasks
Reproduction
I just copied the example from the readme.md, i.e.:
Stored that in a file named test.sh and executed it (
bash test.sh
).This gives me the following output:
I also tried using more recent versions of the docker image, e.g.,
ghcr.io/huggingface/text-generation-inference:2.1.1
but theError: ShardCannotStart
always occurs.Expected behavior
Should run as described in the readme, e.g., not get this error.