search

PygmalionAI / aphrodite-engine

PygmalionAI's large-scale inference engine
https://pygmalion.chat
GNU Affero General Public License v3.0
660 stars 80 forks source link

[Bug]: PermissionError: [Errno 13] Permission denied: '/app/aphrodite-engine/.triton' #458

Closed theobjectivedad closed 3 weeks ago

theobjectivedad commented 3 weeks ago

Your current environment

Note I am running alpindale/aphrodite-engine:latest as of 2024-05-08.

Collecting environment information...
PyTorch version: 2.2.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000

Nvidia driver version: 535.86.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             64
On-line CPU(s) list:                0-63
Vendor ID:                          AuthenticAMD
Model name:                         AMD Ryzen Threadripper PRO 5975WX 32-Cores
CPU family:                         25
Model:                              8
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          1
Stepping:                           2
Frequency boost:                    enabled
CPU max MHz:                        7006.6401
CPU min MHz:                        1800.0000
BogoMIPS:                           7186.95
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization:                     AMD-V
L1d cache:                          1 MiB (32 instances)
L1i cache:                          1 MiB (32 instances)
L2 cache:                           16 MiB (32 instances)
L3 cache:                           128 MiB (4 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.2.0
[pip3] triton==2.2.0
[conda] Could not collect ROCM Version: Could not collect
Aphrodite Version: 0.5.2
Aphrodite Build Flags:
CUDA Archs: 6.1 7.0 7.5 8.0 8.6 8.9 9.0+PTX; ROCm: Disabled

Additionally, here is the docker run command I'm using (triggered via a Makefile):

docker run -it -d \
    --name=$(APHRODITE_NAME)-$(APHRODITE_PROFILE) \
    --restart=always \
    --shm-size=15g \
    --ulimit memlock=-1 \
    --ipc=host \
    --entrypoint=python3 \
    --gpus='"device=$(APHRODITE_GPUS)"' \
    --publish=$(APHRODITE_PORT):8000 \
    --volume=/models:/models:ro \
    --health-cmd="timeout 5 bash -c 'cat < /dev/null > /dev/tcp/localhost/8000'" \
    --health-start-period=240s \
    --health-interval=15s \
    --health-timeout=8s \
    --health-retries=3 \
    $(APHRODITE_CONTAINER) \
        -m aphrodite.endpoints.openai.api_server \
        --model /models/$(APHRODITE_MODEL) \
        --worker-use-ray \
        --context-shift \
        --tensor-parallel-size 4 \
        --gpu-memory-utilization 0.9 \
        --served-model-name $(APHRODITE_MODEL_NAME) \
        --kv-cache-dtype auto \
        --load-format auto \
        --tokenizer-mode auto \
        --dtype auto \
        --response-role gpt \
        --port 8000 \
        --host 0.0.0.0

🐛 Describe the bug

Description

When executing a completion request I'm getting the exception below. I believe the root cause is the official container user is running as UID 1000 and /app/aphrodite-engine is owned by root:root. Seems to only happen under heavy load. Can replicate with unquantized version of miqu-1-70b-sf.

Full error message

INFO:     172.17.0.1:3226 - "POST /v1/completions HTTP/1.1" 500
ERROR:    Exception in ASGI application
ERROR:    Traceback (most recent call last):
ERROR:      File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR:        result = await app(  # type: ignore[func-returns-value]
ERROR:      File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR:        return await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
ERROR:        await super().__call__(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR:        await self.app(scope, receive, _send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR:        await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 758, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 778, in app
ERROR:        await route.handle(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 299, in handle
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 79, in app
ERROR:        await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 74, in app
ERROR:        response = await func(request)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
ERROR:        raw_response = await run_endpoint_function(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR:        return await dependant.call(**values)
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 281, in create_completion
ERROR:        generator = await openai_serving_completion.create_completion(
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 177, in create_completion
ERROR:        async for i, res in result_generator:
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 81, in consumer
ERROR:        raise item
ERROR:      File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR:        result = await app(  # type: ignore[func-returns-value]
ERROR:      File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR:        return await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
ERROR:        await super().__call__(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR:        await self.app(scope, receive, _send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR:        await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 758, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 778, in app
ERROR:        await route.handle(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 299, in handle
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 79, in app
ERROR:        await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 74, in app
ERROR:        response = await func(request)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
ERROR:        raw_response = await run_endpoint_function(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR:        return await dependant.call(**values)
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 281, in create_completion
ERROR:        generator = await openai_serving_completion.create_completion(
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 177, in create_completion
ERROR:        async for i, res in result_generator:
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 81, in consumer
ERROR:        raise item
ERROR:      File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR:        result = await app(  # type: ignore[func-returns-value]
ERROR:      File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR:        return await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
ERROR:        await super().__call__(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR:        await self.app(scope, receive, _send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR:        await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 758, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 778, in app
ERROR:        await route.handle(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 299, in handle
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 79, in app
ERROR:        await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 74, in app
ERROR:        response = await func(request)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
ERROR:        raw_response = await run_endpoint_function(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR:        return await dependant.call(**values)
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 281, in create_completion
ERROR:        generator = await openai_serving_completion.create_completion(
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 177, in create_completion
ERROR:        async for i, res in result_generator:
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 81, in consumer
ERROR:        raise item
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 66, in producer
ERROR:        async for item in iterator:
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 625, in generate
ERROR:        raise e
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 619, in generate
ERROR:        async for request_output in stream:
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 72, in __anext__
ERROR:        raise result
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 66, in producer
ERROR:        async for item in iterator:
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 625, in generate
ERROR:        raise e
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 619, in generate
ERROR:        async for request_output in stream:
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 72, in __anext__
ERROR:        raise result
ERROR:      File "/app/aphrodite-engine/aphrodite/endpoints/openai/serving_completions.py", line 66, in producer
ERROR:        async for item in iterator:
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 625, in generate
ERROR:        raise e
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 619, in generate
ERROR:        async for request_output in stream:
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 72, in __anext__
ERROR:        raise result
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 33, in _raise_exception_on_finish
ERROR:        task.result()
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 468, in run_engine_loop
ERROR:        has_requests_in_progress = await asyncio.wait_for(
ERROR:      File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
ERROR:        return fut.result()
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 442, in engine_step
ERROR:        request_outputs = await self.engine.step_async()
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 208, in step_async
ERROR:        all_outputs = await self._run_workers_async(
ERROR:      File "/app/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 293, in _run_workers_async
ERROR:        all_outputs = await asyncio.gather(*coros)
ERROR:      File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:        result = self.fn(*self.args, **self.kwargs)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR:        return func(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/task_handler/worker.py", line 235, in execute_model
ERROR:        output = self.model_runner.execute_model(seq_group_metadata_list,
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR:        return func(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/task_handler/model_runner.py", line 692, in execute_model
ERROR:        hidden_states = model_executable(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR:        return self._call_impl(*args, **kwargs)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR:        return forward_call(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 415, in forward
ERROR:        hidden_states = self.model(input_ids, positions, kv_caches,
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR:        return self._call_impl(*args, **kwargs)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR:        return forward_call(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 340, in forward
ERROR:        hidden_states, residual = layer(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR:        return self._call_impl(*args, **kwargs)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR:        return forward_call(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 287, in forward
ERROR:        hidden_states = self.self_attn(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR:        return self._call_impl(*args, **kwargs)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR:        return forward_call(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/modeling/models/llama.py", line 228, in forward
ERROR:        attn_output = self.attn(q, k, v, k_cache, v_cache, input_metadata,
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
ERROR:        return self._call_impl(*args, **kwargs)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl
ERROR:        return forward_call(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/modeling/layers/attention.py", line 213, in forward
ERROR:        context_attention_fwd(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR:        return func(*args, **kwargs)
ERROR:      File "/app/aphrodite-engine/aphrodite/modeling/layers/triton_kernel/prefix_prefill.py", line 684, in context_attention_fwd
ERROR:        _fwd_kernel[grid](
ERROR:      File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 532, in run
ERROR:        self.cache[device][key] = compile(
ERROR:      File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 484, in compile
ERROR:        fn_cache_manager = get_cache_manager(make_hash(fn, target, get_env_vars(), _device_backend, **kwargs))
ERROR:      File "/usr/local/lib/python3.10/dist-packages/triton/runtime/cache.py", line 152, in get_cache_manager
ERROR:        return __cache_cls(key)
ERROR:      File "/usr/local/lib/python3.10/dist-packages/triton/runtime/cache.py", line 66, in __init__
ERROR:        os.makedirs(self.cache_dir, exist_ok=True)
ERROR:      File "/usr/lib/python3.10/os.py", line 215, in makedirs
ERROR:        makedirs(head, exist_ok=exist_ok)
ERROR:      File "/usr/lib/python3.10/os.py", line 215, in makedirs
ERROR:        makedirs(head, exist_ok=exist_ok)
ERROR:      File "/usr/lib/python3.10/os.py", line 225, in makedirs
ERROR:        mkdir(name, mode)
ERROR:    PermissionError: [Errno 13] Permission denied: '/app/aphrodite-engine/.triton'
theobjectivedad commented 3 weeks ago

As a workaround I can pass --user=root to the docker run args although this obviously this isn't a good practice. If anyone else uses this workaround I recommend the following aphrodite-engine options: always use --load-format safetensors and never use --trust-remote-code.

Looking at the contents of /app/aphrodite-engine/.triton:

root@6c985414e457:~/.triton# find .
.
./cache
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttir
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/__grp___fwd_kernel.json
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttgir
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.llir
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ptx
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.json
./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.cubin
./cache/e848d61dc399d578c7a9dd3c6ff8a1a1
./cache/e848d61dc399d578c7a9dd3c6ff8a1a1/_fwd_kernel.so
./cache/6e97c2a1f7a095255f6dd5de1807841d
./cache/6e97c2a1f7a095255f6dd5de1807841d/cuda_utils.so
./dump
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttir
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ttgir
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.llir
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.ptx
./dump/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.cubin
root@6c985414e457:~/.triton# more ./cache/0d823c7fdd45a2b9f6ef82f9d235a68d/_fwd_kernel.json
{"num_warps": 8, "num_ctas": 1, "num_stages": 1, "enable_warp_specialization": false, "enable_persistent": false, "constants": {"14": 1, "31": 1, "35": 1, "36": 128, "37": 128, "38": 1
28}, "debug": null, "target": {"capability": 86, "num_warps": 8, "enable_fp_fusion": true}, "AMDGCN_ENABLE_DUMP": false, "DISABLE_FAST_REDUCTION": false, "DISABLE_MMA_V3": false, "ENAB
LE_TMA": false, "LLVM_IR_ENABLE_DUMP": false, "MLIR_ENABLE_DUMP": false, "TRITON_DISABLE_LINE_INFO": false, "device_type": "cuda", "shared": 65538, "name": "_fwd_kernel_0d1d2d3d4d5d67d
8d9d10de11e12d1314c15de16de17de18de19de20de21de22de23de24de25de26de27de28de29de30e31c32de33de34de35c", "clusterDims": [1, 1, 1]}
AlpinDale commented 3 weeks ago

That's a weird one - the process should have rw access to the /app/aphrodite-engine directory. I guess we'll need to explicitly modify the perms for the directory in the Dockerfile. Thanks for the report.

theobjectivedad commented 3 weeks ago

Cool - I'll send over a PR