search

InftyAI / llmaz

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!
Apache License 2.0
30 stars 10 forks source link

Mount /dev/shm for shared memory files #73

Open kerthcet opened 3 months ago

kerthcet commented 3 months ago

What would you like to be added:

This is how it should look like:

    volumeMounts:
    - mountPath: /dev/shm
      name: dshm

But this memory size is unknown.

Why is this needed:

Accelerate model loading and model inference.

Completion requirements:

This enhancement requires the following artifacts:

The artifacts should be linked in subsequent comments.

kerthcet commented 3 months ago

/kind feature