search

kiwigrid / k8s-sidecar

This is a docker container intended to run inside a kubernetes cluster to collect config maps with a specified label and store the included files in a local folder.
MIT License
580 stars 181 forks source link

Using .url functionality with large files can result in OOMKill #336

Open cryptk opened 7 months ago

cryptk commented 7 months ago

Using the k8s-sidecar .url functionality to download files from a URL, if those files are quite large (in this case they are model files from HugingFace.co for AI systems) it is quite possible for the container to run out of memory. There should be an option to download the files directly to disk rather than buffering the downloads in memory. You could do this with a .partial suffix on the filename while the file is being downloaded and then have a rename at the end when the file download is complete in order to prevent issues if the sidecar were interrupted mid-download.

github-actions[bot] commented 2 weeks ago

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.