search

jozu-ai / kitops

Tools for easing the handoff between AI/ML and App/SRE teams.
https://KitOps.ml
Apache License 2.0
266 stars 26 forks source link

Avoid copying layer into local storage during pack #261

Closed amisevsk closed 2 months ago

amisevsk commented 2 months ago

Description

Avoid requiring a data copy during the packing process to cut down on peak storage usage and slightly increase speed. Previously, the process was

  1. Compress layer files to a .tar.gz in $TMPDIR and read digest from file
  2. Construct a manifest and push the .tar.gz into local OCI storage (requires copying)
  3. Remove the temporary .tar.gz from step 1.

Since OCI storage is content-addressible, instead of pushing into OCI storage during step 2, we instead move the temporary .tar.gz directly into its expected location and verify that the OCI store has picked it up correctly.

Linked issues

N/A

amisevsk commented 2 months ago

Performance wise, it's not really much faster assuming the disk is fast -- 77s vs 81s for ghcr.io/jozu-ai/llama-2:7b-chat-q4_0 on my machine, but it should avoid some storage issues we run into with larger models.