search

RemcodM / git-lfs-s3-caching-adapter

Standalone custom transfer agent for Git LFS that caches LFS objects in a configurable S3 bucket.
MIT License
0 stars 0 forks source link

question about testing with https://github.com/charmbracelet/soft-serve #1

Open gedw99 opened 1 month ago

gedw99 commented 1 month ago

hey @RemcodM

https://github.com/charmbracelet/soft-serve supports git lfs, and my team uses soft server to hold a lot of documents and images. Basically like a file based git cms :)

S3 would be ideal to offloading the storage and also allow geo offload. I use Minio and NATS to spread all s3 buckets to many regions globally.

So I was wondering if you think your git-lfs-s3-caching-adapter could work with soft serve ?

sorry to ask such a lame question, but I read the readme but was not sure.

gedw99 commented 1 month ago

Just saw that its on their todo:, to support s3 storage https://github.com/charmbracelet/soft-serve/blob/main/pkg/web/git_lfs.go#L91

RemcodM commented 1 month ago

It is really difficult to answer if this project will solve your problem without me knowing exactly what you are trying to achieve.

First and foremost: This project aims to deliver S3 caching for Git LFS objects as a oppertunistic functionality: It needs to be configured on the client side, and if it is not configured there, it will still use the Git LFS storage of whatever Git server you are using. For example, it would directly communicate with soft-serve in that case.

When the adapter is configured, it tries to download Git LFS objects from a configured S3 bucket first, before trying upstream. If multiple clients use the same S3 bucket, you can achieve a multi-user cache. I mostly developed this to overcome the high bandwidth cost of GitHub's LFS functionality, by offloading most of the bandwidth to the configured S3 bucket.

I suppose if you can control all the clients you could install this and start caching the Git LFS objects in Minio as you see fit. Please remember that new Git LFS objects will always be uploaded to the upstream Git server. That is the main source of truth and if the cache would fail miserably, everything on the remote side should still be consistent. Only upon checking out locally unknown Git LFS objects the cache would be checked first.

With multiple clients the cache would be filled very fast probably, as every LFS object downloaded from the Git server missing in the caching bucket would be uploaded automatically.