At WarpBuild, we provide an alternative to GitHub's Cache in our custom runners. Recently, some of our customers have requested support for a cache backend in buildkit, similar to GitHub Actions (GHA).
We initially explored the S3 backend, but it doesn't suit our needs as it lacks support for pre-signed URLs and requires access keys and secret keys, which we cannot inject into all of our runners. The GHA backend also poses challenges because, while it downloads using pre-signed URLs, uploads are managed through GitHub's backend servers.
Given the complexity of maintaining support for various cache backends from different services, we propose contributing a generic remote cache backend that uses pre-signed URLs for both downloading and uploading layers.
This backend would be straightforward, requiring only a remote URL and a token as input. The remote APIs would be responsible for generating pre-signed URLs for downloads and uploads. This approach can be supported by any storage type that offers pre-signed URL capabilities (e.g., S3, GCS, Azure Blob, R2, etc.).
The methods would be similar to those in the gha backend:
get: Sends a pre-signed URL, which will then be downloaded.
download: Downloads the file from the pre-signed URL.
reserve: Sends a pre-signed URL for upload.
upload: Uploads the file directly to the pre-signed URL.
commit: Signals to the backend that the file has been uploaded.
As this is a significant feature, we would like to seek your opinions before proceeding. Is this a feature that BuildKit would consider accepting? Additionally, could you provide guidance on how we should contribute such a feature?
guptaankit015
commented
1 month ago
At WarpBuild, we provide an alternative to GitHub's Cache in our custom runners. Recently, some of our customers have requested support for a cache backend in
buildkit, similar to GitHub Actions (GHA).We initially explored the S3 backend, but it doesn't suit our needs as it lacks support for pre-signed URLs and requires access keys and secret keys, which we cannot inject into all of our runners. The GHA backend also poses challenges because, while it downloads using pre-signed URLs, uploads are managed through GitHub's backend servers.
Given the complexity of maintaining support for various cache backends from different services, we propose contributing a generic remote cache backend that uses pre-signed URLs for both downloading and uploading layers.
This backend would be straightforward, requiring only a remote URL and a token as input. The remote APIs would be responsible for generating pre-signed URLs for downloads and uploads. This approach can be supported by any storage type that offers pre-signed URL capabilities (e.g., S3, GCS, Azure Blob, R2, etc.).
The methods would be similar to those in the
ghabackend:get: Sends a pre-signed URL, which will then be downloaded.download: Downloads the file from the pre-signed URL.reserve: Sends a pre-signed URL for upload.upload: Uploads the file directly to the pre-signed URL.commit: Signals to the backend that the file has been uploaded.As this is a significant feature, we would like to seek your opinions before proceeding. Is this a feature that BuildKit would consider accepting? Additionally, could you provide guidance on how we should contribute such a feature?
Regards,
Prajjwal