search

milkey-mouse / git-lfs-s3-proxy

Store Git LFS files on any service with an S3 API
https://git-lfs-s3-proxy.pages.dev
Creative Commons Zero v1.0 Universal
59 stars 39 forks source link

Test on files large enough to require multipart upload #7

Open milkey-mouse opened 10 months ago

whoabuddy commented 3 months ago

So first off, love the tool, the setup from end to end was a breeze with all the instructions and super helpful!!

I was using git-lfs to store audio and video from our meetings and way exceeded the GitHub quote without realizing it (8gb/1gb).

Repo: https://github.com/aibtcdev/communication

The audio files in mp3 format are ~50mb in size, video files average closer to ~550mb in mp4 format.

I have everything configured and setup correctly as far as I can tell, but I'm getting HTTP 413 errors when trying to upload a 41M audio file, log snippets below:

$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 git lfs push origin main meetings/audio/Bitcoin\ x\ AI\ WG\ \(2024-02-01\ 09_03\ GMT-7\)\ Intro\ Meeting\ .mp3
...
06:26:23.773720 trace git-lfs: tq: sending batch of size 2                                                                                                                                                           
06:26:23.773882 trace git-lfs: api: batch 2 files
06:26:23.774164 trace git-lfs: HTTP: POST https://659c53613df106d51e6a44f5e69ed0d7:36ca01dfa94ec8a24dadb32fef76752e33ce3f32f9ee3ca08c32a7115dc163a8@git-lfs-s3-proxy.aibtc.dev/bucket.aibtc.dev/aibtcdev-communications/objects/batch
06:26:23.975312 trace git-lfs: HTTP: 200
06:26:23.980332 trace git-lfs: HTTP: {"transfer":"basic","objects":[{"oid":"0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89","size":566756978,"authenticated":true,"actions":{"upload":{"href":"https://bucket.aibtc.dev/aibtcdev-communications/0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89?X-Amz-Expires=3600&X-Amz-Date=20240726T132623Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=659c53613df106d51e6a44f5e69ed0d7%2F20240726%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=7fcb2b3df24fb6
06:26:23.980348 trace git-lfs: HTTP: 9ed62f6b803750c32bf22d87f3c121e0bd4b7c2338f4f16127","expires_in":3600}}},{"oid":"4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43","size":55989959,"authenticated":true,"actions":{"upload":{"href":"https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43?X-Amz-Expires=3600&X-Amz-Date=20240726T132623Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=659c53613df106d51e6a44f5e69ed0d7%2F20240726%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=64c86a703d73a58954d530433ac8f0a1d2e52e75b4e3c0c1d67d3475e4ca09c5","expires_in":3600}}}]}
06:26:23.980432 trace git-lfs: tq: starting transfer adapter "basic"                                                                                                                                                 
06:26:23.980438 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
06:26:23.980452 trace git-lfs: xfer: adapter "basic" started
06:26:23.980469 trace git-lfs: xfer: adapter "basic" worker 4 starting
06:26:23.980477 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
06:26:23.980448 trace git-lfs: xfer: adapter "basic" worker 0 starting
06:26:23.980488 trace git-lfs: xfer: adapter "basic" worker 1 starting
06:26:23.980493 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
06:26:23.980473 trace git-lfs: xfer: adapter "basic" worker 6 starting
06:26:23.980464 trace git-lfs: xfer: adapter "basic" worker 3 starting
06:26:23.980511 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
06:26:23.980498 trace git-lfs: xfer: adapter "basic" worker 2 starting
06:26:23.980484 trace git-lfs: xfer: adapter "basic" worker 7 starting
06:26:23.980527 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
06:26:23.980496 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89"
06:26:23.980530 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
06:26:23.980483 trace git-lfs: xfer: adapter "basic" worker 5 starting
06:26:23.980650 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
06:26:23.980502 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
06:26:23.980952 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89
06:26:24.072527 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
06:26:24.072544 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
06:26:24.072549 trace git-lfs: xfer: adapter "basic" worker 3 processing job for "4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43"
06:26:24.072556 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
06:26:24.072563 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
06:26:24.072538 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
06:26:24.072572 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
06:26:24.072554 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
06:26:24.072842 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:26:24.140272 trace git-lfs: HTTP: 413
06:26:24.140367 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89"
06:26:24.140386 trace git-lfs: tq: retrying object 0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89: LFS: Client error &{%!!(string=https) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=bucket.aibtc.dev) %!!(string=/aibtcdev-communications/0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=X-Amz-Expires=3600&X-Amz-Date=20240726T132623Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=659c53613df106d51e6a44f5e69ed0d7%2F20240726%2Fus-east-1%2F%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=7fcb2b3df24fb69ed62f6b803750c32bf22d87f3c121e0bd4b7c2338f4f16127) %!!(string=) %!!(string=)}s(MISSING) from HTTP 413
06:26:24.140396 trace git-lfs: tq: enqueue retry #1 after 0.25s for "0847e309058f07274733e22a9c125184cf4967bf4c8795e057d8f9fca27a9c89" (size: 566756978)
06:27:59.810771 trace git-lfs: HTTP: 401 | 588 KB/s                                                                                                                                                                  
06:27:59.810891 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:29:33.420108 trace git-lfs: HTTP: 401
06:29:33.420241 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:31:07.045073 trace git-lfs: HTTP: 401
06:31:07.045187 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43
06:32:40.650030 trace git-lfs: HTTP: 401
06:32:40.650180 trace git-lfs: HTTP: PUT https://bucket.aibtc.dev/aibtcdev-communications/4f52c763375188024d13fdc83dda785037ece75712771a85ab8ec46949189c43

Happy to collab/test this on larger files too if still interested?