Open thehale opened 8 months ago
Hi, I'll look into this. Can you tell me one of the larger repos you have troubles with or tell me the size of such a repository, so I'll try to recreate that.
So far, all the failing repositories have been private. A fresh git clone
of the smallest one is 148 MB.
Interestingly, one of my public repositories worked just fine even though it's ~370 MB.
I just tested it on a private repository, which is about 400 MB and it worked like a charm.
I changed the Fatal
to a Warning
so that it doesn't quit the backup:
https://github.com/cooperspencer/gickup/tree/issue_191
Can you share your configuration? I'd just like to see how you configured the destination part.
I just noticed in your log that the clone happens only once. I adjusted the log output by adding errors to the warning level. Can you try to run it with the artifact from this build? https://github.com/cooperspencer/gickup/actions/runs/7337421497
Or just build the docker container from branch issue_191
Can you share your configuration? I'd just like to see how you configured the destination part.
Here's the destination block of my config:
destination:
local:
- path: /backups/gickup
structured: true # checks repos out like hostersite/user|organization/repo
zip: true # zips the repository after cloned and removes the repository afterwards
keep: 5 # only keeps x backups
bare: true # clone the repositories as bare
lfs: false # clone lfs repos, !! ATTENTION !! git and git-lfs must be installed on the system!
I also experimented with bare: false
, but that didn't seem to make a difference.
Or just build the docker container from branch issue_191
Those changes successfully allow Gickup to progress beyond the failing repositories.
Here's a snippet of the logs, containing the lines relevant to one of the still failing repositories:
gickup-runner-1 | 2023-12-29 23:49:58 INF starting backup for https://github.com/REDACTED stage=backup
gickup-runner-1 | 2023-12-29 23:49:58 INF cloning github.com/REDACTED/1703918998 path=/backups/gickup stage=locally
gickup-runner-1 | 2023-12-29 23:50:35 WRN retry 1 from 5 error="stream error: stream ID 71; CANCEL; received from peer" path=/backups/gickup stage=locally
gickup-runner-1 | 2023-12-29 23:50:40 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1 | 2023-12-29 23:50:40 WRN retry 2 from 5 error="repository does not exist" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1 | 2023-12-29 23:50:41 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1 | 2023-12-29 23:50:46 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1 | 2023-12-29 23:50:46 WRN retry 3 from 5 error="repository does not exist" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1 | 2023-12-29 23:50:46 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1 | 2023-12-29 23:50:51 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1 | 2023-12-29 23:50:51 WRN retry 4 from 5 error="repository does not exist" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1 | 2023-12-29 23:50:51 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1 | 2023-12-29 23:50:56 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1 | 2023-12-29 23:50:56 WRN repository does not exist path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1 | 2023-12-29 23:50:56 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
stream error: stream ID 71; CANCEL; received from peer
seems like there is a connection issue.
I added a force remove after the initial clone if anything went wrong during the clone. Can you please try to build it again?
I tested out your changes over a week ago and forgot to post the update. Sorry!
The latest changes successfully allow the backup to complete even if one or more individual repos fail, resolving the original ask of this ticket.
While some of the remaining repos are regular failures, most of them aren't consistently failing (i.e. they sporadically succeed). Once I've identified a more reproducible example of a consistent failure, I'll open a new ticket. Thanks!
I'm getting the same error message as described in https://github.com/cooperspencer/gickup/issues/112, but I'm running the latest version (0.10.24). Is this perhaps a regression?
I seem to get the error most often for larger repositories (measured in MB of storage, not number of commits). Note that these problematic repos do NOT use LFS.
Expected Behavior
Ideally, gickup would be able to clone these larger repos.
More pressingly, I would hope that gickup would continue on to backup other repositories even after one fails to clone. I don't want my daily backup to miss 80% of my repos just because the fifth one had a problem.