search

cooperspencer / gickup

https://cooperspencer.github.io/gickup-documentation/
Apache License 2.0
955 stars 33 forks source link

`FTL repository does not exist` crashing gickup #191

Open thehale opened 8 months ago

thehale commented 8 months ago

I'm getting the same error message as described in https://github.com/cooperspencer/gickup/issues/112, but I'm running the latest version (0.10.24). Is this perhaps a regression?

I seem to get the error most often for larger repositories (measured in MB of storage, not number of commits). Note that these problematic repos do NOT use LFS.

gickup-runner-1  | 2023-12-27 00:33:56 INF cloning github.com/REDACTED/1703662436 path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-27 00:34:40 WRN retry 1 from 5 path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-27 00:34:45 INF opening github.com/REDACTED/1703662436 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-27 00:34:45 WRN retry 2 from 5 path=/backups/gickup repo=github.com/REDACTED/1703662436 stage=locally
gickup-runner-1  | 2023-12-27 00:34:50 INF opening github.com/REDACTED/1703662436 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-27 00:34:50 WRN retry 3 from 5 path=/REDACTED/1703662436 stage=locally
gickup-runner-1  | 2023-12-27 00:34:55 INF opening github.com/REDACTED/1703662436 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-27 00:34:55 WRN retry 4 from 5 path=/backups/gickup repo=github.com/REDACTED/1703662436 stage=locally
gickup-runner-1  | 2023-12-27 00:35:00 INF opening github.com/REDACTED/1703662436 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-27 00:35:00 FTL repository does not exist path=/backups/gickup repo=github.com/REDACTED/1703662436 stage=locally
gickup-runner-1 exited with code 1

Expected Behavior

Ideally, gickup would be able to clone these larger repos.

More pressingly, I would hope that gickup would continue on to backup other repositories even after one fails to clone. I don't want my daily backup to miss 80% of my repos just because the fifth one had a problem.

cooperspencer commented 8 months ago

Hi, I'll look into this. Can you tell me one of the larger repos you have troubles with or tell me the size of such a repository, so I'll try to recreate that.

thehale commented 8 months ago

So far, all the failing repositories have been private. A fresh git clone of the smallest one is 148 MB.

Interestingly, one of my public repositories worked just fine even though it's ~370 MB.

cooperspencer commented 8 months ago

I just tested it on a private repository, which is about 400 MB and it worked like a charm. I changed the Fatal to a Warning so that it doesn't quit the backup: https://github.com/cooperspencer/gickup/tree/issue_191

Can you share your configuration? I'd just like to see how you configured the destination part.

cooperspencer commented 8 months ago

I just noticed in your log that the clone happens only once. I adjusted the log output by adding errors to the warning level. Can you try to run it with the artifact from this build? https://github.com/cooperspencer/gickup/actions/runs/7337421497

Or just build the docker container from branch issue_191

thehale commented 8 months ago

Can you share your configuration? I'd just like to see how you configured the destination part.

Here's the destination block of my config:

destination:
  local:
    - path: /backups/gickup
      structured: true # checks repos out like hostersite/user|organization/repo
      zip: true # zips the repository after cloned and removes the repository afterwards
      keep: 5 # only keeps x backups
      bare: true # clone the repositories as bare
      lfs: false # clone lfs repos, !! ATTENTION !! git and git-lfs must be installed on the system!

I also experimented with bare: false, but that didn't seem to make a difference.

Or just build the docker container from branch issue_191

Those changes successfully allow Gickup to progress beyond the failing repositories.

Here's a snippet of the logs, containing the lines relevant to one of the still failing repositories:

gickup-runner-1  | 2023-12-29 23:49:58 INF starting backup for https://github.com/REDACTED stage=backup
gickup-runner-1  | 2023-12-29 23:49:58 INF cloning github.com/REDACTED/1703918998 path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-29 23:50:35 WRN retry 1 from 5 error="stream error: stream ID 71; CANCEL; received from peer" path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-29 23:50:40 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-29 23:50:40 WRN retry 2 from 5 error="repository does not exist" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1  | 2023-12-29 23:50:41 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1  | 2023-12-29 23:50:46 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-29 23:50:46 WRN retry 3 from 5 error="repository does not exist" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1  | 2023-12-29 23:50:46 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1  | 2023-12-29 23:50:51 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-29 23:50:51 WRN retry 4 from 5 error="repository does not exist" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1  | 2023-12-29 23:50:51 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1  | 2023-12-29 23:50:56 INF opening github.com/REDACTED/1703918998 locally path=/backups/gickup stage=locally
gickup-runner-1  | 2023-12-29 23:50:56 WRN repository does not exist path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
gickup-runner-1  | 2023-12-29 23:50:56 WRN couldn't remove /backups/gickup/github.com/REDACTED/1703918998 error="unlinkat github.com/REDACTED/1703918998/objects/pack: directory not empty" path=/backups/gickup repo=github.com/REDACTED/1703918998 stage=locally
cooperspencer commented 8 months ago

stream error: stream ID 71; CANCEL; received from peer seems like there is a connection issue.

cooperspencer commented 8 months ago

I added a force remove after the initial clone if anything went wrong during the clone. Can you please try to build it again?

thehale commented 7 months ago

I tested out your changes over a week ago and forgot to post the update. Sorry!

The latest changes successfully allow the backup to complete even if one or more individual repos fail, resolving the original ask of this ticket.

While some of the remaining repos are regular failures, most of them aren't consistently failing (i.e. they sporadically succeed). Once I've identified a more reproducible example of a consistent failure, I'll open a new ticket. Thanks!