search

AkashRajpurohit / git-sync

🔄 A simple tool to backup and sync your git repositories
https://github.com/AkashRajpurohit/git-sync/wiki
MIT License
154 stars 8 forks source link

Git-sync crashes when I try syncing with my Forgejo instance #42

Closed jasonbraganza closed 1 month ago

jasonbraganza commented 1 month ago

Describe the Bug

I’m new to this, so if there’s anything you folks need clarification on or additional data about, please let me know.

The status quo is as follows …

  1. I have configured Forgejo to be a pull mirror for a few repos from my git hub account
  2. These are private repos on my forgejo account
  3. I have created a token (various tokens all with varying amounts of read access. Is write access required? Right now the token in use, is a read only, admin level access token

Steps to reproduce

run git-sync with the configured file against a forgejo instance

my config is as follows

# Git Sync Configuration for code.janusworx.com (Forgejo)

# Repository settings
include_forks: true # Include forked repositories
include_wiki: false # Include wiki's temporary. check in future versions
include_repos: [] # Include specific repositories
exclude_repos: [] # Exclude specific repositories
include_orgs: [] # Include repositories from specific organizations
exclude_orgs: [] # Exclude repositories from specific organizations

# Forgejo (code.janusworx.com) settings
username: my-user-name
token: some-admin-token
backup_dir: /code-bkp-folder
platform: forgejo
server:   
  domain: code.janusworx.com
  protocol: https

I’ve pasted the output of few runs below.

Relevant log output

{"level":"fatal","time":"2024-09-30T18:19:00.297+0530","message":"Error updating repo jasonbraganza/mjb-internet-weblog-mirror: exit status 128\n","stacktrace":"github.com/AkashRajpurohit/git-sync/pkg/logger.Fatalf\n\t/home/runner/work/git-sync/git-sync/pkg/logger/logger.go:96\ngithub.com/AkashRajpurohit/git-sync/pkg/sync.CloneOrUpdateRepo\n\t/home/runner/work/git-sync/git-sync/pkg/sync/sync.go:39\ngithub.com/AkashRajpurohit/git-sync/pkg/forgejo.ForgejoClient.Sync.func1\n\t/home/runner/work/git-sync/git-sync/pkg/forgejo/forgejo.go:52"}

---

{"level":"fatal","time":"2024-09-30T18:18:44.737+0530","message":"Error updating repo jasonbraganza/mjb-frappe-books-mirror: exit status 128\n","stacktrace":"github.com/AkashRajpurohit/git-sync/pkg/logger.Fatalf\n\t/home/runner/work/git-sync/git-sync/pkg/logger/logger.go:96\ngithub.com/AkashRajpurohit/git-sync/pkg/sync.CloneOrUpdateRepo\n\t/home/runner/work/git-sync/git-sync/pkg/sync/sync.go:39\ngithub.com/AkashRajpurohit/git-sync/pkg/forgejo.ForgejoClient.Sync.func1\n\t/home/runner/work/git-sync/git-sync/pkg/forgejo/forgejo.go:52"}

---

{"level":"fatal","time":"2024-09-30T18:20:25.716+0530","message":"Error updating repo jasonbraganza/mjb-hugo-PaperMod-mirror: exit status 128\n","stacktrace":"github.com/AkashRajpurohit/git-sync/pkg/logger.Fatalf\n\t/home/runner/work/git-sync/git-sync/pkg/logger/logger.go:96\ngithub.com/AkashRajpurohit/git-sync/pkg/sync.CloneOrUpdateRepo\n\t/home/runner/work/git-sync/git-sync/pkg/sync/sync.go:39\ngithub.com/AkashRajpurohit/git-sync/pkg/forgejo.ForgejoClient.Sync.func1\n\t/home/runner/work/git-sync/git-sync/pkg/forgejo/forgejo.go:52"}

---

{"level":"fatal","time":"2024-09-30T18:23:35.341+0530","message":"Error updating repo jasonbraganza/mjb-Awesome-CV-mirror: exit status 128\n","stacktrace":"github.com/AkashRajpurohit/git-sync/pkg/logger.Fatalf\n\t/home/runner/work/git-sync/git-sync/pkg/logger/logger.go:96\ngithub.com/AkashRajpurohit/git-sync/pkg/sync.CloneOrUpdateRepo\n\t/home/runner/work/git-sync/git-sync/pkg/sync/sync.go:39\ngithub.com/AkashRajpurohit/git-sync/pkg/forgejo.ForgejoClient.Sync.func1\n\t/home/runner/work/git-sync/git-sync/pkg/forgejo/forgejo.go:52"}

Which version of the software are you using?

v0.10.0

Which operating system are you using?

Linux Mint 22 Wilma (Based of Ubuntu 24.04)

runtime

Binary

AkashRajpurohit commented 1 month ago

Hey, can you run it with --log-level=debug and share the output.

jasonbraganza commented 1 month ago

Here’s a single complete run. If you want more, please do tell

$ git-sync --config ./code-janusworx.yaml --log-level=debug

{"level":"info","time":"2024-09-30T19:23:25.693+0530","message":"Logger initialized"}
{"level":"debug","time":"2024-09-30T19:23:25.693+0530","message":"Using config file: ./code-janusworx.yaml"}
{"level":"debug","time":"2024-09-30T19:23:25.693+0530","message":"Using config file: ./code-janusworx.yaml"}
{"level":"info","time":"2024-09-30T19:23:25.693+0530","message":"Config loaded from: ./code-janusworx.yaml"}
{"level":"info","time":"2024-09-30T19:23:25.693+0530","message":"Validating config ⏳"}
{"level":"debug","time":"2024-09-30T19:23:25.693+0530","message":"Creating new Forgejo client ⏳"}
{"level":"debug","time":"2024-09-30T19:23:25.773+0530","message":"Forgejo client created ✅"}
{"level":"info","time":"2024-09-30T19:23:25.774+0530","message":"Valid config found ✅"}
{"level":"info","time":"2024-09-30T19:23:25.774+0530","message":"Using Platform: forgejo"}
{"level":"debug","time":"2024-09-30T19:23:25.774+0530","message":"Fetching list of repositories ⏳"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-Awesome-CV-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-Baikal-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-anki21-addons_minimize-to-tray-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-awesome-taleb-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-calibre-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-espanso-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-frappe-books-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-huginn-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-hugo-PaperMod-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-hugo-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-internet-weblog-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-miniflux-v2-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-p2pu-sicp-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-python-feedgen-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-sicp-pdf-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-simply-scheme-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-thelounge-deb-mirror"}
{"level":"debug","time":"2024-09-30T19:23:25.834+0530","message":"Repo included: mjb-thelounge-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.834+0530","message":"Total repositories: 18"}
{"level":"info","time":"2024-09-30T19:23:25.834+0530","message":"Updating repo: jasonbraganza/mjb-thelounge-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.834+0530","message":"Updating repo: jasonbraganza/mjb-Awesome-CV-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.834+0530","message":"Updating repo: jasonbraganza/mjb-awesome-taleb-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.835+0530","message":"Updating repo: jasonbraganza/mjb-anki21-addons_minimize-to-tray-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.838+0530","message":"Updating repo: jasonbraganza/mjb-calibre-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.839+0530","message":"Updating repo: jasonbraganza/mjb-Baikal-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.838+0530","message":"Updating repo: jasonbraganza/mjb-frappe-books-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.834+0530","message":"Updating repo: jasonbraganza/mjb-espanso-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.850+0530","message":"Updating repo: jasonbraganza/mjb-miniflux-v2-mirror"}
{"level":"info","time":"2024-09-30T19:23:25.852+0530","message":"Updating repo: jasonbraganza/mjb-sicp-pdf-mirror"}
{"level":"debug","time":"2024-09-30T19:23:26.870+0530","message":"Output: remote: Verify\nfatal: Authentication failed for 'https://code.janusworx.com/jasonbraganza/mjb-awesome-taleb-mirror.git/'\n\n"}
{"level":"fatal","time":"2024-09-30T19:23:26.870+0530","message":"Error updating repo jasonbraganza/mjb-awesome-taleb-mirror: exit status 128\n","stacktrace":"github.com/AkashRajpurohit/git-sync/pkg/logger.Fatalf\n\t/home/runner/work/git-sync/git-sync/pkg/logger/logger.go:96\ngithub.com/AkashRajpurohit/git-sync/pkg/sync.CloneOrUpdateRepo\n\t/home/runner/work/git-sync/git-sync/pkg/sync/sync.go:39\ngithub.com/AkashRajpurohit/git-sync/pkg/forgejo.ForgejoClient.Sync.func1\n\t/home/runner/work/git-sync/git-sync/pkg/forgejo/forgejo.go:52"}
jbonadiman commented 1 month ago

From my tests, I think you only need a token with read access to repository and user permissions.

Did I understand right that those repos were previously cloned from another remote? If so, the remote URL is probably wrong for the fetch command that git-sync is trying to run. Could you confirm by sharing the output of git remote -v inside the repo folder? (and please, remember to remove any authentication information from the URLs before sharing)

jasonbraganza commented 1 month ago

@jbonadiman i pulled a Doh! and somehow managed to totally bork my Pi which was running Forgejo. Will it set it all up again and ping back here in a day or so.
Very sorry!

jasonbraganza commented 1 month ago

got the Pi up and running again. saw there was a new version, so moved to v0.11.0 and switched the clone_type to full and all works well now. I’ve done close to 22 repos which was more than what I had before.

Thank you for helping @AkashRajpurohit and @jbonadiman and for making such a lovely utility!

AkashRajpurohit commented 1 month ago

That's great to hear. However, just to confirm, does it still fail with bare repo setup? Since you already have forgejo setup ready, can you run git-sync (with different backup dir so it doesn't mess with your current backup directory) and with clone_type as bare?

jasonbraganza commented 1 month ago

Yes it does. 39 repos synced (of 39). Thank you @AkashRajpurohit

AkashRajpurohit commented 1 month ago

@all-contributors please add @jasonbraganza for bug and @jbonadiman for bug and question.

allcontributors[bot] commented 1 month ago

@AkashRajpurohit

I've put up a pull request to add @jasonbraganza! :tada:

AkashRajpurohit commented 1 month ago

@all-contributors please add @jasonbraganza for bug. please add @jbonadiman for bug and question.

allcontributors[bot] commented 1 month ago

@AkashRajpurohit

@jasonbraganza already contributed before to bug

AkashRajpurohit commented 1 month ago

@all-contributors please add @jasonbraganza for bug. please add @jbonadiman for bug and question.

Hopefully it does what I am expecting it to do this time 🤞🏽

allcontributors[bot] commented 1 month ago

@AkashRajpurohit

I've put up a pull request to add @jasonbraganza! :tada:

jbonadiman commented 1 month ago

From my tests, I think you only need a token with read access to repository and user permissions.

Did I understand right that those repos were previously cloned from another remote? If so, the remote URL is probably wrong for the fetch command that git-sync is trying to run. Could you confirm by sharing the output of git remote -v inside the repo folder? (and please, remember to remove any authentication information from the URLs before sharing)

Sorry guys, I didn't get it. The issue was really the remote being misconfigured? And if so, how the clone_type solves this? And why the shallow and bare clone types look the same?

If the issue was the remote-urls setup in the previously cloned/synced repo, why git-sync doesn't try to detect this difference and just add the remote it needs to successfully sync?

AkashRajpurohit commented 1 month ago

So here is what I think happened.

The issue was really the remote being misconfigured?

I believe yes. So when @jasonbraganza setup his Pi and forgejo again, the git-sync command started with fresh setup and he tried with clone_type as full which worked.

What I wanted to confirm was with fresh install does the bare clone fails for repository which are already mirror clones, which he confirmed works fine as well.

If the issue was the remote-urls setup in the previously cloned/synced repo, why git-sync doesn't try to detect this difference and just add the remote it needs to successfully sync?

I think this could be a good addition, we can do a check if the remote fails with authentication error, we can try re-setting the remote url as per the latest auth details present for the user from the config (consider the scenario where user have short lived access token which expired and they replace the token in the config file but their updates still fails).

We can open an issue for tracking this enhancement.

jbonadiman commented 1 month ago

So here is what I think happened.

The issue was really the remote being misconfigured?

I believe yes. So when @jasonbraganza setup his Pi and forgejo again, the git-sync command started with fresh setup and he tried with clone_type as full which worked.

What I wanted to confirm was with fresh install does the bare clone fails for repository which are already mirror clones, which he confirmed works fine as well.

If the issue was the remote-urls setup in the previously cloned/synced repo, why git-sync doesn't try to detect this difference and just add the remote it needs to successfully sync?

I think this could be a good addition, we can do a check if the remote fails with authentication error, we can try re-setting the remote url as per the latest auth details present for the user from the config (consider the scenario where user have short lived access token which expired and they replace the token in the config file but their updates still fails).

We can open an issue for tracking this enhancement.

Yeah, that's what I was thinking. Now, if someone have their token expired or if they decide to use a different client (even though the repo is exactly the same), this, which would be a simple remote update, would require git-sync to sync from scratch.

As you suggested, I'm gonna start a new issue so we can discuss how to handle it