Not able to mirror gitlab to github

[defyjoy]

I have tried whatever combination

• Given all permissions to token
• Used username + token
• Used token file /gickup/gitlab-token.txt

I keep getting error as below . I am not sure what am I doing wrong here

Here is my configuration

source:
  gitlab:
    # - token_file: /gickup/gitlab-token.txt
    # - token_file: gitlab-token.txt
    - ssh: false # can be true or false
      # username: myemail@gmail.com
      token: glpat-abczyx
      includeorgs: 
        - myapp
      include: # this includes the repo "foobar"
        - myapp
destination:
  github:    
    - token_file: /gickup/github-token.txt # alternatively, specify token in a file
    # - token_file: github-token.txt # alternatively, specify token in a file
      organization: defyjoy   
      force: true
      visibility:
        repositories: private # private, public, default: private

log: # optional
  timeformat: 2006-01-02 15:04:05 
  file-logging: # optional
    dir: log # directory to log into
    file: gickup.log # file to log into
    maxage: 7 # keep logs for 7 days

Here are the gitlab token permissions -

Here are the error logs -

myapp-gickup-runner-1  | 2024-01-24T09:11:59+05:30 INF Reading /gickup/conf.yml file=/gickup/conf.yml
myapp-gickup-runner-1  | 2024-01-24 09:11:59 INF Configuration loaded destinations=1 pairs=1 sources=1
myapp-gickup-runner-1  | 2024-01-24 09:11:59 INF Backup run starting
myapp-gickup-runner-1  | 2024-01-24 09:11:59 INF grabbing repositories from  stage=gitlab url=
myapp-gickup-runner-1  | 2024-01-24 09:12:05 INF starting backup for https://gitlab.com/myapp/myapp.git stage=backup
myapp-gickup-runner-1  | 2024-01-24 09:12:05 INF mirroring myapp to https://github.com stage=github url=https://github.com
myapp-gickup-runner-1  | 2024-01-24 09:12:48 ERR authentication required stage=tempclone url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-24 09:12:52 INF Backup run complete duration=53.866037817s

[cooperspencer]

Hi, I'll have a look and get back to you on that.

[cooperspencer]

I had a look and ran into the same issue as you did. Solution for me was to add the user parameter to my gitlab config. For the next release I'll try to find the tokens user, so that you don't need to add that in the future anymore.

source:
  gitlab:
    # - token_file: /gickup/gitlab-token.txt
    # - token_file: gitlab-token.txt
    - ssh: false # can be true or false
      user: your-username
      # username: myemail@gmail.com
      token: glpat-abczyx
      includeorgs: 
        - myapp
      include: # this includes the repo "foobar"
        - myapp

[defyjoy]

OK so I tried to fill in user with both the username in gitlab eg: defyjoy1212343 or my email . Both didn't work . What is the user we should be putting there ?

[cooperspencer]

It should be your gitlab username. I just committed code to the main branch to automatically get your username, via the token, if not specifically mentioned in the config file. You can try to run that via the docker containers buddyspencer/gickup:dev or buddyspencer/gickup:ubuntu-dev

[defyjoy]

So this time it got weirder . It did receive the username from buddyspencer/gickup:dev but look at that duration. It took almost 19 minutes and was doing nothing -

╰─(base) ○ docker compose up
[+] Building 0.0s (0/0)                                                                                                                  docker:desktop-linux
[+] Running 1/0
 ✔ Container myapp-gickup-runner-1  Created                                                                                                           0.0s
Attaching to myapp-gickup-runner-1
myapp-gickup-runner-1  | 2024-01-25T16:02:37+05:30 INF Reading /gickup/conf.yml file=/gickup/conf.yml
myapp-gickup-runner-1  | 2024-01-25 16:02:37 INF Configuration loaded destinations=1 pairs=1 sources=1
myapp-gickup-runner-1  | 2024-01-25 16:02:37 INF Backup run starting
myapp-gickup-runner-1  | 2024-01-25 16:02:38 INF grabbing repositories from myusername1234 stage=gitlab url=https://gitlab.com
myapp-gickup-runner-1  | 2024-01-25 16:02:45 INF starting backup for https://gitlab.com/myapp/myapp.git stage=backup
myapp-gickup-runner-1  | 2024-01-25 16:02:45 INF mirroring myapp to https://github.com stage=github url=https://github.com

myapp-gickup-runner-1  | 2024-01-25 16:21:37 ERR authentication required stage=tempclone url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-25 16:21:41 INF Backup run complete duration=19m3.891281938s
myapp-gickup-runner-1 exited with code 0

The conf.yaml . I have removed the user since it's fetched automatically -

source:
  gitlab:
    - ssh: false # can be true or false
      token: "glpat-abcxyz"
      includeorgs: 
        - myapp
      include: # this includes the repo "foobar"
        - myapp
destination:
  github:    
    - token_file: /gickup/github-token.txt 
      organization: myapp   
      force: true
      visibility:
        repositories: private 
log:
  timeformat: 2006-01-02 15:04:05 
  file-logging: # optional
    dir: log # directory to log into
    file: gickup.log # file to log into
    maxage: 7 

If a gitlab has a project , do we include the same in includeorgs ?

I am not very sure where am I wrong .

[cooperspencer]

the step "mirrorin myapp to https://github.com" contains the local clone and then the push to Github. Weird that it takes so long, is it a very large repo?

The authentication required seems to be coming from github directly. I will do more investigation on that, but I can reproduce the problem.

[defyjoy]

If the authentication issue is coming from github ( destination ) I can create a new PAT token and see if it works . I was confused if the authentication failed on gitlab (source) side.

[defyjoy]

Ok so I tried with a new token on github and gitlab both tokens does not work for auth still . Yes the repo is on larger side but this time it took around 1minute around . So probably my intermittent internet issue .

I suspect the issue is more likely towards gitlab

[cooperspencer]

I found the error... I didn't add the authentication during the git fetch. You can retry it with the dev containers

[defyjoy]

Great work !! However one last issue. I am able to work through other smaller repos but this one a pretty heavy one and receiving the message -

myapp-gickup-runner-1  | 2024-01-26 07:23:09 INF mirroring myapp to https://github.com stage=github url=https://github.com
myapp-gickup-runner-1  | 2024-01-26 07:24:25 ERR some refs were not updated stage=tempclone url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-26 07:24:37 INF Backup run complete duration=1m32.857118001s

Anyway to get around this ? When I clone it on my local it doesn't throw tantrums of course .

Here are the logs when I clone it on my local

remote: Enumerating objects: 510981, done.
remote: Counting objects: 100% (391/391), done.
remote: Compressing objects: 100% (384/384), done.
remote: Total 510981 (delta 228), reused 12 (delta 0), pack-reused 510590
Receiving objects: 100% (510981/510981), 369.51 MiB | 35.60 MiB/s, done.
Resolving deltas: 100% (388574/388574), done.

The downloaded size is arounf 450 MB on disk

Looking at docker memory stats I didn't find any spike which might also cause this to fail

[cooperspencer]

That is a great use case because I don't have such big repositories, so thanks a lot for testing it so extensively. I created a new container for this issue: buddyspencer/gickup:issue_200

After some digging, I found that someone had a similar issue with go-git and there solution was to add Force: true to git fetch. I added that to this branch. Can you please use buddyspencer/gickup:issue_200 and test if it works?

[defyjoy]

Went quite far but not quite . I received this error now -

myapp-gickup-runner-1 ERR command error on refs/heads/release-myapp-one: pre-receive hook declined stage=github url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-26 12:42:50 INF Backup run complete duration=9m34.106607763s

[cooperspencer]

damn, this is a hard one. Is gickup able to backup this repo locally? So instead of using github as destination can you try to use local and tell me if that works?

[defyjoy]

Thanks for responding so quick . So yes there is absolutely no issue in backing up locally .

myapp-gickup-runner-1  | 2024-01-26T13:20:41+05:30 INF Reading /gickup/conf.yml file=/gickup/conf.yml
myapp-gickup-runner-1  | 2024-01-26 13:20:41 INF Configuration loaded destinations=1 pairs=1 sources=1
myapp-gickup-runner-1  | 2024-01-26 13:20:41 INF Backup run starting
myapp-gickup-runner-1  | 2024-01-26 13:20:42 INF grabbing repositories from joydeep7840104 stage=gitlab url=https://gitlab.com
myapp-gickup-runner-1  | 2024-01-26 13:20:46 INF starting backup for https://gitlab.com/myapp/myapp.git stage=backup
myapp-gickup-runner-1  | 2024-01-26 13:20:46 INF cloning gitlab.com/myapp/myapp.git/1706255446 path=/gickup/backup stage=locally
myapp-gickup-runner-1  | 2024-01-26 13:21:48 INF zipping gitlab.com/myapp/myapp.git/1706255446 path=/gickup/backup stage=locally
myapp-gickup-runner-1  | 2024-01-26 13:21:53 INF Backup run complete duration=1m12.177087158s

The destination config

destination:
  local:
    - path: /gickup/backup
      structured: true
      zip: true
      keep: 5
      bare: true

Docker compose mounted volume

    volumes:
      - ${PWD}/conf.yml:/gickup/conf.yml # Change the path of your local config ${PWD} is your current directory (where the docker-compose.yml is located)
      - ${PWD}/gitlab-token.txt:/gickup/gitlab-token.txt
      - ${PWD}/github-token.txt:/gickup/github-token.txt
      - ${PWD}/tmp:/tmp/
      - ${PWD}/backup:/gickup/backup

And the zipped file got created

[cooperspencer]

I pushed a new container on buddyspencer/gickup:issue_200. It won't fix the issue but hopefully tells me where it comes from. Can you please post the error message afterwards?

[defyjoy]

Ok so I removed the stale docker container and rerun to pull the new image however the issue remains the same and no information provided too .

myapp-gickup-runner-1  | 2024-01-26 14:30:29 ERR command error on refs/heads/release-myapp-one: pre-receive hook declined git=push stage=github url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-26 14:30:42 INF Backup run complete duration=9m43.378001265s

[cooperspencer]

It comes from the push. Can you check if you have any pre-receive hooks on your repo?

[defyjoy]

It comes from the push. Can you check if you have any pre-receive hooks on your repo?

Which repo ? Github or Gitlab ?

On github side the repo was created by gickup so no webhooks are there

However on gitlab side there is a sentry webhook . I believe that should not cause any issue as that just listens to gitlab events

[cooperspencer]

doesn't matter actually. does your repo have a .git/hooks directory?

[defyjoy]

You're right . The repo folder indeed does have .git/hooks directory

[cooperspencer]

can you check if any of the hooks in there give you issues regarding pushing the code? also try to remove the force parameter from the github push.

[defyjoy]

I tried creating one branch and pushing it . It doesn't seem to fail . If thats what you wanted to ask

[cooperspencer]

destination:
  github:    
    - token_file: /gickup/github-token.txt # alternatively, specify token in a file
    # - token_file: github-token.txt # alternatively, specify token in a file
      organization: defyjoy   
      force: false
      visibility:
        repositories: private # private, public, default: private

the other question was if there is actually anything in the hooks directory

[defyjoy]

Yes there are some sample stuff but not sure if that is causing the issue

[cooperspencer]

the pre-receive is most likey the culprit

[cooperspencer]

damn, no all them are just samples, nothing is active...

[defyjoy]

Okay so another thing I did is to remove the force parameter from github destination but yet the issue remains the same .

[defyjoy]

Few good news I found from another tool .

remote: warning: File release-admin/.meteor/local/db/journal/WiredTigerPreplog.0000000001 is 100.00 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File src/private/export-2019-07-25T09_00_07.167Z.json is 92.02 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File release-admin/.meteor/local/db/journal/WiredTigerLog.0000000001 is 100.00 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File idrodata/dji-logs/dji_go_4.apk is 97.20 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: Trace: 115082b1f60349ef7f2be39747de59735528ee23f89d6e9d0f491dc0610115d0
remote: error: See https://gh.io/lfs for more information.
remote: error: File src/imports/situation/server/tests/data/data-zones.json is 130.65 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com/.

So there are some huge files which is why it was rejecting for push . But gickup doesn't show this issue.

[cooperspencer]

Which tool were you using? Maybe I can check what they do differently than me.

[defyjoy]

I used this - https://github.com/Enteee/git-sync-mirror

[cooperspencer]

seems like go-git doesn't give me the full error, but thanks a lot for testing it so much. I will add git lfs to the sync option, then it should be possible to mirror your big repository to github.

[cooperspencer]

Quick question, is the repository on Gitlab already using LFS?

[defyjoy]

I think so . But I am not sure . But I do know that Gitlab is actually storing those huge files . But can't identify which branch right now

[defyjoy]

Hi any updates on this. ?

[cooperspencer]

Not yet. I'll try to do it tomorrow.

[cooperspencer]

I started working on it. Already had some success, but for now it only pushes your default branch. Also I am not fully happy with the error messages for it. You can track the progress right here: https://github.com/cooperspencer/gickup/pull/204

[cooperspencer]

can you try to run it on buddyspencer/gickup:issue_200?

It worked in my tests. I had an LFS repository on Gitlab. Had two branches and mirrored the whole repo to GitHub.

[defyjoy]

Hi Sorry for the delay . I will test this and report tomorrow on status for this .

[cooperspencer]

Did you already test it?

[defyjoy]

I will keep you posted tomorrow on this

[petarov]

Hi, I guess I have a similar problem but the other way around - GitHub to GitLab. gickup 0.10.36

INF mirroring repo-name to https://gitlab.com stage=gitlab url=https://gitlab.com ERR command error on refs/heads/master: pre-receive hook declined git=push stage=gitlab url=https://github.com/petarov/repo-name.git

I've noticed that this only happens when syncing LFS-enabled repositories.

[cooperspencer]

I will run some tests on this and get back to you

[cooperspencer]

Mind if I ask how large the repository, you try to mirror, is?

[petarov]

Sure, so I got maybe 10 lfs repos of different sizes, but I would say the smallest is around 1.5 MiB and the biggest around 38 MiB. Thanks for replying.