search

martinmelin / githubmirror

A small script to keep a local bare copy of every repo in a Github organization, for faster cloning and deploys
40 stars 14 forks source link

init works great, but sync fails #1

Closed feoh closed 11 years ago

feoh commented 11 years ago

Here's what I'm seeing:

(Gazelle - Utility:util_gemserver_clone)(/apps/github-mirror)(0)# github-mirror  init secondrotation
Please give me a Github API token, create on https://github.com/settings/applications : [REDACTED by mm]
(Gazelle - Utility:util_gemserver_clone)(/apps/github-mirror)(0)#
(Gazelle - Utility:util_gemserver_clone)(/apps/github-mirror)(0)# github-mirror  sync secondrotation
Fetching git@github.com:secondrotation/pcp_middleware.git in /mnt/ebs/apps/github-mirror/pcp_middleware.git...
Traceback (most recent call last):
  File "/usr/local/bin/github-mirror", line 9, in <module>
    load_entry_point('githubmirror==0.2.4', 'console_scripts', 'github-mirror')()
  File "/usr/local/lib/python2.6/dist-packages/githubmirror/__init__.py", line 40, in cmd
    main.fetch(repos, workdir)
  File "/usr/local/lib/python2.6/dist-packages/githubmirror/main.py", line 90, in fetch
    remote.fetch(progress=FetchProgress())
  File "/usr/local/lib/python2.6/dist-packages/git/remote.py", line 593, in fetch
    return self._get_fetch_info_from_stderr(proc, progress or RemoteProgress())
  File "/usr/local/lib/python2.6/dist-packages/git/remote.py", line 535, in _get_fetch_info_from_stderr
    fp = open(join(self.repo.git_dir, 'FETCH_HEAD'),'r')
IOError: [Errno 2] No such file or directory: u'/mnt/ebs/apps/github-mirror/pcp_middleware.git/FETCH_HEAD'
(Gazelle - Utility:util_gemserver_clone)(/apps/github-mirror)(1)#

This is on an Ubuntu 10.04.1 LTS system running Python 2.6.5

martinmelin commented 11 years ago

Hi, thanks for the report! I currently catch a very broad range of Git errors when initing the repos and would like to improve this so github-mirror can complain about problems earlier. I think this might be a permissions problem when initing, but it just doesn't crash until trying to fetch.

Can you please give me the output of:

$ cd /mnt/ebs/apps/github-mirror/pcp_middleware.git/ && ls -al && cat config

Also, please go to https://github.com/settings/applications and delete the personal API token that was included in the output you pasted - it can be used as password to your account.

feoh commented 11 years ago

Done and done. Thanks so much for the quick response!

cpatti@ip-10-242-247-224:/apps/github-mirror$ cd /mnt/ebs/apps/github-mirror/pcp_middleware.git/ && ls -al && cat config
total 20
drwxr-xr-x  7 root root  111 2013-07-10 20:03 .
drwxr-xr-x 55 root root 4096 2013-07-10 20:03 ..
drwxr-xr-x  2 root root    6 2013-07-10 20:03 branches
-rw-r--r--  1 root root  185 2013-07-10 20:03 config
-rw-r--r--  1 root root   73 2013-07-10 20:03 description
-rw-r--r--  1 root root   23 2013-07-10 20:03 HEAD
drwxr-xr-x  2 root root 4096 2013-07-10 20:03 hooks
drwxr-xr-x  2 root root   20 2013-07-10 20:03 info
drwxr-xr-x  4 root root   28 2013-07-10 20:03 objects
drwxr-xr-x  4 root root   29 2013-07-10 20:03 refs
[core]
    repositoryformatversion = 0
    filemode = true
    bare = true
[remote "origin"]
    url = git@github.com:secondrotation/pcp_middleware.git
    fetch = +refs/heads/*:refs/remotes/origin/*
cpatti@ip-10-242-247-224:/mnt/ebs/apps/github-mirror/pcp_middleware.git$
feoh commented 11 years ago

One other thing that MIGHT be worthy of note - the native code compile section of the pip install failed, and pip said it was using the native Python versions. I have no idea if that's important here but thought it bore mentioning anyway.

martinmelin commented 11 years ago

Sorry for the late reply here, I haven't had time to look at this until now. I've published a new version to pypi, 0.3.5, that fixes a lot of problems by using raw git commands instead of the GitPython library's wrappers.

Can you please try installing that and let me know if you still get this issue? Thanks!

One thing to note is that I had to delete pip's build directory in /tmp/pip* to make it upgrade to 0.3.5. Another way to do it is:

$ pip install --upgrade --build=/tmp/not-pips-default-dir githubmirror==0.3.5
feoh commented 11 years ago

Hey there, thanks so much for the response! I got much farther this time, it syncs all our repositories except for a repo that we use to archive old projects:

root@ip-10-242-247-224:/mnt/ebs/apps/github-mirror# github-mirror  sync secondrotation
Fetching git@github.com:secondrotation/pcp_middleware.git in /mnt/ebs/apps/github-mirror/pcp_middleware.git...
Fetching git@github.com:secondrotation/sr_middleware.git in /mnt/ebs/apps/github-mirror/sr_middleware.git...
Fetching git@github.com:secondrotation/channel_advisor.git in /mnt/ebs/apps/github-mirror/channel_advisor.git...
Fetching git@github.com:secondrotation/reporting_area.git in /mnt/ebs/apps/github-mirror/reporting_area.git...
Fetching git@github.com:secondrotation/pcp.git in /mnt/ebs/apps/github-mirror/pcp.git...
Fetching git@github.com:secondrotation/rack_middleware.git in /mnt/ebs/apps/github-mirror/rack_middleware.git...
Fetching git@github.com:secondrotation/health_checks.git in /mnt/ebs/apps/github-mirror/health_checks.git...
Fetching git@github.com:secondrotation/cms.git in /mnt/ebs/apps/github-mirror/cms.git...
Fetching git@github.com:secondrotation/useragent.git in /mnt/ebs/apps/github-mirror/useragent.git...
Fetching git@github.com:secondrotation/pimp.git in /mnt/ebs/apps/github-mirror/pimp.git...
Fetching git@github.com:secondrotation/mc.git in /mnt/ebs/apps/github-mirror/mc.git...
Fetching git@github.com:secondrotation/srbase.git in /mnt/ebs/apps/github-mirror/srbase.git...
Fetching git@github.com:secondrotation/masterdev.git in /mnt/ebs/apps/github-mirror/masterdev.git...
Fetching git@github.com:secondrotation/hoptoad_notifier.git in /mnt/ebs/apps/github-mirror/hoptoad_notifier.git...
Fetching git@github.com:secondrotation/rack_page_caching.git in /mnt/ebs/apps/github-mirror/rack_page_caching.git...
Fetching git@github.com:secondrotation/delayed_job_extras.git in /mnt/ebs/apps/github-mirror/delayed_job_extras.git...
Fetching git@github.com:secondrotation/g3.git in /mnt/ebs/apps/github-mirror/g3.git...
Fetching git@github.com:secondrotation/warp_drive.git in /mnt/ebs/apps/github-mirror/warp_drive.git...
Fetching git@github.com:secondrotation/rack_url_stripper.git in /mnt/ebs/apps/github-mirror/rack_url_stripper.git...
Fetching git@github.com:secondrotation/cachetastic.git in /mnt/ebs/apps/github-mirror/cachetastic.git...
Fetching git@github.com:secondrotation/is_paranoid.git in /mnt/ebs/apps/github-mirror/is_paranoid.git...
Fetching git@github.com:secondrotation/giza.git in /mnt/ebs/apps/github-mirror/giza.git...
Fetching git@github.com:secondrotation/srapi.git in /mnt/ebs/apps/github-mirror/srapi.git...
Fetching git@github.com:secondrotation/srgems.git in /mnt/ebs/apps/github-mirror/srgems.git...
Fetching git@github.com:secondrotation/gazelle_ldap.git in /mnt/ebs/apps/github-mirror/gazelle_ldap.git...
Fetching git@github.com:secondrotation/ic.git in /mnt/ebs/apps/github-mirror/ic.git...
Fetching git@github.com:secondrotation/boi.git in /mnt/ebs/apps/github-mirror/boi.git...
Fetching git@github.com:secondrotation/cmo.git in /mnt/ebs/apps/github-mirror/cmo.git...
Fetching git@github.com:secondrotation/smx.git in /mnt/ebs/apps/github-mirror/smx.git...
Fetching git@github.com:secondrotation/active_shipping.git in /mnt/ebs/apps/github-mirror/active_shipping.git...
Fetching git@github.com:secondrotation/bootstrap.git in /mnt/ebs/apps/github-mirror/bootstrap.git...
Fetching git@github.com:secondrotation/erp.git in /mnt/ebs/apps/github-mirror/erp.git...
Fetching git@github.com:secondrotation/sales.git in /mnt/ebs/apps/github-mirror/sales.git...
Fetching git@github.com:secondrotation/activemessaging.git in /mnt/ebs/apps/github-mirror/activemessaging.git...
Fetching git@github.com:secondrotation/gzpuppet.git in /mnt/ebs/apps/github-mirror/gzpuppet.git...
Fetching git@github.com:secondrotation/archive.git in /mnt/ebs/apps/github-mirror/archive.git...
Traceback (most recent call last):
  File "/usr/local/bin/github-mirror", line 9, in <module>
    load_entry_point('githubmirror==0.3.5', 'console_scripts', 'github-mirror')()
  File "/usr/local/lib/python2.6/dist-packages/githubmirror/__init__.py", line 40, in cmd
    main.fetch(repos, workdir)
  File "/usr/local/lib/python2.6/dist-packages/githubmirror/main.py", line 88, in fetch
    gitdir.git.fetch(REMOTE_NAME)
  File "/usr/local/lib/python2.6/dist-packages/git/cmd.py", line 227, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/git/cmd.py", line 456, in _call_process
    return self.execute(call, **_kwargs)
  File "/usr/local/lib/python2.6/dist-packages/git/cmd.py", line 377, in execute
    raise GitCommandError(command, status, stderr_value)
git.exc.GitCommandError: 'git fetch origin' returned exit status 128: ERROR: Repository not found.
fatal: The remote end hung up unexpectedly

Here's the output of that debug command you asked for last time in case it's helpful:

root@ip-10-242-247-224:/mnt/ebs/apps/github-mirror# cd /mnt/ebs/apps/github-mirror/archive.git/ && ls -al && cat config
total 20
drwxr-xr-x  7 root root  128 2013-07-23 15:08 .
drwxr-xr-x 56 root root 4096 2013-07-23 15:06 ..
drwxr-xr-x  2 root root    6 2013-07-23 15:06 branches
-rw-r--r--  1 root root  172 2013-07-23 15:08 config
-rw-r--r--  1 root root   73 2013-07-23 15:06 description
-rw-r--r--  1 root root    0 2013-07-23 15:08 FETCH_HEAD
-rw-r--r--  1 root root   23 2013-07-23 15:06 HEAD
drwxr-xr-x  2 root root 4096 2013-07-23 15:06 hooks
drwxr-xr-x  2 root root   20 2013-07-23 15:06 info
drwxr-xr-x  4 root root   28 2013-07-23 15:06 objects
drwxr-xr-x  4 root root   29 2013-07-23 15:06 refs
[core]
    repositoryformatversion = 0
    filemode = true
    bare = true
[remote "origin"]
    url = git@github.com:secondrotation/archive.git
    fetch = +refs/*:refs/*
    mirror = true
root@ip-10-242-247-224:/mnt/ebs/apps/github-mirror/archive.git#
feoh commented 11 years ago

Note that as the dumping ground for where old projects go to die, we'd be more than happy to skip this if it were possible to do so.

I thought maybe removing the archive.git subdir would accomplish this, but it didn't work.

Any thoughts on how I might just skip it if a fix is problematic? Thanks for all your help.

martinmelin commented 11 years ago

I ran into something like this, too. The problem was that the user that was running github-mirror didn't have access to a repo, but the user that created the API token did. We have a machine user that clones from Github, but I had generated the API token on my personal account. Since I am an admin I could see more repos than the machine user could.

Are you able to clone the repo when logged in as the user that is running github-mirror? If not, try changing the token in the .githubmirror file to be one belonging to the same Github user.

feoh commented 11 years ago

Works perfectly, thank you sir!