search

samrocketman / gitlab-mirrors

A set of scripts adding the ability of managing remote mirrors to GitLab.
MIT License
818 stars 169 forks source link

newbie q. - what where? #37

Closed TinajaLabs closed 10 years ago

TinajaLabs commented 10 years ago

Sorry to be dense, but it is not clear to me where I should be running the installs.

I have a central gitlab installed; have been using it for a couple of months, and I set up a system that I assumed will be my read-only mirror system.

The install script does not make it clear (to me), where I am to configure the pre-requisites or where I set up the users, groups, etc.

I was starting to do this on the remote mirror, but then I thought that all the action should happen at the server; where the cron job should be running to update the remote mirrors.

As a newbie, I'm not seeing the big picture.

Any tips would be appreciated.

Thanks, Chris.

samrocketman commented 10 years ago

So as long as the gitmirror GitLab user has the SSH keys from the local gitmirror system user, the scripts I wrote can operate on a remote host not on the GitLab server or on the same host as the GitLab server. (note: gitmirror is just an arbitrary name I chose. Realistically, they are configurable and can be any name (the scripts support that).

TinajaLabs commented 10 years ago

Thanks.

So far I did the pre-requisites on the remote mirror host:

Then:

I manually ran the git-mirrors.sh script and got an error:

# /home/gitmirror/gitlab-mirrors/git-mirrors.sh
ls: cannot access /home/gitmirror/repositories/Mirrors: No such file or directory

So I added the Mirrors directory.

Now no errors but also no action. What's next? Should I be adding the gitmirror/Mirrors setup to my main gitlab server ... Still, I'm not seeing how this all ties together. Apologies for being so dense.

Thanks in advance, Chris.

samrocketman commented 10 years ago

So you need to use add_mirror.sh script to add git repositories to mirror. Since you didn't install any of the helpers I'd assume you only want to mirror git repositories. So your command to add a mirror will almost always look like...

./add_mirror.sh --git --project-name someproject --mirror http://somehost/somerepo.git

That will create a project on your GitLab instance and set up gitlab-mirrors to sync that remote somerepo.git to it. Once you have one or more mirrors then you can start using the other management scripts. Perhaps I should add a check to git-mirrors.sh explaining to use add_mirror.sh if it is apparent there are no mirrors yet.

samrocketman commented 10 years ago

Since I've received no response and this isn't a bug in the software I'm going to go ahead and close this.

TinajaLabs commented 10 years ago

OK, so I'm back and thought I was close. I know this is not a bug, but definitely a lack of clear understanding on my part about what goes where. Maybe it will eventually help someone else.

For clarification, I think of a mirror site as a remote twin of a central site. I use secondary remote mirror sites to retrieve copies of files if the main repo is too busy. In your documentation you seem to be referring to the Gitlab site as the mirror. From my perspective, in my setup, Gitlab is my main site and gitlab (small g, remote) is my mirror site

To start: git.corp.xyz is our central active Gitlab server go.soc.xyz is our remote gitlab mirror site

When I run this from my remote mirror site:

su - gitmirror
cd gitlab-mirrors
./add_mirror.sh --git --project-name somenewproj --mirror http://git.corp.xyz/chris.j/icnga-server.git

I get:

Resolving gitlab remote.
Creating mirror from http://git.corp.xyz/chris.j/icinga-server.git
Initialized empty Git repository in /home/gitmirror/repositories/Mirrors/somenewproj/
error: The requested URL returned error: 401 Unauthorized while accessing http://git.corp.xyz/chris.j/icinga-server.git/info/refs

fatal: HTTP request failed

It creates the directory /home/gitmirror/repositories/Mirrors on my remote gitlab site, but not the somenewproj directory.

It does create a project in our central active Gitlab server called: Mirrors / somenewproj described as Mirror of http://git.corp.xyz/chris.j/icinga-server.git

It is an empty project/repo; no files.

I would think it would create that in my mirror server.

On my remote gitlab mirror site, I have defined the gitlab_url in config.sh as my central active Gitlab server: gitlab_url="http://git.corp.xyz". Is that the right way to do this?

Thanks for your patience, Chris.

config.sh from our remote gitlab mirror site:

Environment file

#
# gitlab-mirrors settings
#

#The user git-mirrors will run as.
system_user="gitmirror"
#The home directory path of the $system_user
user_home="/home/${system_user}"
#The repository directory where gitlab-mirrors will contain copies of mirrored repositories before pushing them to gitlab.
repo_dir="${user_home}/repositories"
#colorize output of add_mirror.sh, update_mirror.sh, and git-mirrors.sh commands.
enable_colors=true
#These are additional options which should be passed to git-svn.  See git help svn
git_svn_additional_options="-s"
#Force gitlab-mirrors to not create the gitlab remote
no_create_set=false

#
# Gitlab settings
#

#This is the Gitlab group where all project mirrors will be grouped.
gitlab_namespace="Mirrors"
#This is the base web url of your Gitlab server.
gitlab_url="http://git.corp.xyz"
#Special user you created in Gitlab whose only purpose is to update mirror sites and admin the $gitlab_namespace group.
gitlab_user="gitmirror"
#Generate a token for your $gitlab_user and set it here.
gitlab_user_token_secret="$(head -n1 "${user_home}/private_token")"
#Verify signed SSL certificates?
ssl_verify=false

#
# Gitlab new project default settings.  If a project needs to be created by gitlab-mirrors
# then it will assign the following values as defaults.
#

#values must be true or false
issues_enabled=false
wall_enabled=false
wiki_enabled=false
snippets_enabled=false
merge_requests_enabled=false
public=false
samrocketman commented 10 years ago

So this might help. Think of the --mirror option as the mirror you want to create a mirror from. You're not pushing to it.

The destination should be set in config.sh that is to say gitlab_url="http://go.soc.xyz". The source of the mirror is passed as the argument to --mirror in other words: --mirror http://git.corp.xyz/chris.j/icnga-server.git.

If you take a look at the post you just gave me you have the source and the destination set to be the same. While it is possible to do that there's no point in doing it.

TinajaLabs commented 10 years ago

Updated the config.sh file to contain the remote gitlab mirror target site. That helped. Thanks.

Now the project and repo are created in my remote gitlab mirror site, go.soc.xyz. Sadly they are empty and I get the same error:

Resolving gitlab remote.
Creating mirror from http://git.corp.xyz/chris.j/icinga-server.git
Initialized empty Git repository in /home/gitmirror/repositories/Mirrors/somenewproj/
error: The requested URL returned error: 401 Unauthorized while accessing http://git.corp.xyz/chris.j/icinga-server.git/info/refs

fatal: HTTP request failed

Under what context is my central source Gitlab repo (http://git.corp.xyz/chris.j/icinga-server.git/info/refs) being accessed that it would return a 401 Unauthorized error?

It says that it is creating the directory: /home/gitmirror/repositories/Mirrors/somenewproj/ But it does not.

The private_token file in the root of /home/gitmirror contains the token from the accounts page of same remote gitlab mirror target site, go.soc.xyz.

The public ssh key of the remote gitlab mirror target sitego.soc.xyz is defined in gitmirror account 'My SSH keys' page of the central source Gitlab.

I'm running the command as gitmirror and all files in /home/gitmirror belong to gitmirror. 

su - gitmirror
cd gitlab-mirrors

The .ssh/config file on remote gitlab mirror target site go.soc.xyz has:

Host go.soc.xyz
    User git

I couldn't clone gitlab-mirrors because go.soc.xyz can't reach the internet, so I copied the zip and placed copied it manually. I can ping/access my central source Gitlab server.

I've just been trying different combinations, and feeling frustrated. As in most cases, this too will be resolved. It's just a matter of finding out where I went wrong.

Thanks...

TinajaLabs commented 10 years ago

I tried to use the ssh form of the repo reference like this:

./add_mirror.sh --git --project-name icnga-server --mirror git@git.corp.xyz:chris.j/icinga-server.git

Then I got this:

Resolving gitlab remote.
Creating mirror from git@git.corp.xyz:chris.j/icinga-server.git
Initialized empty Git repository in /home/gitmirror/repositories/Mirrors/icnga-server/
remote: Counting objects: 71, done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 71 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (71/71), 78.97 KiB, done.
Resolving deltas: 100% (26/26), done.
Adding gitlab remote to project.
Checking the mirror into gitlab.
ssh: Could not resolve hostname gitlab.example.com: Name or service not known
fatal: The remote end hung up unexpectedly

Looked like it was going to work but choked when it saw the hostname gitlab.example.com which of course does not exist.

As mentioned above, I have in fact set the gitlab_url="http://go.soc.xyz" in the config.sh file.

I have also check the file /etc/gitlab/gitlab.rb and it appears correct:

external_url 'http://go.soc.xyz'

I did a search to find this reference in my gitlab-mirrors dir tree and found the reference in some docs, and in:

repositories/Mirrors/icnga-server/config:   url = git@gitlab.example.com:mirrors/icnga-server.git

Where might it be pulling the gitlab.example.com reference?

TinajaLabs commented 10 years ago

That might have been a gitlab problem:

Seems there was still a reference to gitlab.example.com in:

embedded/service/gitlab-rails/config/gitlab.yml:    host: gitlab.example.com

The yml is auto generated as part of rails config so I ran: sudo gitlab-ctl reconfigure to reset the yml and that seemed to fix that.

TinajaLabs commented 10 years ago

Next problem:

Running like this (as gitmirror):

gitmirror@go:~/gitlab-mirrors$ ./add_mirror.sh --git --project-name icnga-server --mirror git@git.corp.xyz:chris.j/icinga-server.git

Ends up with:

Resolving gitlab remote.
Creating mirror from git@git.corp.xyz:chris.j/icinga-server.git
Initialized empty Git repository in /home/gitmirror/repositories/Mirrors/icnga-server/
remote: Counting objects: 71, done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 71 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (71/71), 78.97 KiB, done.
Resolving deltas: 100% (26/26), done.
Adding gitlab remote to project.
Checking the mirror into gitlab.
git@go.soc.xyz's password:
fatal: 'mirrors/icnga-server.git' does not appear to be a git repository
fatal: The remote end hung up unexpectedly
TinajaLabs commented 10 years ago

Tracing add_mirror.sh, I see that at the point it asks for the git@go.soc.xyz's password, it is performing a git push gitlab in the if ${git} section. That's where it throws the error, fatal, about it not being a git repo.

I installed gitlab from Omnibus version like this:

wget https://downloads-packages.s3.amazonaws.com/centos-6.5/gitlab-7.1.0_omnibus-1.el6.x86_64.rpm
sudo yum install openssh-server
sudo yum install postfix  # Select 'Internet Site', using sendmail or exim is also OK
sudo rpm -i gitlab-7.1.0_omnibus-1.el6.x86_64.rpm

# Edit the configuration file to add your hostname
sudo -e /etc/gitlab/gitlab.rb

# Install and start GitLab
sudo gitlab-ctl reconfigure
sudo lokkit -s http -s ssh

# Browse to the hostname and login 
# Username: root 
# Password: 5iveL!fe

I created a password for git so I could get past this. But alas...

samrocketman commented 10 years ago

Hi @TinajaLabs, Can you tell me the sha1 commit ID of the version of gitlab-mirrors you're using? This way I can look at the code for the revision you have checked out to get a better understanding of what you may be encountering.

Hang in there, when I first set up GitLab it was very complicated and hard to do so I understand your plight. I've not tried the omnibus version though I hear it's a little easier than compiling from source. None-the-less gitlab takes a bit to wrap your head around. Get me that ref and I'll see what I can do. Perhaps I'll fire up a VM and install the omnibus version of gitlab to play with it and better understand it.

TinajaLabs commented 10 years ago

Sadly, I had to download it as a zip and copy the folder due to no internet access from that box. Not sure where I might find the sha1 commit ID, otherwise. So I'm not likely to be able to refresh the code via git. I see the file VERSION in /lib has v0.4.8.

Looks like most of the files I unzipped are from February 27th. I installed them on July 25th.

I hope that helps... And thanks.

TinajaLabs commented 10 years ago

When I get the error (in add_mirror.sh, git section, git push gitlab), fatal: 'mirrors/icnga-server.git' does not appear to be a git repository, I can look at a newly created directory at, /home/gitmirror/repositories/Mirrors/icnga-server. It looks like a .git directory and has a config file like this:

[core]
    repositoryformatversion = 0
    filemode = true
    bare = true
[remote "origin"]
    fetch = +refs/*:refs/*
    mirror = true
    url = git@git.corp.xyz:chris.j/icinga-server.git
[remote "gitlab"]
    url = git@go.soc.xyz:mirrors/icnga-server.git
    fetch = +refs/heads/*:refs/remotes/gitlab/*
    push = +refs/heads/*:refs/heads/*
    push = +refs/tags/*:refs/tags/*

Also, how do these dirs play into the mirroring - if at all:

/var/opt/gitlab/git-data/gitlab-satellites/mirrors
/var/opt/gitlab/git-data/repositories/mirrors

They seem to have partial files for the icnga-server repo so I assume this is where GitLab stores the repos.

The error above seems to be referring to: /var/opt/gitlab/git-data/repositories/mirrors/icnga-server.git which has a config file like this:

[core]
    repositoryformatversion = 0
    filemode = true
    bare = true

Just trying to see if anything looks out of place.

TinajaLabs commented 10 years ago

As root, I get this info:

root@go.soc.xyz:~ # gitlab-rake gitlab:env:info RAILS_ENV=production
Instance method "lock!" is already defined in ActiveRecord::Base, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.

System information
System:
Current User:   git
Using RVM:  no
Ruby Version:   2.1.1p76
Gem Version:    2.2.1
Bundler Version:1.5.3
Rake Version:   10.3.2
Sidekiq Version:2.17.0

GitLab information
Version:    7.0.0
Revision:   d1e424b
Directory:  /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL:        http://go.soc.xyz
HTTP Clone URL: http://go.soc.xyz/some-project.git
SSH Clone URL:  git@go.soc.xyz:some-project.git
Using LDAP: no
Using Omniauth: no

GitLab Shell
Version:    1.9.6
Repositories:   /var/opt/gitlab/git-data/repositories
Hooks:      /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:        /opt/gitlab/embedded/bin/git

As gitmirror, I get this:

gitmirror@go:~$ gitlab-rake gitlab:env:info RAILS_ENV=production
chpst: fatal: unable to setgroups: permission denied
samrocketman commented 10 years ago

There's also a typo: git@go.soc.xyz:mirrors/icnga-server.git; note icnga.

TinajaLabs commented 10 years ago

Yep, typo in the --project-name got passed in... Is this critical? I thought that it would just create the name I assigned to the project name for the new repo.

In any case, this is what I got when I fixed the name. Notice the extra green_echo commands (fetched..., pruned...) to know that the program seems to break when it executes the git push gitlab command.

gitmirror@go:~/gitlab-mirrors$ ./add_mirror.sh --git --project-name icinga-server --mirror git@git.corp.xyz:chris.j/icinga-server.git
Resolving gitlab remote.
Creating mirror from git@git.corp.xyz:chris.j/icinga-server.git
Cloning into bare repository 'icinga-server'...
remote: Counting objects: 71, done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 71 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (71/71), 78.97 KiB | 0 bytes/s, done.
Resolving deltas: 100% (26/26), done.
Checking connectivity... done.
Adding gitlab remote to project.
Checking the mirror into gitlab.
fetched...
pruned...
git@go.soc.xyz's password:
fatal: 'mirrors/icinga-server.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Am I supposed to pre-create the icinga-server repo in Gitlab? I created the gitmirror user and the Mirrors group (with gitmirror as owner).

samrocketman commented 10 years ago

It should create it remotely automatically. I don't have the latest gitlab. I'll have to install it in a VM and test some things out.

TinajaLabs commented 10 years ago

Our central Gitlab is 6.7.5 (our working git repo and source).

Our mirror, target system (the one I'm working on) is 7.0.0.

TinajaLabs commented 10 years ago

Cleared out all test projects in gitlab. No projects existed at all.

gitmirror@go:~/gitlab-mirrors$ ./add_mirror.sh --git --project-name icinga-server --mirror git@git.corp.xyz:chris.j/icinga-server.git
Resolving gitlab remote.
Traceback (most recent call last):
  File "lib/manage_gitlab_project.py", line 94, in <module>
    found_project=createproject(project_name)
  File "lib/manage_gitlab_project.py", line 83, in createproject
    new_project=git.group(found_group.id).transfer_project(new_project.id)
AttributeError: 'bool' object has no attribute 'id'
There was an unknown issue with manage_gitlab_project.py

Ran it again without doing anything, and got this:

gitmirror@go:~/gitlab-mirrors$ ./add_mirror.sh --git --project-name icinga-server --mirror git@git.corp.xyz:chris.j/icinga-server.git
Resolving gitlab remote.
Traceback (most recent call last):
  File "lib/manage_gitlab_project.py", line 94, in <module>
    found_project=createproject(project_name)
  File "lib/manage_gitlab_project.py", line 80, in createproject
    new_project=git.add_project(pname,description=description,issues_enabled=options.issues,wall_enabled=options.wall,merge_requests_enabled=options.merge,wiki_enabled=options.wiki,snippets_enabled=options.snippets,public=options.public)
  File "/usr/lib/python2.6/site-packages/gitlab3-0.5.4-py2.6.egg/gitlab3/__init__.py", line 184, in fn
    data = parent._post(api._uq_url, data=kwargs)
  File "/usr/lib/python2.6/site-packages/gitlab3-0.5.4-py2.6.egg/gitlab3/__init__.py", line 418, in _post
    return self._request(requests.post, api_url, addl_keys, data)
  File "/usr/lib/python2.6/site-packages/gitlab3-0.5.4-py2.6.egg/gitlab3/__init__.py", line 435, in _request
    self._check_status_code(r.status_code, url, data)
  File "/usr/lib/python2.6/site-packages/gitlab3-0.5.4-py2.6.egg/gitlab3/__init__.py", line 411, in _check_status_code
    raise self._code_to_exc[status_code](msg)
gitlab3.exceptions.ResourceNotFound: URL: http://go.soc.xyz/api/v3/projects, Data: {'wall_enabled': False, 'snippets_enabled': False, 'description': 'Mirror of git@git.corp.xyz:chris.j/icinga-server.git', 'name': 'icinga-server', 'issues_enabled': False, 'wiki_enabled': False, 'public': False, 'merge_requests_enabled': False}
There was an unknown issue with manage_gitlab_project.py

What I noticed was that there was a shell project in Gitlab that was under namespace of Administrator/icinga-server when I was expecting Mirrors/icinga-server. gitmirror user is the owner of Mirrors group, but Administrator is not. gitmirror can create groups and is checked as an Admin.

TinajaLabs commented 10 years ago

Based on another issues thread I ran this. You can see it is in a state where running add_mirror.sh created the shell of project icinga-server but because it fails at a certain point (see below), it doesn't include the files.

root@go.soc.xyz:/home/gitmirror/gitlab-mirrors # gitlab-rake gitlab:check RAILS_ENV=production
Instance method "lock!" is already defined in ActiveRecord::Base, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.
Checking Environment ...

Git configured for git user? ... yes

Checking Environment ... Finished

Checking GitLab Shell ...

GitLab Shell version >= 1.9.6 ? ... OK (1.9.6)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
Satellites access is drwxr-x---? ... yes
update hook up-to-date? ... yes
update hooks in repos are links: ...
Administrator / icinga-server ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Check directories and files:
    /var/opt/gitlab/git-data/repositories: OK
    /var/opt/gitlab/.ssh/authorized_keys: OK
Test redis-cli executable: redis-cli 2.8.2
Send ping to redis server: PONG
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
Database contains orphaned UsersGroups? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
Administrator / icinga-server ... yes
Projects have satellites? ...
Administrator / icinga-server ... can't create, repository is empty
Redis version >= 2.0.0? ... yes
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 1.7.10 ? ... yes (2.0.0)

Checking GitLab ... Finished

The command that is failing in lib/manage_gitlab_project.py:

new_project=git.add_project(pname,description=description,issues_enabled=options.issues,wall_enabled=options.wall,merge_requests_enabled=options.merge,wiki_enabled=options.wiki,snippets_enabled=options.snippets,public=options.public)

I'm printing some values to try and see what happens:

  gitlab_namespace: Mirrors
  project_name: icinga-server
  Found group: {u'path': u'mirrors', u'id': 3, u'name': u'Mirrors', u'owner_id': None}
  Found project: False
  descrption: Mirror of http://git.corp.ecm/chris.j/icinga-server.git
  pname: icinga-server
  options.issues: False
  options.wall: False
  options.merge: False
  options.wiki: False
  options.snippets: False
  options.public: False
samrocketman commented 10 years ago

Hi Chris, Have you made any headway this week? I'll set up the latest GitLab instance to test my scripts against. Thanks for your progress update.

SAM

samrocketman commented 10 years ago

I have tested gitlab-mirrors v0.4.8 with the as of now latest GitLab CE 7.2.1. It works just fine. It's more likely you need to fix your configuration with GitLab or gitlab-mirrors. I have verified this is not a bug.

meldavis commented 10 years ago

Although this issue is closed and relatively old, the issue OP was seeing in his last post is directly related to a bothersome problem I was seeing and eventually fixed. It's not an issue with gitlab-mirrors or gitlab, but with ssh config.

My problem was that every time I tried to add a mirrored project, it would fail at the point of creating the project on the gitlab server. The only way I could work around the problem was to login to the gitlab web as the gitmirror user before executing the "add_mirror.sh" command.

I fixed the problem by adding "RequestTTY no" to gitmirror's ~/.ssh/config file.

To diagnose the problem: as the gitmirror user, try to ssh directly to the gitlab server, using the full path as specified in Host directive in the ~/.ssh/config file:

ssh gitlab.example.com

If it reports back "PTY allocation request failed on channel 0", then add "RequestTTY no" to the .ssh/config file.

It's not a one size fits all solution, it depends upon the ssh versions on the gitlab server and the gitlab-mirror machine (if different).

samrocketman commented 10 years ago

Thanks for reporting that @meldavis.