Repositories tab not working; adding troubleshooting documentation

[ohthehugemanatee]

Thanks for the great work on this plugin. Since lots of people are still new to git, and the integration is not a simple thing, it would be helpful to have some troubleshooting steps, or logging that we can use when things aren't working.

I'm presently stuck, and am having a hardtime working out what's broken because of a lack of logging, or maybe I just can't find it. I have everything set up, running Chiliproject 1.3.0, with the latest version of the plugin. But it does not create repositories for new projects, and it does not add chiliproject users to existing repositories.

• Gitolite works great on it's own.
• I followed the shell script (slightly modified for CentOS), and had no trouble
• I verified that Apache can sudo to git, and has read/write access to the gitolite-admin repo via publickey
• gitolite's log shows that redmine isn't pulling or pushing the gitolite-admin repo at all.

So I turned to logging to see what else might be the trouble. After switching Redmine to debug level logging, I can see a few relevant lines in the log, but nothing that notes how the plugin is trying to create a project, or add a new user to authorized_keys or to existing projects. I can see:

Repository Load (0.2ms)   SELECT * FROM `repositories` WHERE (`repositories`.project_id = 15) LIMIT 1
Repository::Git Create (0.1ms)   INSERT INTO `repositories` (`path_encoding`, `project_id`, `git_http`, `url`, `log_encoding`, `type`, `git_daemon`, `login`, `password`, `root_url`) VALUES(NULL, 15, 1, '/home/git/repositories/woogoo.git', NULL, 'Git', 1, '', '', '/home/git/repositories/woogoo.git')

That's it. I can also visit the Repository tab on my newly created project, and get a nice debug output of

Repository Load (0.2ms)   SELECT * FROM `repositories` WHERE (`repositories`.project_id = 15) LIMIT 1
Shelling out: 'git' '--git-dir' '/home/git/repositories/woogoo.git' 'branch' '--no-color'
EnabledModule Load (0.5ms)   SELECT name FROM `enabled_modules` WHERE (`enabled_modules`.project_id = 15) 
Shelling out: 'git' '--git-dir' '/home/git/repositories/woogoo.git' 'ls-tree' '-l' 'HEAD:'
Rendering template within layouts/base
Rendering repositories/git_instructions
GitolitePublicKey Load (0.4ms)   SELECT * FROM `gitolite_public_keys` WHERE (`gitolite_public_keys`.user_id = 3) AND (`gitolite_public_keys`.`active` = 1)

So I can see that the SQL calls are working fine, but I have no idea what's actually going wrong when it tries to create the repo. I have the same problem with creating a new user key, or adding a user to an existing project. I can see that the database is working perfectly, but I can't see what's failing or where.

We can use this issue to build out a good set of troubleshooting steps. Even a step by step of this is how the plugin tries to create a repo would be useful, so people can try to parallel the experience in the shell. An alternative might be to add log lines similar to the "Shelling Out:" line above. Or maybe just a logfile for error output from Git. Thoughts?

[ohthehugemanatee]

Just noting - yes, I've seen https://github.com/ericpaulbishop/redmine_git_hosting/issues/25 . He had the same issue with not being able to get much information from logs, finally resorting to debugging Ruby. For the record, I am not having quite the same symptoms, since the commands in /tmp/redmine_git_hosting work fine for me (as apache), and Passenger is running as apache (verified by looking at the file ownership of the contents of /tmp/redmine_git_hosting).

[ohthehugemanatee]

Update:

• I updated the code to make sure I was getting the faster sudo method.
• turns out Passenger WAS running as "nobody" after all. Not clear on why /tmp/redmine_git_hosting was owned by apache, but setting the Passenger default user helped a lot.
• turns out the reason Redmine wasn't creating repos was that at some point in testing I made my own change to the gitolite-admin repo (to make sure gitolite was working). Later, when redmine tried to update gitolite-admin it created a merge conflict. This errored out silently, which is a problem. Better to do a fresh pull before each redmine gitolite operation, or to explicitly say that manually creating repos borks the module. Anyway, it's a simple fix: rm -rf /tmp/redmine_git_hosting/gitolite-admin

Now Redmine creates repos and adds users to gitolite correctly. But the Repositories tab still insists that there is no repo for this project. I can use the repo just fine, and I verified that Redmine is using the /tmp/redmine_git_hosting/run_git_as_git_user script when it tries to generate the tab. As the apache user, if I run (on a freshly created project):

'/tmp/redmine_git_hosting/run_git_as_git_user' '--git-dir' 'repositories/wooga.git' 'branch' '--no-color'

I get no output, it returns 0 but nothing else. If I run

'/tmp/redmine_git_hosting/run_git_as_git_user' '--git-dir' 'repositories/wooga.git' 'ls-tree' '-l' 'HEAD:'

I get "fatal: Not a valid object name HEAD:" .

This is because it's an empty repo, there are no commits in it yet so there is no HEAD. If I manually make a first commit, then both commands return normal more normal responses... but the Repositories tab still insists that the repo doesn't exist.

I'm stuck here. It looks like Redmine cannot tell that there's a repo for the project. What test does it run? The two redmine_git_hosting scripts work fine, so it's something else that's failing.

None of it is ever logged in debug output, so at this point I'm really favoring just adding a troubleshooting section to the documentation, which walks through each of the commands that Redmine uses so the user can catch errors by duplicating those commands at the prompt. It's silly, but if we're talking about having to add logging to Redmine/Chiliproject AND this plugin, documentation is the path of least resistance.

UPDATE: noticed that it has this problem even for previously-working repos. Updated to redmine 2.0 , problem persists.

[ohthehugemanatee]

I don't know what else to do here.

• Apache has permission to access the repos directly in the filesystem
• Apache has permission to sudo to git, and vice versa.
• Passenger is running as the Apache user, though I see occasional threads spawned as git
• If I run the commands listed in "shelling out" lines of the chiliproject debug log, they return valid output.
• I tried modifying lib/redmine/scm/adapters/abstract_adapter.rb so it logs error output of commands shelled out from production. There is apparently no error output.

Gitolite integration works well - chiliproject can manipulate repos and users. I can use the repos just fine. But the Repositories tab in Redmine always shows "Repository does not exist. Create one using the instructions below." with the Git instructions.

I've got no background in Ruby, but from what I can tell that error appears when the format of '/tmp/redmine_git_hosting/run_git_as_git_user' '--git-dir' '/home/git/repositories/testing.git' 'ls-tree' '-l' 'HEAD:' is not as expected. When I run it from the shell (as Apache) on a project that has an extablished repo, I get output in this format:

100644 blob 8f6f7134d3b6aa2600536f67928cc106a9f195df   45285    CHANGELOG.txt
100644 blob 94bfd618e49f1f48f4e531062d3485f5cca4a48d     925    COPYRIGHT.txt

If I disable the redmine_git_hosting plugin entirely, I get a commit not found error for all repository tabs. (UPDATE: not for the projects that have an existing repo. This only happens for projects that the plugin created, presumably because they have no initial commit)

which looks good to me, but I don't know what Redmine is expecting. Normally, I would add more debug information in the code when it goes to print the Git instructions... but I have a fair amount of Ruby learning to do before I can figure that out, or even what information is available in the environment at that time.

Any ideas?

[ohthehugemanatee]

I managed to get the Repositories tab working for projects with > 0 commits. I just edited the /tmp/redmine_git_hosting/run_git_as_git_user file, and removed the sudo, and the printf... now it just executes "git $@", whch works because I did the legwork of permissions. Obviously that's not a real fix, but it tells me that the issue is in sudo and printf. Very weird.

Again, running from the shell I can copy/paste the output of the printf command there, and it works fine:

sudo -u git -i eval " git "--git-dir" "/home/git/repositories/testing.git" "ls-tree" "-l" "HEAD:"  "

So maybe there's something funny in my sudo version, or somewhere else in my environment? I'm using pretty vanilla CentOS5, if that helps. This needs to be addressed programmatically, too. For now, just an improved if statement that actually checks if you have read/write permissions before trying sudo would help.

I still have the problem of newly-created repositories needing a first commit before they are visible.

[that0n3guy]

I think I might be having similar issues as you, see #52. I'm running ubuntu 10.04 though. What version of sudo are you running?

[ohthehugemanatee]

sudo -V

Sudo version 1.7.2p2

How about you?

[birdming22]

I have the problem 'sudo: sorry, you must have a tty to run sudo' try 'visudo' Defaults requiretty Defaults:www !requiretty or Defaults !requiretty

[ericpaulbishop]

There are a large number of improvements to solve some of these configuration errors in v0.4.0. I'm going to close this issue for now. If you continue to have problems with the latest version, please open a new issue with the symptoms you experience under v0.4.0