search

splunk / splunk-app-splunkgit

GitHub App
Apache License 2.0
49 stars 23 forks source link

Unable to connect to private repository #11

Open sometheycallme opened 11 years ago

sometheycallme commented 11 years ago

@petterik

Hello @petterik

I've just installed the splunkgit app on our internal splunk instance, and while it seems to work okay connecting to public repositories, when I provide a read-only account git url it doesn't seem to like this.

I'm perusing the code base to see if you are able to support private repositories or not, although the wiki seemed to mention support of both..

cc @jumanjiman

sometheycallme commented 11 years ago

We get our data from github (public repo only?), via their v3 API, and the git repository itself.

You can of course use this app for a git repository that is not on github, i.e. a local repository on your computer.

petterik commented 11 years ago

If you are trying to get data from a private repository, you can only do so if you've got your .ssh keys set up properly. See here how to do that: https://help.github.com/articles/generating-ssh-keys

However, you will only get git data and not github data (i.e. issues).

sometheycallme commented 11 years ago

Yes that makes sense, thanks for the friendly reminder.

Any way you know that we can get issue data into the splunkgit app as is? Plans to mod your python scripts?

FYI….we’re messing around with the v3 api now for some reporting, and we might extend what you’ve done here for the splunk app (the visualization is nice).

atul-tewari commented 11 years ago

I have created the ssh rsa keys and successfully cloned the repository locally. How do I configure splunkgit app in the splunkgit.conf to get git data for private repos? And has the python scripts been modified to get github data (issues)?

This would make the SplunkGit a great tool for reporting and metrics at a glance.

Thanks!

petterik commented 11 years ago

@atul-tewari you should be able to get git data for private repositories by using the ssh link to the private repository, if you've set up your ssh keys correctly. You will, however, only get git data and not github data and issue tracking. For that we will have to implement support for githubs v3 api, as @sometheycallme was mentioning. I might consider looking at this when I have the time.

@sometheycallme I totally missed your comment, and as I just mentioned, you're not able to get the issue data as the app is right now. v3 api would have to be implemented and if you do decide to do that, it'd be awesome :)

FYI: I'm fixing some of the charts for Splunk 5.x and I'm also indexing commit messages now, but the messages doesn't have any visualizations yet.

sometheycallme commented 11 years ago

@petterik

No worries. we completed our recent 5.x upgrade. we will incorporate what you have done, and any updates I'll certainly share them.

if you do decide to do that, it'd be awesome

you got it.

we've decided to take a stab at it. not likely to start until after the holiday season later in january due to some current workload, but it is up there in priority.

We will most likely use this as a foundation:

http://developer.github.com/v3/issues/events/

Good example of what it might look like

https://plugins.atlassian.com/plugins/com.atlassian.jirawallboard.atlassian-wallboard-plugin

https://github.com/archiloque/rest-client

http://www.intridea.com/blog/2008/4/1/ruby-github-simple-access-to-the-github-api

atul-tewari commented 11 years ago

Thank you so much Petterik! And just to be sure, the private and public key files have to exist in the $HOME/.ssh directory correct? Is there a configuration setting that I have to set to point to the private/public RSA keys in addition to the $HOME/.ssh directory ?

I was able to connect using these keys successfully through "SmartGit" and create a local clone of the private repositories. However, from within splunk, I was unable to connect to GitHub using ssh.

Thanks!

petterik commented 11 years ago

Awesome @sometheycallme!

@atul-tewari Yes, the keys should live in $HOME/.ssh/. Make sure you've done everything in this guide: https://help.github.com/articles/generating-ssh-keys If that's not enough, then make sure the permissions on the ssh directory and its content are secure enough. You can run: chmod 700 $HOME/.ssh and chmod 600 $HOME/.ssh/* to fix the permissions.

Another FYI: I released a new version with some small fixes. I'm not sure what version you've been using of Splunkgit all this time, but you might want to check it out.

atul-tewari commented 11 years ago

After some further investigation, I had to drop the ssh rsa key pass phrase to make it work. Thanks for your timely help. We are very keen on using Splunkgit and would love to see GIT issues retrieved through the v3 API.

After I got this to work, I upgraded to 1.3.0.1 SplunkGit. In the earlier version and the new one, I get the following error in Splunkd.log for all the private repos I connected to.

ERROR ExecProcessor - message from "$HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/fetch_git_repo_data.sh" git: $HOME/infrastructure-tool/splunk/lib/libz.so.1: no version information available (required by git).

Git data is still retrieved, though. Let me know if you have come across this error.

Thanks again!

atul-tewari commented 11 years ago

Petterik, I cleaned all the data, uninstalled splunk and reinstalled 1.3.0.1 splunkgit. I restarted Splunk with Splunkgit 1.3.0.1 after making changes to the splunkgit.conf for our repos. I get the same error about "no version available". But in addition to that I get the following errors:

I can no longer see any data come across other than number of forks and number of watchers. All other searches do not display any data. It does not get all the authors, coders, and file types like it did earlier.

Is this version a little different from 1.2.1? 1.2.1 version seemed to get all data except of course GIT issues. Are there more config changes I have to make? I enabled jQuery-hackathon as well in inputs.conf

12-19-2012 20:42:13.872 -0600 ERROR SearchResults - Failed to remove "$HOME/infrastructure-tool/splunk/etc/users/admin/splunkgit/history/u16552355.csv.tmp1": No such file or directory 12-19-2012 20:42:13.874 -0600 WARN DispatchSearch - Unable to saved search history for user=admin, app=splunkgit, sid=rt_1355971331.79, search='search index=splunkgit source="git_repo" repository=git://github.com:xxxxxxxx/xxxxxxx-api.git | dedup commit_hash | timechart count(commit_hash) by author_name | streamstats sum() as ' 12-19-2012 20:52:47.821 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" fetching git repo data for repository: git://github.com:xxx-xxxx/Xxxggh.git 12-19-2012 20:52:47.823 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" git: $HOME/infrastructure-tool/splunk/lib/libz.so.1: no version information available (required by git) 12-19-2012 20:52:47.828 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" fatal: unable to connect to github.com: 12-19-2012 20:52:47.828 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" github.com: Servname not supported for ai_socktype 12-19-2012 20:52:47.829 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" Cloning into bare repository '$HOME/infrastructure-tool/splunk/etc/apps/splunkgit/git-repositories/Xxxggh.git'... 12-19-2012 20:52:47.829 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" Unable to clone repository: git://github.com:xxx-xxxx/Xxxggh.git 12-19-2012 20:52:47.831 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" repository does not exist! 12-19-2012 20:52:47.831 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" fetching git repo data for repository: git://github.com:xxx-xxxx/abjfcch-dfj.git 12-19-2012 20:52:47.832 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" git: $HOME/infrastructure-tool/splunk/lib/libz.so.1: no version information available (required by git) 12-19-2012 20:52:47.848 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" fatal: unable to connect to github.com: 12-19-2012 20:52:47.848 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" github.com: Servname not supported for ai_socktype 12-19-2012 20:52:47.849 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" Cloning into bare repository '$HOME/infrastructure-tool/splunk/etc/apps/splunkgit/git-repositories/abjfcch-dfj.git'... 12-19-2012 20:52:47.849 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" Unable to clone repository: git://github.com:xxx-xxxx/abjfcch-dfj.git 12-19-2012 20:52:47.851 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" repository does not exist! 12-19-2012 20:52:47.851 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" fetching git repo data for repository: git://github.com:xxx-xxxx/ jhgad-asdjg.git 12-19-2012 20:52:47.853 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" git: $HOME/infrastructure-tool/splunk/lib/libz.so.1: no version information available (required by git) 12-19-2012 20:52:47.857 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" fatal: unable to connect to github.com: 12-19-2012 20:52:47.857 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" github.com: Servname not supported for ai_socktype 12-19-2012 20:52:47.857 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" Cloning into bare repository '$HOME/infrastructure-tool/splunk/etc/apps/splunkgit/git-repositories/ jhgad-asdjg.git'... 12-19-2012 20:52:47.857 -0600 ERROR ExecProcessor - message from $HOME/infrastructure-tool/splunk/etc/apps/splunkgit/bin/git_source_code.sh '.xml$'" Unable to clone repository: git://github.com:xxx-xxxx/jhgad-asdjg.git 12-19-2012 20:52:47.897 -0600 INFO ExecProcessor - Ran script: $HOME/infrastructure-tool/splunk/etc/apps/unix/bin/top.sh, took 528.4 milliseconds to run, 11840 bytes read

Thanks!

petterik commented 11 years ago

@atul-tewari Never seen that "libz.so.1: no version information available (required by git)." Error before. A quick google search suggests that you should update your git version. My gitversion is: $ git --version git version 1.8.0.2

The other errors are from enabling jQuery-hackathon. Enabling it won't give you anything right now. So I recommend keeping it disabled.

Splunkgit hasn't changed much from 1.2.1 to 1.3.0.1, and if everything except github issues was working for you with 1.2.1, I might even suggest to use that version, until I've made 1.3.x more stable.

sometheycallme commented 11 years ago

@rupak98

dreamyguy commented 11 years ago

@sometheycallme and @petterik any luck with this yet? :8ball:

dreamyguy commented 11 years ago

@petterik and @atul-tewari (since you got it to work somewhat), could you post a step-by-step guide how one gets Splunk to work with local git information? I have a few SSH keys available. Should I make Splunk point to it locally somehow? Where should the repo reside in my local machine to get its data fetched by Splunk?

Hopefully your answer(s) will help other too...

Cheers!

sometheycallme commented 11 years ago

@dreamyguy @petterik

We are actually back to the drawing board at the moment. We've been able to successfully pull information from the github api (currently pulling json for private repo notifications) And messing with passing a "time" parameter to the api query.

Next up is parsing the json output. (looking at spath, and splunkbase at the moment.)

@petterik has done quite an extensive job with python and consuming local git log data for a mirrored repo. We're actually after something a bit different. When we have some working field extractions, and / or something to play with, I'll put it up on github under the https://github.com/organizations/phlowy organization.

Sorry for the slow work, but it's difficult to find time to hack away.

sometheycallme commented 11 years ago

The ruby script we are playing with (you'll need rkj/gh github api wrapper)

https://gist.github.com/sometheycallme/5101772

sometheycallme commented 11 years ago

https://github.com/rkh/gh

sometheycallme commented 11 years ago

http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Spath


Example 3: Extract and expand JSON events with multvalued fields

The mvexpand command only works on one multivalued field. This example walks through how to expand a JSON event with more than one multivalued field into individual events for each fields's values. For example, given this event, with sourcetype=json:

{"widget": {
    "text": {
        "data": "Click here",
        "size": 36,
        "data": "Learn more",
        "size": 37,
        "data": "Help",
        "size": 38,
}}
sometheycallme commented 11 years ago

update: we were able to successfully pull some test JSON from a private github repository (and organization) using "notifications"

Then using spath - worked like a charm. automatically detected array items and pulled them out for us.

We will continue with this today, and try to post some rudimentary example of a script (or that's at least our goal).

cc: @jcwx

gnarf commented 9 years ago

Bump - any more movement, we'd like to use this with private repos.

petterik commented 9 years ago

There's not much movement in this project. I can take a look at it after the new year!

On Dec 22, 2014, at 8:18 PM, Corey Frang notifications@github.com<mailto:notifications@github.com> wrote:

Bump - any more movement, we'd like to use this with private repos.

— Reply to this email directly or view it on GitHubhttps://github.com/splunk/splunk-app-splunkgit/issues/11#issuecomment-67920557.