Support for multiple repositories

[GoogleCodeExporter]

It would be sooo cool if I could e.g.:

gource ~/workspace/*

And it would show multiple trees (1 per repository) in one window.
I think many developers work on several projects and this way you could see
a really nifty overview of all activity.

Original issue reported on code.google.com by vanzonneveld@gmail.com on 28 Sep 2009 at 1:11

[GoogleCodeExporter]

Sorry. Not a defect but an Enhancement. Trying to change the type.. Quite 
unsuccessfully.

Original comment by vanzonneveld@gmail.com on 28 Sep 2009 at 1:13

[GoogleCodeExporter]

Interesting idea. 

Original comment by acaudw...@gmail.com on 28 Sep 2009 at 10:04

• Added labels: Priority-Low, Type-Enhancement
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

I thought it works such way. Please add this functionality.

Original comment by slatv...@gmail.com on 19 Nov 2009 at 11:36

[GoogleCodeExporter]

would be a nice option, but i can't think on a way you can connect them by any 
logic, 
what's common between them so it would be interesting to see?

bi can think of something nice if you use the "submodule" options in git. run 
it on a 
repository and it's submodules.

Original comment by igal.al...@gmail.com on 12 Apr 2010 at 9:36

[GoogleCodeExporter]

The current version in development supports reading conf files with multiple 
'gource'
sections. In this mode it will automatically toggle between the repos in the 
conf
file ever 90 seconds, or under the rules you specify. Eg:

demo.conf:
[display]
viewport=1280x720
fullscreen=true

[gource]
background-colour=000000
path=logs/linux-kernel.log
stop-at-time=60.0

[gource]
path=logs/papervision.log
start-position=0.5
stop-position=0.75

Run with command:

gource --load-config demo.conf

You can generate a conf file using --save-config:

gource -1280x720 -f logs/linux-kernel.log --save-config linux-kernel.conf

Original comment by acaudw...@gmail.com on 13 Apr 2010 at 4:04

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi. I went and tried adding this as you suggested, but then it didn't actually 
appear
to change anything:

http://github.com/acaudwell/Gource/commit/fde3c9f82257d8593883472c9b905729f09cee
ea

Any ideas?

Cheers

Andrew

Original comment by acaudw...@gmail.com on 11 May 2010 at 9:35

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Neither I could get this to work with the proposed method of having many 
[gource]
sections pointing to different log files, with Gource 0.27 just compiled from 
Git.
Only the first entry from the list is rendered.

Also I tried joining the different log files together, and then Gource displays 
them
one after another. I suspect it would be possible to create a small Python 
script
that generates submodule log files, prepends the submodule path to the commit
filenames, and outputs all commits in the correct sequence based on time stamps.

Original comment by mikael.l...@gmail.com on 11 May 2010 at 9:42

[GoogleCodeExporter]

Yes, I proposed "submodule=log", but as I experimented with it, I noticed it 
did not
actually do what I think it should be doing and deleted the comment. :)

So if this command does not exist in Git, maybe it would be reasonabe to write a
simple submodule log analyzer that could combine logs from submodules (and also 
from
completely separate repositories) together.

It might be useful to track a certain user, not just a specific repository. 
Users
also work across different version control systems, so a "Gource" log format 
would be
useful in having support for watching a user working with many separate 
projects.

Original comment by mikael.l...@gmail.com on 11 May 2010 at 9:50

[GoogleCodeExporter]

Ah ok. No worries.

I think I kind of missed the point of the original feature request. The feature 
I
added lets you play a series of repositories one after another, which is quite 
good
for variety and showcasing a series of distinct projects, but not for say 
watching
the cross over between Linux and Git development, or something like that.

Showing users moving between repositories is a really interesting idea. I guess 
how
it could work would be to just add an extra level to the tree before the root 
of each
project, but not connect it by lines or draw the central node at all.

Original comment by acaudw...@gmail.com on 11 May 2010 at 11:37

[GoogleCodeExporter]

Then I missed the point of the actual feature. :)

But there is a problem. The submodules I have are created before the main 
repository.
The render timeline starts from the wrong position, somewhere at the end. 
Navigating
before that point does not work, the point just jumps to the create date of the 
first
path=log.file. How can I navigate to the next?

This is the ad-hoc script I wrote that generates the gource.conf and runs Gource
http://gist.github.com/397230

My actual request is that all submodules are rendered simultaneously in the same
graph. :) 

Original comment by mikael.l...@gmail.com on 11 May 2010 at 12:23

[GoogleCodeExporter]

If you press enter it will go to the next repo specified in the conf file.

Also, it will change by itself after 60 seconds, or you can also use 
--stop-at-time
SECONDS to specify how long each one lasts.

Original comment by acaudw...@gmail.com on 11 May 2010 at 9:02

[GoogleCodeExporter]

'--show-submodules' support could be implemented as:
- read .gitmodules file
- open log for main project
- open log for each submodule
- re-write submodule file paths to be prefixed by the location of the submodule 
in
the main repository.
- show them on one tree.

Original comment by acaudw...@gmail.com on 11 May 2010 at 11:07

[GoogleCodeExporter]

That could be quite simple with Python and git-python:
http://packages.python.org/GitPython/tutorial.html#tutorial-label

However I am a bit puzzled how to get the "raw" output from the Commit object.
http://packages.python.org/GitPython/reference.html#module-git.objects.commit

My C++ is quite rusty, I tried to browse through the source but maybe you could 
tell
me how strict Gource is regarding the "raw" content of the git logs?

For example:
:000000 100644 0000000... 42c0fb8... A  README

Are the first three fields used for anything? The fourth field is the sha in 
short form.

Should not the commits be in proper time sequence, or does Gource reorder the 
entries
by their timestamp?

What about "submodule update" commits from the main repo, that operate just on 
the
directory? Should they be removed?

Original comment by mikael.l...@gmail.com on 12 May 2010 at 10:13

[GoogleCodeExporter]

Ah, git diff manual explains these fields.
http://www.kernel.org/pub/software/scm/git/docs/v1.6.0.6/git-diff-files.html

   1. a colon.
   2. mode for "src"; 000000 if creation or unmerged.
   3. a space.
   4. mode for "dst"; 000000 if deletion or unmerged.
   5. a space.
   6. sha1 for "src"; 0{40} if creation or unmerged.
   7. a space.
   8. sha1 for "dst"; 0{40} if creation, unmerged or "look at work tree".
   9. a space.
  10. status, followed by optional "score" number.
  11. a tab or a NUL when -z option is used.
  12. path for "src"
  13. a tab or a NUL when -z option is used; only exists for C or R.
  14. path for "dst"; only exists for C or R.
  15. an LF or a NUL when -z option is used, to terminate the record.

Possible status letters are:

    * A: addition of a file
    * C: copy of a file into a new one
    * D: deletion of a file
    * M: modification of the contents or mode of a file
    * R: renaming of a file
    * T: change in the type of the file
    * U: file is unmerged (you must complete the merge before it can be committed)
    * X: "unknown" change type (most probably a bug, please report it)

Original comment by mikael.l...@gmail.com on 12 May 2010 at 10:28

[GoogleCodeExporter]

Hi.

If you're constructing a combined log file it's probably best to use the 
'custom' log
format option which is really simple to write rather than emulate the git log 
format.

Submodule update commits would need to be removed (they currently show up as 
white
files, but I probably could detect them and filter them out).

Gource will play the log back in the order it is presented. If entries are out 
of
order it will notice and time travel (the clock will jump backwards).

Original comment by acaudw...@gmail.com on 12 May 2010 at 11:15

[GoogleCodeExporter]

Ah, the 'custom' log format is documented on the man page! Very neat.

Maybe it is best that Gource plays exactly the log file order. The log can be
preprocessed outside the core.

Original comment by mikael.l...@gmail.com on 12 May 2010 at 11:30

[GoogleCodeExporter]

This is a small bit of Python I wrote, and it seems to work. :)
At least I can see one of my Git projects with submodules to render 
simultaneously in
the timeline and the branching looks quite ok too.
http://gist.github.com/398626

Usage:
Run the Python program in your main repo, it will generate submodules.log.

Put this in gource.conf:
[gource]
  path=submodules.log

Launch:
gource --load-config gource.conf --log-format custom -s 0.1

Original comment by mikael.l...@gmail.com on 12 May 2010 at 2:14

[GoogleCodeExporter]

Cool :)

You probably don't need --log-format custom as it will guess. You only require 
that
if you are reading from STDIN.

Original comment by acaudw...@gmail.com on 12 May 2010 at 9:47

[GoogleCodeExporter]

I created a video and put it up on Vimeo.
http://www.vimeo.com/11694038

The first three branches are Git submodules (migrated from SVN) and somewhere 
at the
end the fourth big branch is the main repository. 

Looks much like I expect it to. :)

Original comment by mikael.l...@gmail.com on 13 May 2010 at 1:17

[GoogleCodeExporter]

For those who wish to run the Python script, I changed the usage I described 
above a
bit. Place the script to your path (eg. /usr/local/bin/), give it execution
privileges (chmod +x) and start in your main repo with submodules. The script 
will
generate the log file, gource.conf (overwrites if one exists) and starts Gource.

Original comment by mikael.l...@gmail.com on 13 May 2010 at 2:12

[GoogleCodeExporter]

Does the multiple [gource] blocks not work with HG? I have tried using this and 
it doesn't seem to work.

Original comment by keber...@gmail.com on 27 Jan 2011 at 5:24

[GoogleCodeExporter]

Not with that Python script.
Adding Mercurial support should be quite easy.
You need it for something particular soon?

Original comment by mikael.l...@gmail.com on 27 Jan 2011 at 8:25

[GoogleCodeExporter]

I would suggest that multiple repos would be connected by date ... I just made 
a gource video and really would love to run across all my repos (I work on any 
of like 8 repos in the course of a year) so I can see all the developer 
activity across the company, and the project sizes compared to each other, as 
well as where we're focusing our development efforts around the company.

I liked the idea before about a hidden head node connecting all the children 
nodes.

So is this implemented for sure or not quite yet?

Oh, and my video http://www.youtube.com/watch?v=WbJ8Gefq_QE ;)

Original comment by drachens...@gmail.com on 23 Mar 2011 at 6:40

[GoogleCodeExporter]

Is it possible to view multiple repos on one tree, each tree branch for one 
repo? I have my repos based on SVN. I was trying to edit and merge log files by 
myself, but the gource output was never as I wanted. Is there any script for 
that?

Original comment by viri...@gmail.com on 23 Mar 2011 at 9:38

[GoogleCodeExporter]

Hi. Ok, here is a quick hack to do this now with a few shell commands:

1. Generate a Gource custom log files for each repo. This can be facilitated by 
the --output-custom-log FILE option of Gource as of 0.29:

  gource --output-custom-log log1.txt repo1
  gource --output-custom-log log2.txt repo2
  ...

2. If you want each repo to appear on a separate branch instead of merged onto 
each other (which might also look interesting), you can use a 'sed' regular 
expression to add an extra parent directory to the path of the files in each 
project:

  sed -i -r "s#(.+)\|#\1|/repo1#" log1.txt
  ...

(note: sed -i does an in-place replacement on the file)

3. Join the logs together, and sort them numerically by the first column (the 
time):

  cat log1.txt log2.txt | sort -n > combined.txt

3. Feed result into gource:

  gource combined.txt

I'll make a wiki page about it.

I should add the hidden root node ('--hide root' I guess) option... I'd 
forgotten about that.

Andrew

Original comment by acaudw...@gmail.com on 23 Mar 2011 at 10:30

[GoogleCodeExporter]

It works, but only when I run gource as you said. When I run it with -i 0 or on 
faster speed, it doesn't add new repo-branches on the tree. When nodes 
disappear (without idle settings) then gource adds new repo-branches. Am I 
doing something wrong?

Original comment by viri...@gmail.com on 23 Mar 2011 at 2:16

[GoogleCodeExporter]

@Andrew ~ Thanks, I'll try it after lunch. I appreciate the followup.

Original comment by drachens...@gmail.com on 23 Mar 2011 at 3:10

[GoogleCodeExporter]

Thanks I hope it works.

Maybe try adding --max-files 0. Also check that the files were sorted 
successfully.

Original comment by acaudw...@gmail.com on 23 Mar 2011 at 7:38

[GoogleCodeExporter]

Worked beautifully for me.
Here is a small script based on the tips presented here.
https://gist.github.com/884376

Original comment by mikael.l...@gmail.com on 24 Mar 2011 at 1:14

[GoogleCodeExporter]

Thanks! After I've added "--max-files 0" switch, it worked just as I wanted :).

Original comment by viri...@gmail.com on 29 Mar 2011 at 7:36

[GoogleCodeExporter]

I've added --hide root in the development version. You can toggle it with the R 
key.

Original comment by acaudw...@gmail.com on 5 Apr 2011 at 2:00

[GoogleCodeExporter]

Yes please support git submodules!

Original comment by ashap...@artec-group.com on 10 Jul 2012 at 8:39

[GoogleCodeExporter]

Submodule support would be wonderful. Also specifying a commit range would help 
a lot. I tried the first script, but it took an hour due to endless submodule 
commits outside of the scope of the main project. Then it crashed due to some 
UTF8 encoding issue...

Original comment by wertep...@gmail.com on 7 Sep 2012 at 6:17

[GoogleCodeExporter]

bump, any news about the submoules feature?

Original comment by vha...@cellbest.biz on 21 Mar 2013 at 8:56

[GoogleCodeExporter]

bump! Please give us a nice x-mas present with submodule support! :-)

Original comment by alexande...@castlabs.com on 14 Nov 2013 at 1:18

[GoogleCodeExporter]

I just tried running the linkes gist. It seems to no longer work. I got the 
following error:

Main repository: <git.Repo "/home/michel/work/gourcer/.git">
Traceback (most recent call last):
  File "gource-submodules.py", line 146, in <module>
    create_log(filename=logfilename)
  File "gource-submodules.py", line 107, in create_log
    commits = repo.iter_commits('master')
AttributeError: 'Repo' object has no attribute 'iter_commits'

This is with gitpython==0.1.7 

What version did you use at the time?

Original comment by exh...@gmail.com on 25 Nov 2013 at 10:56

[GoogleCodeExporter]

Update: gitpython is not pep386 compatible so pip installed a very old version. 
Manually running the following fixed it: 

./env/bin/pip install gitpython==0.3.2.rc1

On the other hand, I just created a new repo with submodules, and it does not 
seem to taks any past history into account. I'll continue fiddling around.

Original comment by exh...@gmail.com on 25 Nov 2013 at 11:01

[GoogleCodeExporter]

In the end, I have given up. It does not work for me, and it seems to be a 
gitpython issue.

I wanted to visualize the work of my team. Most of our projects are unrelated, 
in the sense that no project uses another as submodule, but some people work on 
more than one project. So, to visualise the work, I decided to create a new, 
empty git repo and add the other projects as submodules:

    git init
    git submodule add foo bar
    git submodule add baz frob
    ...

And then run the linked gist on it. It seems, that gitpython does not work 
properly with submodules in this way.

Having submodule support baked in in gource would be really nice.

Original comment by exh...@gmail.com on 25 Nov 2013 at 1:12

[GoogleCodeExporter]

Hi. I'm the author of the Python script and after reading your trouble with it, 
I felt obliged to comment.

Please consider the Python script obsolete.

Here is a better shell script that works with any repository type Gource itself 
supports. So you can use Git, Bazaar, Mercurial, etc. They don't need to be 
submodules.

https://gist.github.com/anonymous/8005593

Pass the list of repositories as arguments to the script. By default, it will 
produce a two-pass-encoded mp4 file for you, but you can tweak the ffmpeg 
settings to anything you like.

Happy holidays. :)

Original comment by mikael.l...@gmail.com on 17 Dec 2013 at 2:24

[GoogleCodeExporter]

Hay, this script looks good, thank you! this is an issue from 2009 :)

I still think this issue can be useful, the initial request was:
> $ gource ~/workspace/*
> And it would show multiple trees (1 per repository) in one window.

This means all repos are inside the same movie, not one movie for each one.
This is a functionality that needs to be coded into gource as i understand.

Initially I needed this to show the work of one repository and it's 
sub-modules, sometimes sub-modules in git are part of the same project, so i 
think it would be nice addition.

Original comment by igal.al...@gmail.com on 17 Dec 2013 at 6:31

[GoogleCodeExporter]

The script does produce one video where all the commits from the repositories 
passed on the command line are combined. It does this by first parsing the 
commits logs into a standard format and then playing it on Gource.

I agree that this feature should be a part of Gource core, and perhaps the 
repositories should not have connections between them, but rather be 
graphically separate entities.

Original comment by mikael.l...@gmail.com on 17 Dec 2013 at 8:10

[GoogleCodeExporter]

ya, sub-modules will be represented kind of like folders, but more separated. 
this gives a whole new prospective on some projects.

Original comment by igal.al...@gmail.com on 17 Dec 2013 at 8:19

[GoogleCodeExporter]

I've made a wiki page documenting my method:

http://code.google.com/p/gource/wiki/GourceMashups

I think I might be coming around to actually doing some work on this...

Original comment by acaudw...@gmail.com on 18 Dec 2013 at 7:13

[GoogleCodeExporter]

great, this what i was looking for! i'll try this out.

Original comment by igal.al...@gmail.com on 19 Dec 2013 at 2:43