svnplot.py Keeps Failing at Calculating Basic Stats for Some of my Repositories

[GoogleCodeExporter]

What steps will reproduce the problem?
1. On my implementation, this statement continues to fail 100%: python 
svnplot.py -v --dpi 70 -p svnplot-long.tmpl -n "MIDAS" 
D:\SVN_Reporting\MIDAS.db D:\SVN_Reporting\MIDAS_Report\
2. However, this statement works, python svnplot.py -v --dpi 70 -p 
svnplot-long.tmpl -n "MDTAPC" D:\SVN_Reporting\MDTAPC.db 
D:\SVN_Reporting\MDTAPC_Report\

What is the expected output? What do you see instead?
I am expecting to see this:
. . .
updating file activity hotness table for revision 90
updating file activity hotness table for revision 91
updating file activity hotness table for revision 92
updating file activity hotness table for revision 93
updating file activity hotness table for revision 94
updating file activity hotness table for revision 95
updating file activity hotness table for revision 96
updating file activity hotness table for revision 97
updating file activity hotness table for revision 98
updating file activity hotness table for revision 99
updating file activity hotness table for revision 100
updating file activity hotness table for revision 101
updating file activity hotness table for revision 102
updating file activity hotness table for revision 103
Calculating Author commits trend histogram graph
Calculating Daily commit count graph
Wasted Effort Trend Graph
Calculating LoC graph
Calculating LoC and Churn graph
Calculating Developer Contribution graph
Calculating Average File Size graph
Calculating File Count graph
Calculating File Types graph
Calculating current Directory size pie graph
Calculating Directory size line graph
Calculating current Directory File Count pie graph
Calculating tag cloud for log messages
Calculating Author Tag Cloud
Calculating Basic stats
Calculating Active (hot) files list
Calculating Active authors list

D:\Program Files\Python_27\Lib\site-packages\svnplot>

Instead I see this:
...
updating file activity hotness table for revision 425
updating file activity hotness table for revision 426
updating file activity hotness table for revision 427
updating file activity hotness table for revision 428
updating file activity hotness table for revision 429
Calculating Author commits trend histogram graph
Calculating Daily commit count graph
Wasted Effort Trend Graph
Calculating LoC graph
Calculating LoC and Churn graph
Calculating Developer Contribution graph
Calculating Average File Size graph
Calculating File Count graph
Calculating File Types graph
Calculating current Directory size pie graph
Calculating Directory size line graph
Calculating current Directory File Count pie graph
Calculating tag cloud for log messages
Calculating Author Tag Cloud
Calculating Basic stats
Traceback (most recent call last):
  File "svnplot.py", line 685, in <module>
    RunMain()
  File "svnplot.py", line 682, in RunMain
    svnplot.AllGraphs(graphdir, options.searchpath, options.thumbsize, options.m
axdircount)
  File "svnplot.py", line 247, in AllGraphs
    graphParamDict = self._getGraphParamDict( thumbsize)
  File "svnplot.py", line 574, in _getGraphParamDict
    graphParamDict["BasicStats"] = self.BasicStats(HTMLBasicStatsTmpl)
  File "D:\Program Files\Python_27\Lib\site-packages\svnplot\svnplotbase.py", li
ne 91, in BasicStats
    basestats = self.svnstats.getBasicStats()
  File "D:\Program Files\Python_27\Lib\site-packages\svnplot\svnstats.py", line
1057, in getBasicStats
    stats['FirstRevDate'] = row[0]
TypeError: 'NoneType' object has no attribute '__getitem__'

D:\Program Files\Python_27\Lib\site-packages\svnplot>

What version of the product are you using? On what operating system?
Windows Server 2k3 SE SP2
Python 2.7.8
py27-pysvn-svn1612-1.7.4-1321.exe
numpy-1.9.1-win32-superpack-python2.7.exe
matplotlib-1.4.2.win32-py2.7.exe
python-dateutil-2.2.win32-py2.7.exe
pytz-2014.7.win32-py2.7.exe
pyparsing-2.0.3.win32-py2.7.exe
setuptools-5.8.win32-py2.7.exe
SVNPlot-0.7.10.win32.exe

Please provide any additional information below.
I have inspected the database created for each SVN repository and I can't see 
any differences in the SVNLog Table - specifically in the revno=1 commitdate - 
all of the SVN Repositories are working on the same SVN Server without issue.

Any assistance would be greatly appreciated!!!!! 

P.S. I have about 20 SVN Repositories - I am trying to get the process down so 
that I can run this for each of the Repositories . . .

Original issue reported on code.google.com by mwkillme...@gmail.com on 11 Nov 2014 at 1:37

Attachments:

• svnlog2sqlite.MDTAPC.log
• svnlog2sqlite.MIDAS.log
• MDTAPC.db
• MIDAS.db

[GoogleCodeExporter]

Not sure if this is still being monitored . . . but I did find what may be the 
common denominator . . . for the Repositories it is failing on, it is failing 
in the svnlog2sqlite.py step - the database is being created and there is no 
error in the log file, but it is not collecting all of the needed information - 
specifically the dif between revisions - it doesn't seem to matter if I attempt 
to collect linecounts (-1) or not - for the database that are failing the 
script seems to be skipping this part:

D:\Program Files\Python_27\Lib\site-packages\svnplot\svnlogclient.py:532: Runtim
eWarning: tempnam is a potential security risk to your program
  outpath = os.tempnam(self.tmppath, 'svnplot')

The other thing to note about these repositories that are in common - in 
revision 3 of the Repository, there is a massive commit - these repositories 
are generally for GOVT work where a solution already exists - the GOVT provides 
us the solution source code which gets committed and from there we start our 
work.

Original comment by mwkillme...@gmail.com on 11 Nov 2014 at 3:16

[GoogleCodeExporter]

Sorry, forgot to add - the manifestation of the failure is that the SVNLog 
table has zeroes for every row for the fields addedfiles, changedfiles, 
deletedfiles - the SVNLogDetail table is also empty . . . 

Original comment by mwkillme...@gmail.com on 11 Nov 2014 at 3:21

[GoogleCodeExporter]

I will check today/tomorrow and update you. 

Original comment by nitinbh...@gmail.com on 11 Nov 2014 at 9:19

[GoogleCodeExporter]

Thanks! - any assistance would be greatly appreciated!

Original comment by mwkillme...@gmail.com on 11 Nov 2014 at 6:20

[GoogleCodeExporter]

On a different note and in an effort to validate my thought process, I started 
all over on a different machine:

Win XP SP2
python-2.7.8.msi
py27-pysvn-svn1612-1.7.4-1321.exe
numpy-1.9.1-win32-superpack-python2.7.exe
matplotlib-1.4.2.win32-py2.7.exe
pyparsing-2.0.3.win32-py2.7.exe
python-dateutil-2.2.win32-py2.7.exe
pytz-2014.7.win32-py2.7.exe
setuptools-5.8.win32-py2.7.exe
six-1.8.0.win32-py2.7.exe
SVNPlot-0.8.3.win32.exe
sqlitebrowser-3.4.0-win32.exe

I am consistently getting the same result - for some of the Repositories it is 
failing during the svnlog2sqlite.py step - there is no error message in the log 
and externally it appears to complete successfully - of note though is that is 
goes quicker than the other repositories - this make sense because it is only 
partially populating the database - the SVNLogDetail table is empty and the 
addedfiles, changedfiles, deletedfiles fields of the SVNLog table has zeroes 
for every row - I am not sure what is different about the repositories except 
for the large commits in the early revisions (as we stand up the new 
repository) - please let me know what I can do to assist - thx!

Original comment by mwkillme...@gmail.com on 11 Nov 2014 at 6:24

[GoogleCodeExporter]

As additional information, I notice that when svnlog2sqlite.py appears to be 
working there is a temp file that is created in the working area 
(<name>.db-journal) - on the occasions when svnlog2sqlite.py appears to be 
failing I do not see that temp file . . .

Original comment by mwkillme...@gmail.com on 11 Nov 2014 at 7:33

Attachments:

• MDOCS.db-journal

[GoogleCodeExporter]

>>  outpath = os.tempnam(self.tmppath, 'svnplot')
Typically on windows 7 onwards, for some reason 'pysvn' is not able to create 
files in temp folder. The solution is to start a command prompt in 
'administrator' mode and then run svnlog2sqlite.py. This will generate the full 
line count information.

Also SVNPlot requires

1. pysvn for running svnlog2sqlite.py.
2. NumPy/Matplotlib is required if you want to svnplot.py
3. If you want to use svnplotjs.py (i.e. use javascript library for generating 
graphs) then NumPy/Matplotlib are not required.
4. dateutil, pyparsing,pytz etc is not required for running svnlog2sqlite or 
svnplot.

Original comment by nitinbh...@gmail.com on 12 Nov 2014 at 9:18

[GoogleCodeExporter]

try following query in sqlite browser on sqlite file generated by 
svnlog2sqlite.py

"select min(revno), max(revno), count(*) from SVNLog". -> it will give you 
minimum and maximum revision number. 

Then try following query

"select datetime(SVNLog.commitdate,"localtime") as "commitdate [timestamp]" 
from SVNLog where SVNLog.revno = <first rev>"

(Note : replace <first rev> by the minimum revision number from the first 
query).

I am just trying to ensure data stored in sqlite is correct.

Original comment by nitinbh...@gmail.com on 12 Nov 2014 at 9:22

[GoogleCodeExporter]

As you have pointed out, for some reason the SVNLog table in midas.db file 
contains 'added, deleted, changed' file counts as 0. However, in log file 
'Changed File Path' (i.e. sum of number of added, deleted and changed files is 
non zero). 

I am not sure what is the issue yet. Some how revision information is changing. 
Trying to debug this. Need more time.

Original comment by nitinbh...@gmail.com on 12 Nov 2014 at 12:41

[GoogleCodeExporter]

Just to be thorough in my responses . . . thinking that this did not apply
and that you saw later in my emails . . . but I was running the process in
a command prompt as a domain admin - just tried again with elevated privs
as a local admin - still same results - no data in the SVNLogDetail table.

Original comment by mwkillme...@gmail.com on 12 Nov 2014 at 10:08

[GoogleCodeExporter]

In the affected "MIDAS" Database the query executes and returns the
following: "1" "429" "368" - to me, these are appropriate and expected
results . . .

The next query (select datetime(SVNLog.commitdate,"localtime") as
"commitdate [timestamp]" from SVNLog where SVNLog.revno = 1) also executes
and returns: 2010-07-16 11:48:36 - this also seems to me to be appropriate
and expected . . .

Original comment by mwkillme...@gmail.com on 12 Nov 2014 at 10:12

[GoogleCodeExporter]

OK - big thanks!!!  Anything I can do to assist, please let me know. . .

Also, as an external observer, it seems to me that the revision information
is not changing - it seems more likely to me that it is not being written
(or possibly erroring on those steps) - I have run the total
svnlog2sqlite.py process many, many times now (on a dozen or so SVN
Repositories) - in observing the process on a successful run there is a
certain amount of time when the revision information is being collected -
the CPU on my VM is pegged and the process labors from step to step during
the Number revisions converted : steps (below) - on my machine, as the
process goes from 1 step to the next it takes approximately 30-45 seconds -
on the unsuccessful runs, the process flies through these steps - each 1
taking 1-3 seconds - this is why it seems to me that it is skipping or
failing on those steps - the MIDAS Repository is much larger than the
MDTAPC Repository and yet the svnlog2sqlite.py process takes like 40% of
the time . . . just my observations . . .

BIG THANKS AGAIN!!!

Number revisions converted : 10 (Rev no : 11)
Number revisions converted : 20 (Rev no : 21)
Number revisions converted : 30 (Rev no : 33)
Number revisions converted : 40 (Rev no : 43)

Original comment by mwkillme...@gmail.com on 12 Nov 2014 at 10:20

[GoogleCodeExporter]

I have added some more logging into svnplot source. Please download the latest 
source zip from bitbucket and try again. 
(https://bitbucket.org/nitinbhide/svnplot/get/tip.zip or 
https://bitbucket.org/nitinbhide/svnplot/get/tip.tar.gz)

For failing repository it seems SVNLogDetail table is remaining empty. I am not 
able find any specific reason for it. Additional log information may provide 
some clue to me. 

Also if you have any public repository where you are facing this problem ? If 
yes, please send me the url and I will try the debugging on the public url.

Original comment by nitinbh...@gmail.com on 16 Nov 2014 at 4:17

[GoogleCodeExporter]

I'm off site all week at a client's supporting one of my contracts... I'm
not sure if I will get a chance to get to this anytime soon... thank you
though!  I will get to this next week.

Original comment by mwkillme...@gmail.com on 19 Nov 2014 at 2:54

[GoogleCodeExporter]

I have uploaded a new version (0.8.4) with some bug fixes to bitbucket. Please 
try that.

Original comment by nitinbh...@gmail.com on 20 Nov 2014 at 5:52

[GoogleCodeExporter]

I just ran version 0.8.4 on both of my environments and it didn't work . . . I 
am attaching the log and the DB. 

Original comment by mwkillme...@gmail.com on 24 Nov 2014 at 8:09

Attachments:

• MIDAS.db
• svnlog2sqlite.MIDAS.log

[GoogleCodeExporter]

As for your previous request, I am trying to use the updated version with the 
additional logging - I apologize, but I am a Windows person with the most basic 
Python experience - how am I supposed to add this this package to my existing 
install? - do I use pip, or python setup.py install, or just overwrite the 
existing files - any guidance would be appreciated - thx!

As for the public repository . . . good question . . . since these are for GOVT 
contracts I am not inclined to go this path . . . let me think about it some 
more.   

Original comment by mwkillme...@gmail.com on 24 Nov 2014 at 8:16

[GoogleCodeExporter]

Goto https://bitbucket.org/nitinbhide/svnplot/downloads. Download the installer 
SVNPlot-0.8.4.win32.exe. 

Since you have already installed pysvn, just running the 
SVNPlot-0.8.4.win32.exe installer will upgrade the svnplot.

Original comment by nitinbh...@gmail.com on 25 Nov 2014 at 12:50

[GoogleCodeExporter]

I'm sorry, I probably confused you since I did your requests out of order - I 
already applied 0.8.4 and provided you the results - I inspected the .log file 
and I didn't really see any additional logging so I assumed that the additional 
logging was enabled in a private version available at 
https://bitbucket.org/nitinbhide/svnplot/get/tip.zip  that is the version that 
I was asking how to install . . .

Original comment by mwkillme...@gmail.com on 25 Nov 2014 at 4:57

[GoogleCodeExporter]

for installing tip.zip. (a) Unzip the file in directory (b) run 'setup.py 
install'. 

Original comment by nitinbh...@gmail.com on 6 Dec 2014 at 2:05

[GoogleCodeExporter]

Original comment by nitinbh...@gmail.com on 6 Dec 2014 at 2:05

• Changed state: Started