TGitCache using 25% CPU

[GoogleCodeExporter]

Tortoisegitcache uses 25% CPU even when I am not using it.

Version : 1.7.9.0  on Windows 7

Original issue reported on code.google.com by nischay...@gmail.com on 26 Jun 2012 at 10:08

[GoogleCodeExporter]

When you use Windows Explorer or other programs that involves the shell folder 
view, the overlay icons need to be displayed. So TGitCache may be running to 
cache them.

Original comment by ch3co...@gmail.com on 26 Jun 2012 at 11:36

[GoogleCodeExporter]

But it shouldn't be using that much CPU right?

Original comment by nischay...@gmail.com on 26 Jun 2012 at 12:49

[GoogleCodeExporter]

If we had the PDB files 
(http://code.google.com/p/tortoisegit/issues/detail?id=1245) we could use xperf 
to trace the CPU usage. But without symbols we can't see the cause.

Original comment by MagicAnd...@live.com on 29 Jun 2012 at 7:23

[GoogleCodeExporter]

try version 1.7.11 which offers debug symbols. If it still has the issue, 
install the Windows Performance Toolkit and run a xperf trace:

xperf -on latency -stackwalk profile -buffersize 1024 -MaxFile 256 -FileMode 
Circular && timeout -1 && xperf -d HighCPUUsage.etl

capture 30s of the cpu usage go back to CMD window and press a key to stop it. 
Zip it and upload the trace (dropbox, skydrive) and post a link here.

Original comment by MagicAnd...@live.com on 3 Jul 2012 at 7:12

[GoogleCodeExporter]

What's the status of this issue?
Btw. you can enable TGitCache systray icon, where you can open a TGitCache 
status window by double clicking on it.

Original comment by sstrickr...@googlemail.com on 12 Aug 2012 at 6:25

[GoogleCodeExporter]

I can confirm that the latest 1.7.12.0 release sounds dramatically consumes 
more CPU than the previous versions. 

I also had to kill once TGIT cache due to an unexpected continuous usage of 
about 25%.

Original comment by vincent....@seasidetech.net on 21 Aug 2012 at 8:17

[GoogleCodeExporter]

run the xperf command and capture 30 - 60s of the high CPU usgae and upload the 
zipped ETL file.

Original comment by MagicAnd...@live.com on 21 Aug 2012 at 8:20

[GoogleCodeExporter]

Since 1.7.12 we use libigt2 to read the git index, because it handles broken 
indexes better. However, there is still some room for improvement and we're 
working with the libgit2 team on a "faster" callback mechanism.

Original comment by sstrickr...@googlemail.com on 21 Aug 2012 at 10:20

[GoogleCodeExporter]

Hope this helps. My TGitCache.exe also occupies one CPU core (of four, that's 
why 25%) in special case: repository with huge amount of untracked (and not 
ignored) files. It can block the core for hours!

You can simply (i hope) reproduce this by using Google Chrome with user profile 
located in Git repository. There are a lot of small files which are overwritten 
by the browser from time to time. No need to track/untrack them, just use the 
browser and wait.

Original comment by c...@chge.ru on 20 Dec 2012 at 10:12

[GoogleCodeExporter]

@chge: I don't understand using Google Chrome with user profile located in Git 
repository. Could you elaborate more?

Original comment by ch3co...@gmail.com on 20 Dec 2012 at 2:26

[GoogleCodeExporter]

It's just an accidentally discovered way to reproduce the problem.

One can use e.g. some sort of database located in the repo. Pattern is the 
same: there must be a lot of untracked files and someone must overwrite them 
sometimes.

Original comment by c...@chge.ru on 20 Dec 2012 at 4:38

[GoogleCodeExporter]

Original comment by ch3co...@gmail.com on 6 Feb 2013 at 3:50

• Changed title: TGitCache using 25% CPU

[GoogleCodeExporter]

Is this issue still valid with the latest version?

Original comment by ch3co...@gmail.com on 16 Aug 2013 at 3:20

[GoogleCodeExporter]

Still occurring regularly here.

Also even when a core isn't used at max, memory usage of TGitCache keeps 
"dancing" up and down by a couple hundred kB in the task manager, so I guess 
something is still polling or looping  (other idle processes don't do that, and 
neither does TSVNCache.exe).

Original comment by zar...@gmail.com on 16 Aug 2013 at 3:28

[GoogleCodeExporter]

TGitCache has to iterate recursively over all files causing this behavior 
(after notified of a change). But when nothing is changed in the directories 
monitored by TGitCache this behavior should stop after a few minutes.

Original comment by sstrickr...@googlemail.com on 16 Aug 2013 at 3:31

[GoogleCodeExporter]

If the CPU time spent for TGitCache to read the repository is proportional to 
the size of the repository, then it is normal.

Original comment by ch3co...@gmail.com on 16 Aug 2013 at 3:36

[GoogleCodeExporter]

> TGitCache has to iterate recursively over all files causing this behavior 
(after notified of a change).

The ReadDirectoryChanges WinAPI function allows to get notification for files 
changes, so you don't have to iterate over all files (its supported since WinXP 
& Win2003)

> If the CPU time spent for TGitCache to read the repository is proportional to 
the size of the repository, then it is normal.

It should be proportional to the amount of modified files, not the repository 
size, otherwise it makes it very punishing to work on large repositories with 
TortoiseGit.

Also there might be something else as TSVNCache doesn't exhibit the issue.

Original comment by zar...@gmail.com on 17 Aug 2013 at 12:17

[GoogleCodeExporter]

> The ReadDirectoryChanges WinAPI function allows to get notification for files 
changes, so you don't have to iterate over all files (its supported since WinXP 
& Win2003)

Well, we have to do. Git works different to SVN. Imagine just the index 
changed, then we have to recheck all files, becasue we don't know what happened 
(e.g. git reset HEAD~50000 might be issued).

Original comment by sstrickr...@googlemail.com on 17 Aug 2013 at 12:38

[GoogleCodeExporter]

Btw. If we know, that the index didn't change, we don't iterate over all 
files...

Original comment by sstrickr...@googlemail.com on 17 Aug 2013 at 12:40

[GoogleCodeExporter]

I must be missing something...

At the moment tgitcache high CPU use seems to kick often here whenever 
something is changed in a large repository, so it can be quite punishing to use 
TortoiseGit over just command-line msysgit, up to making my git repositories 
here slower than TortoiseSVN ones for most operations (even blames and diffs 
queries which should be purely local).

In terms of repository size, the .git directory is 1 GB large, under VCS there 
is about 50k in 10k folders. I also have several other open-source repositories 
under VCS (both git and svn) for another 70k files and 20k folders.

The total of file sizes is multiple gigabytes, so any kind of brute-force 
rechecks just aren't workable, but at any time, only a very very small fraction 
of those files is actually seeing changed, even "large" pulls are 
proportionately very small.

Original comment by zar...@gmail.com on 19 Aug 2013 at 6:37

[GoogleCodeExporter]

everyone who has a high CPU usage should capture the CPU usage with a xperf 
trace and upload the trace file.

Original comment by MagicAnd...@live.com on 19 Aug 2013 at 11:17

[GoogleCodeExporter]

Also commented on the Issue #48
--------------------------------------------------------------

Explain me how to do an "xperf trace" in Windows and I will upload the data.

--------------------------------------------------------------

Windows 7 x64
TortoiseGIT Release 1.8.5.0 Released: 2013-08-18
GIT 1.8.3.msysgit.0

It left the computer running for more than 30 hours and it never ended the 
processing that it was doing... I terminated the process with Task Manager.

Original comment by robe...@staredition.net on 28 Aug 2013 at 1:07

[GoogleCodeExporter]

read my comment #4. Install the WPT and run the command.

Original comment by MagicAnd...@live.com on 28 Aug 2013 at 1:24

[GoogleCodeExporter]

It is not a simple process because I have to use an older version and make a 
few reboots.

Not sure if you can do anything with it, you can see a screenshot with the 
thread stack consuming the high CPU.

It there a method to debug the version installed or we really need to install a 
version with debug symbols?

Original comment by robe...@staredition.net on 28 Aug 2013 at 1:45

Attachments:

• tgitcache.png

[GoogleCodeExporter]

The Screenshot doesn't help.

In Windows7 you can use the Windows 8 SDK 
(http://social.technet.microsoft.com/wiki/contents/articles/4847.install-the-win
dows-performance-toolkit-wpt.aspx), this works fine. And you only need 1 reboot 
to enable the "DisablePagingExecutive" and Patch variable change. 

Original comment by MagicAnd...@live.com on 28 Aug 2013 at 2:05

[GoogleCodeExporter]

where you able to capture a xperf trace of the high CPU usage?

Original comment by MagicAnd...@live.com on 31 Aug 2013 at 6:23

[GoogleCodeExporter]

Sorry for the delay, but I could not reboot the machine in the last week :)

Attached you can find xperf with TGitCache.exe consuming a high % of CPU.

Original comment by robe...@staredition.net on 4 Sep 2013 at 11:31

Attachments:

• HighCPUUsage_TGitCache.zip

[GoogleCodeExporter]

@roberto: Do you have any process which constantly writes to or updates files 
in the working tree?

Original comment by sstrickr...@googlemail.com on 4 Sep 2013 at 3:13

[GoogleCodeExporter]

@roberto: Can you please try to captire debug strings (see 
http://code.google.com/p/tortoisegit/wiki/HOWTO_Debug)?

Original comment by sstrickr...@googlemail.com on 4 Sep 2013 at 3:26

[GoogleCodeExporter]

the CPU usage comes from String operations:

TGitCache.exe!CTGitPath::ArePathStringsEqual converts all string to lower chars 
and this causes most CPU usage. Next the 
TGitCache.exe!CTGitPath::AppendPathString causes CPU usgage.

Original comment by MagicAnd...@live.com on 4 Sep 2013 at 4:04

Attachments:

• CPUusage_TGit.png

[GoogleCodeExporter]

Hi sstrickr,

As far as I know, I don't have any process writing / updating the tree.

I do use the Apache reading from the tree, I have several VHOSTS mapped to 
several GIT projects.

I don't exactly under which conditions it starts consuming 25% CPU (constant) 
but usually it is after 2 or 3 commits.

Original comment by robe...@staredition.net on 5 Sep 2013 at 8:04

[GoogleCodeExporter]

@roberto: What we need now is the debug strings and maybe also a full dump of 
the process. The xperf output doesn't help at all.

Original comment by sstrickr...@googlemail.com on 5 Sep 2013 at 10:56

[GoogleCodeExporter]

Hi sstrickr,

When I have that information, I will let you know because it happens randomly 
and in the last days I am not commiting very often to GIT.

When I have the info, I will upload it.

Thanks for your attention.

Rgds,

Original comment by robe...@staredition.net on 6 Sep 2013 at 7:58

[GoogleCodeExporter]

xperf only logs you what you tell it. Add the File flags to the commandline to 
get the filenames. 

xperf -on latency+FILENAME+FILE_IO+FILE_IO_INIT -stackwalk 
profile+FileCreate+FileCleanup+FileClose+FileRead+FileWrite+FileSetInformation+F
ileDelete+FileRename -buffersize 1024 -MaxFile 512 -FileMode Circular && 
timeout -1 && xperf -d HighCPUUsageWithFileInfo.etl

Original comment by MagicAnd...@live.com on 6 Sep 2013 at 1:33

[GoogleCodeExporter]

have you made the new trace with the fileinfo flags?

Original comment by MagicAnd...@live.com on 7 Oct 2013 at 5:25

[GoogleCodeExporter]

Just a side note.

"TGitCache using 25% CPU" probably better be stated as "uses 100% of single 
CPU(core) on quad-core system" - that coveys a better image of what is 
happening.
Same goes for 33%, 50% and maybe 17,5% and 12,5% :)

Original comment by Arioch...@gmail.com on 10 Oct 2013 at 7:14

[GoogleCodeExporter]

Same issue here, TGitCache.exe is using 100% of a CPU core + 138M working set 
memory. Versions: TortoiseGit 1.8.5.0, git version 1.8.1.msysgit.1, Windows 7 
x64. The minidump from Sysinternals Process Explorer is attached, for what it's 
worth. Full dump is available as well if you need it.

Now I'll be trying to upgrade to the latest msysgit. The reason I was holding 
back was something to do with a problem in msysgit 1.8.2/1.8.3.

Original comment by halil.oz...@gmail.com on 12 Oct 2013 at 2:13

Attachments:

• TGitCache-mini.7z

[GoogleCodeExporter]

If nobody is providing the requested information, I'm going to close this issue.

Original comment by sstrickr...@googlemail.com on 12 Oct 2013 at 2:39

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

your dump only shows what we know. It spends most time in 
TGitCache!CCachedDirectory::RefreshStatus

provide a xperf trace (with the command from Sep 6, 2013).

Original comment by MagicAnd...@live.com on 12 Oct 2013 at 5:34

[GoogleCodeExporter]

Had to kill TGitcache tonight. It was choking my computer big time. 25%> CPU 
time...

Original comment by Richard....@gmail.com on 4 Jan 2014 at 4:41

[GoogleCodeExporter]

if no one provides the xperf files, the issue can't be fixed.

Original comment by MagicAnd...@live.com on 4 Jan 2014 at 6:06

[GoogleCodeExporter]

I've sent a full dump (with Process Explorer)​ and an xperf trace (with the 
command https://code.google.com/p/tortoisegit/issues/detail?id=1242#c34) to 
sstrickr, but haven't received any reply. And I had​ offered to provide any 
file in any format. Note that I didn't expect a personal reply, but a running 
process to fix this. Note that I said "didn't", because thankfully, this is not 
a concern for me anymore, as I've disabled the icon cache, and more importantly 
switched to Ubuntu :) Good luck to y'all...

Original comment by halil.oz...@gmail.com on 4 Jan 2014 at 9:47

[GoogleCodeExporter]

post a link to the xperf file here, so that I can take a short look at it.

Original comment by MagicAnd...@live.com on 4 Jan 2014 at 9:59

[GoogleCodeExporter]

Pretty good workaround, unless you have a really large project tree: 
Set Icon Overlay Status Cache to "Shell Extended"
See 2.35.2 of http://tortoisegit.org/docs/tortoisegit/tgit-dug-settings.html
It is also a good idea to keep your working dirs on SSD.

Original comment by ferenczi.viktor on 21 Aug 2014 at 2:54

[GoogleCodeExporter]

I have this issue as well. Recently converted a large svn repo to git and 
TortoiseSVN never had any problem keeping the cache up to date and the icon 
overlays correct. TGitCache not only uses 25% of my CPU but also causes 
Explorer to freeze until it's finished doing what it does. Shell Extended just 
doesn't show the recursive folder status no matter how long I wait.

Can someone explain why TSVN can monitor the file system without serious delays 
but TGIT can't? Yes, I understand that the index may change and require a 
refresh but isn't this the same as doing an update in TSVN and it processing 
those changes? And anyway, most of the time the index ISN'T changing, nor am I 
even changing any files, just walking through the directory structure!?!

Original comment by adam.k...@gmail.com on 12 Mar 2015 at 1:42

[GoogleCodeExporter]

i can attest that adam.k's point is well taken but after a year of griefing by
TGitCache.exe it has gone away a few minutes after applying change TortoiseGit 
- Settings - Icon Overlays - Status cache - Shell Extended - Apply per 
ferenczi.viktor (thanks).

Original comment by gherson...@gmail.com on 2 Jun 2015 at 4:31

[GoogleCodeExporter]

Same here. I've lost some functionality, but it's either do this or completely 
drop TortoiseGit since it's really not worth all the trouble (constantly 
restarting explorer.exe/tgitcache.exe and etc).

Original comment by p...@patricksmedium.com on 2 Jun 2015 at 4:56

[GoogleCodeExporter]

If nobody posts a xperf trace with the fileIO (filenames) 
(https://code.google.com/p/tortoisegit/issues/detail?id=1242#c34) the issue 
can't be fixed.

Original comment by MagicAnd...@live.com on 2 Jun 2015 at 5:07