Git on Windows 10 is extremely slow (start up/execute command...)

tinhduong commented 5 years ago

[x] I was not able to find an open or closed issue matching what I'm seeing

Setup

Which version of Git for Windows are you using? Is it 32-bit or 64-bit? 2.22.0.windows.1, 64-bit
```
$ git --version --build-options
```

$ git --version --build-options

git --version --build-options git version 2.22.0.windows.1 cpu: x86_64 built from commit: d003d728ffa6c0006da875ec6318d3f6b28a4ddb sizeof-long: 4 sizeof-size_t: 8 ++ __git_ps1 ++ local exit=0 ++ local pcmode=no ++ local detached=no ++ local 'ps1pc_start=\u@\h:\w ' ++ local 'ps1pc_end=\$ ' ++ local 'printf_format= (%s)' ++ case "$#" in ++ printf_format=' (%s)' ++ local ps1_expanded=yes ++ '[' -z '' ']' ++ '[' -z '4.4.23(1)-release' ']' ++ shopt -q promptvars ++ local repo_info rev_parse_exit_code +++ git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree --short HEAD ++ repo_info='.git false false true 0cfe289fd9' ++ rev_parse_exit_code=0 ++ '[' -z '.git false false true 0cfe289fd9' ']' ++ local short_sha= ++ '[' 0 = 0 ']' ++ short_sha=0cfe289fd9 ++ repo_info='.git false false true' ++ local inside_worktree=true ++ repo_info='.git false false' ++ local bare_repo=false ++ repo_info='.git false' ++ local inside_gitdir=false ++ local g=.git ++ '[' true = true ']' ++ '[' -n '' ']' ++ local r= ++ local b= ++ local step= ++ local total= ++ '[' -d .git/rebase-merge ']' ++ '[' -d .git/rebase-apply ']' ++ '[' -f .git/MERGE_HEAD ']' ++ '[' -f .git/CHERRY_PICK_HEAD ']' ++ '[' -f .git/REVERT_HEAD ']' ++ '[' -f .git/BISECT_LOG ']' ++ '[' -n '' ']' ++ '[' -h .git/HEAD ']' ++ local head= ++ __git_eread .git/HEAD head ++ test -r .git/HEAD ++ IFS=' ' ++ read head ++ b=refs/heads/develop ++ '[' 'ref: refs/heads/develop' = refs/heads/develop ']' ++ '[' -n '' ']' ++ local w= ++ local i= ++ local s= ++ local u= ++ local c= ++ local p= ++ '[' true = false ']' ++ '[' true = true ']' ++ '[' -n '' ']' ++ '[' -n '' ']' ++ '[' -n '' ']' ++ '[' -n '' ']' ++ local 'z= ' ++ '[' no = yes ']' ++ b=develop ++ '[' no = yes ']' ++ local f= ++ local gitstring=develop ++ '[' no = yes ']' ++ printf -- ' (%s)' develop ++ return 0
```
- Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
```

Windows 10 Enterprise, 64-bit

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.17134.829]

What options did you set as part of the installation? Or did you choose the defaults? Defaults.


# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Nano Custom Editor Path: Path Option: Cmd SSH Option: OpenSSH CURL Option: OpenSSL CRLF Option: CRLFAlways Bash Terminal Option: MinTTY Performance Tweaks FSCache: Enabled Use Credential Manager: Enabled Enable Symlinks: Disabled Enable Builtin Interactive Add: Disabled



 - Any other interesting things about your environment that might be related
   to the issue you're seeing?

Unplug the network cable/disable network adapter (currently Lan adapter), issue is temporary solved.

### Details

 - Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

MinGW64 (right click, choose "Git Bash Here").

 - What commands did you run to trigger this issue? If you can provide a
   [Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/mcve)
   this will help us understand the issue.

Just start Git Bash, Git GUI, or execute any command like git push, git pull...

 - What did you expect to occur after running these commands?

Git Bash, Git GUI start in normal time, git command execute in reasonable time (include git in Android Studio)

 - What actually happened instead?

Everything is slow, Git Bash take about 10 mins to start, other commands tooks about 10-15 mins to execute. Unplug the network cable/ disable network (Lan) adapter temporary solve this problem.

 - If the problem was occurring with a specific repository, can you provide the
   URL to that repository to help us with testing?

It occurs in the whole computer environment, not specific repository.

dscho commented 5 years ago

I saw such issues when on a domain-joined machine that had troubles reaching the domain controller. Same issue on your side?

tinhduong commented 5 years ago

I saw such issues when on a domain-joined machine that had troubles reaching the domain controller. Same issue on your side?

Yes, mine is a domain-joined machine. Since all machines (same model with mine) are installed using one image, they are having same problem. but there are other models, which have different specs, also joined to company domain, they dont have this problem.

And previously we dont have this problem in Windows 7 (domain-joined), this problem only occurs since we installed Windows 10.

Vivacio commented 5 years ago

Had the same issue here when I updated to Git 2.22.0. Was pulling out my hair and trying all sorts of options.

But today i thought you know what... let me roll back to 2.21.0 and see if i can repro the issue there as well. But lo and behold all my troubles are gone. VSCode/Git were back to be fast again.

So might be an option for you as well to try it out! just for completion sake i'm running it on Win10 (Enterprise) v1903 with the latest windows updates

fourpastmidnight commented 5 years ago

Make sure its not an over-zealous anti-virus/firewall program like Sophos or TrendMicro. I’ve seen this happen ok my machine due to this. I typically add the directory containing the git program binaries to the exclusion list.

Sent from my Windows 10 phone

From: Vivacio Sent: Monday, July 1, 2019 10:18 To: git-for-windows/git Cc: Subscribed Subject: Re: [git-for-windows/git] Git on Windows 10 is extremely slow (startup/execute command...) (#2250)

Had the same issue here when I updated to Git 2.22.0. Was pulling out my hair and trying all sorts of options. But today i thought you know what... let me roll back to 2.21.0 and see if i can repro the issue there as well. But lo and behold all my troubles are gone. VSCode/Git were back to be fast again. So might be an option for you as well to try it out! just for completion sake i'm running it on Win10 (Enterprise) v1903 with the latest windows updates — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Vivacio commented 5 years ago

Yeah thats a good one, i did check it on my laptop which uses MS Defender and Palo Alto Traps but it looks like those weren't the culprits :)

When i'm done completing my workproject i'll upgrade again and start some tracing to see if i can pinpoint where the issue starts

tinhduong commented 5 years ago

Had the same issue here when I updated to Git 2.22.0. Was pulling out my hair and trying all sorts of options.

But today i thought you know what... let me roll back to 2.21.0 and see if i can repro the issue there as well. But lo and behold all my troubles are gone. VSCode/Git were back to be fast again.

So might be an option for you as well to try it out! just for completion sake i'm running it on Win10 (Enterprise) v1903 with the latest windows updates

I tried downgrade git version from 2.22.0 to 2.21.0, and it is really useful. The start-up time and execute time is faster than before.

The anti virus is Ahnlab V3 Internet Security 9.0, I even tried to disable this, but it still occurs. on other computer, this Antivirus is installed as well, but no problem found. Since we are using company machines, it seems that there are some software to monitoring & control. They may cause problem.

XeptionE commented 5 years ago

I had this problem in sourcetree. Using sourcetree, it was very slow to do something with git. Even "cd" was very slow. Then I realized, that on windows command line git was very fast. git --version was 2.21. Then I switched in source tree to "system" git instead of "embedded". Now source tree is fast again. The embedded git has the older version 2.20. Maybe this helps.

CoppensBart commented 5 years ago

I have the same issue with version 2.23. Downgrading to version 2.22 solves the issue

h888866j commented 4 years ago

I am using a domain joint windows 10 machine. I have the same issue when I use Git-2.20.1-64-bit, so does version 2-27.0 . The symptom is really similar, a simple ls command would run 17 seconds or so when I was on company vpn. After disconnecting my vpn, using purely home network, this process would be faster, 5 seconds or so.

Before my investigation, I tried to find solution one the web, some solutions were found, especially on stackoverflow. Some resolutions has no effect or totally unusable like disabling AMD display card. What is useful is that modifying the PS1 will make the process faster...but just faster than before, not to mention the git bash start time.

I revert PS1 change and start investigation using procmon and wireshark. And I add a set -x at the beginning of the /etc/profile so bash will show every command it is executing.

A simple ls would take 17-22 seconds.
- it takes about 9 seconds before ls showing result
- then it takes 11.8seconds to show PS1
- Operation with domain controller would take 7 seconds or so（procmon -- Tools -- File Summary）.
- __git_ps1 takes most of the 11.8 seconds.
disconnect vpn and try again: it take about 5-12 seconds to complete.
- Contacts with domain controller shows bad network path, takes only 0.004609 second
- __git_ps1 takes time

From the procmon result, I saw several bash process and ls process contacted with domain controller, if you run other non bash built-in command like grep, you can see them here as well. Ok, question. Why a simple ls command raising several process? I then realized how bash executing process... which in return explained why the changing of __git_ps1 would work.

You can google how does shell/bash execute commands and script.

check Url1.

check Url2

Simply say, bash open new bash process to execute external commands and script and the command is running in the sub-process, except that script executed using source script.sh or . script.sh which is executed in current bash process. Keep in mind that your script.sh may contain some external commands...and the usage of | $() etc would invoke sub-process.

While the subshell processes each line of the script, the parent shell waits for its child process to finish. When there are no more lines in the shell script to read, the subshell terminates. The parent shell awakes and displays a new prompt.

It is hard for me to properly explain this process, but it is very common in linux systems.

If you understand the shell execute process, you will go back to check the /etc/profile to see the chain of the profile loaded and all the scripts executed inside those profiles, and the external commands inside those profile and scripts.

The goal is to reduce the number of subprocess and reduce the time-consuming calls including the replacement of __git_ps1 function but not limited to this change. I will explain the reason later.

I then chose to modify /etc/profile....hmmm, totally overwrite it :), because this profile was loaded when shell starts and this profile would execute some other profile and shell scripts including the setting of PS1. I want to know the impact of those profile and script on the timing. Well it boost the start-up time down to few seconds, and then all the command running inside git bash is much more fast (PS1 was changed to shell default at this time).

Now try again, check procmon, I can still see the contacts with domain controller...well, this is not a profile issue now. I clicked each time-consuming process in procmon to see the stack of those operation. Compare them, and found that they all have advapi32.dll, and it was trying to lookupAccountSidA, QueryUserServiceNameForContext, LsaLookupPrivilegeValue。 I suppose those are the process of contacting domain controller, and googled them. Results showing they are related with domain controller checking. Looks like this is nothing we can do here...get rid of this dll? may cause other software crash. But why they choose to use it...

This is not a git issue...but a mingw64 issue. I then checked the git-for-windows' release note to see why v2.21 is working (although i didn't try it, based on previous posts). I saw a bug been fixed in v2.22.0, I suspect the fast of v2.21.0 is caused by that bug...although i dont have evidence. git-bugfix2 21

In the release note of git-for-windows, I know that mingw64 is based on Cygwin, I then went to their official site to check why they are so slow. I found that they have explanation and possible resolution in FAQ!! It is good for machine not using AD login, but would slow things down for domain joint computers/users. I haven't try each of those options yet, but things is clear now CygwinFAQ

Shell opens subprocesses, there are notable extra costs on new process like the interaction with Domain Controller. Shell open subprocess is common on Linux systems and it is how shell works, but on Windows, on Cygwin, on git for Windows, when your computer joint domain, there are extra cost.

The default __git_ps1 consume time, it is running some commands every time after your commands to make your prompt more clear like the branch name you are on. You have your own choice to overcome those, for example, replace it with a faster one.

You may find that sometimes, your git-for-windows is faster then usual when you connected to company network, the reason could be that your company have more than one domain controller, and the latency to those server is different. You can find out all of your domain controller servers by running net group “domain controllers” /domain in your CMD.

dscho commented 4 years ago

Do things get miraculously faster after issuing

$ mkpasswd -l -c > /etc/passwd
$ mkgroup -l -c > /etc/group

in a Git Bash that was run as administrator?

(This tip comes e.g. from https://bjg.io/guide/cygwin-ad/)

h888866j commented 4 years ago

@dscho I didn't test yet. I saw fewer domain controller requests for some unknown reasons, but that is when I use a closer domain controller. I will try when I was switched to previous domain controller.

tinhduong commented 4 years ago

I solved my problem by adding an Firewall/Anti virus scan folder exception, then put git installations to this folder.

Guess that the problem in Joined Domain machine happens when git tried to connect to domain controller, then Firewall blocked these connections. Exception folder solved this.

h888866j commented 4 years ago

@tinhduong,

Your Administrator has locked this feature.

My company's anti virus software and company security strategy does not allow me to add any exceptions. Thank your for letting us know your solution.

h888866j commented 4 years ago

@dscho I don't have admin privilege but a run elevated privilege. I tried the tip in the url you shared, it is not working for me. I tried the option 4 then(in https://cygwin.com/faq.html#faq.using.startup-slow ) to modify passwd and group to files only in /etc/nsswitch.conf, it did not make things faster as far as what i saw. I then made some more changes to /etc/nsswitch.conf based on the options i found on https://cygwin.com/cygwin-ug-net/ntsec.html, try to re-generate /etc/passwd and /etc/group and try again. I dont see any difference on the timing. Maybe I was not doing it correctly, anyone who see this comment can give it a try and see if it works or not. As for the option 3 in the FAQ resolution, looks like cygserver is not installed on git-for-windows, I'm not sure so i didn't proceed with option 3.

dscho commented 4 years ago

Hmm, I am out of ideas, then... 😞

aaronsnoswell commented 4 years ago

Hello! Long time Git for windows user here, coming to +1 this issue.

I've been using Git for windows for several years on my machine. I always update to the latest Git version after the daily update checks, and recently all Git commands seem to have ground to a sluggish halt. E.g. - git status normally takes <1 second for me. Now, on a new, empty repo, git status takes over 7 seconds to run.

If I start CMD / Windows Terminal / PowerShell / Github Desktop as an administrator, all Git commands run at their usual, zippy speed.

Also, If I use Git Bash, instead of any of the above programs, Git runs at its usual speed.

Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.28.0.windows.1
cpu: x86_64
built from commit: 77982caf269b7ee713a76da2bcf260c34d3bf7a7
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh

Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.18363.1016]

What options did you set as part of the installation? Or did you choose the defaults?

$ type "C:\Program Files\Git\etc\install-options.txt"

Editor Option: SublimeText
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: LFOnly
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Enabled
Enable Pseudo Console Support: Disabled

Any other interesting things about your environment that might be related to the issue you're seeing?

My laptop is a 'work' laptop, meaning many settings are managed by the IT department. Others in this thread seem to have this in common.

Details

Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

I use Windows Terminal (Github build), or Command Prompt, or PowerShell or GitHub desktop - the problem is common across all of these tools.

if I use Git Bash, the problem doesn't occur, and Git is fast again.

What commands did you run to trigger this issue?

mkdir "Test"
cd Test
git init
git status

What did you expect to occur after running these commands?

Expected git status to run very quickly (e.g. < 1 second)

What actually happened instead?

Git status thinks for about 8 seconds before completing.

@tinhduong and others (e.g. on various Stack Overflow issues) have suggested adding Anti Virus exceptions for the Git install folder. I've tried this and it doesn't fix the issue.

I've got admin privileges for my machine, and am very keen to fix this issue - happy to work with anyone here to try any and all suggestions to get to the bottom of this.

Thank you in advance.

aaronsnoswell commented 4 years ago

dscho commented 3 years ago

Closing this stale ticket.

git-for-windows / git

Git on Windows 10 is extremely slow (start up/execute command...) #2250

Setup

Details