All tools from shell are very slow

[mmozeiko]

Any tool I run in msys2 shell (ls, mv, ...) are very slow. Simply ls takes something like 3 seconds.

Running "strace ls" I see following output:

   15   39327 [main] ls 7460 DLL build:    2014-11-21 06:38
   16   39343 [main] ls 7460 dtable::extend: size 32, fds 0x1802FB488
  381   39724 [main] ls 7460 transport_layer_pipes::connect: Try to connect to named pipe: \\.\pipe\msys-07c9b4f1c823d951-lpc
   40   39764 [main] ls 7460 transport_layer_pipes::connect: Error opening the pipe (2)
   31   39795 [main] ls 7460 client_request::make_request: cygserver un-available
--- Process 7460, exception 00000000 at 000007FEFD22940D
1804450 1844245 [main] ls 7460 reg_key::build_reg: failed to create key ServicesForNFS in the registry
 2011 1846256 [ldap_init] ls 7460 cygthread::stub: thread 'ldap_init', id 0x2410, stack_ptr 0x2FCAC90
310497 2156753 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <...USERNAME...:*:1599881:1049089:....SOME_REGISTRY_KEY...:/home/...USERNAME...:/usr/bin/bash>
 6571 2163324 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Domain Users:S-1-5-21-1915207013-2615040368-3076929458-513:1049089:>
  768 2164092 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Performance Log Users:S-1-5-32-559:559:>
  402 2164494 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Users:S-1-5-32-545:545:>

Could slowness be related to this? I am on non-admin user on this machine.. Can that be avoided?

[maoueh]

Are you using LDAP to connect to your computer? If yes, this is similarly related to this issue, but I'm not 100% sure.

To test that you are facing the same issue, you could create a /etc/passwd with your account information so they are not fetch from Windows directly. This should be fairly simple to test it out, however, I never did it myself yet.

Information in this thread or on cygwin's nsswitch.conf could prove useful. Maybe others have a fairly simple cookbook to follow.

[mmozeiko]

I using Windows by logging in with domain user.

I created /etc/passwd and /etc/group files using mkgroup and mkpasswd utilities. They include all groups and usernames that were in "strace ls" output (and more, but that's irrelevant). But running "ls" is still slow. "strace ls" now show this:

   12   43607 [main] ls 14816 dtable::extend: size 32, fds 0x1802FB3E8
  236   43843 [main] ls 14816 transport_layer_pipes::connect: Try to connect to named pipe: \\.\pipe\msys-07c9b4f1c823d951-lpc
   39   43882 [main] ls 14816 transport_layer_pipes::connect: Error opening the pipe (2)
   21   43903 [main] ls 14816 client_request::make_request: cygserver un-available
--- Process 14816, exception 00000000 at 000007FEFD22940D
1459351 1503254 [main] ls 14816 reg_key::build_reg: failed to create key ServicesForNFS in the registry
966780 2470034 [main] ls 14816 cygheap_user::ontherange: what 2, pw 0x1802FB6B8
   41 2470075 [main] ls 14816 cygheap_user::ontherange: HOME is already in the environment /home/USERNAME
   96 2470171 [main] ls 14816 build_argv: argv[0] = 'ls'
10469 2480640 [main] ls 14816 build_argv: argc 1

As you can see it has no more "pwdgrp::fetch_account_from_windows" lines. But there are still errors: 1) something about connecting to named pipe + cygserver 2) something about failure to create registry key + ServicesForNFS

[jmichae3]

• if it's running anything over a network like a network share, you can expect significant lag
• if it's 32-bit code running on a 64-bit box, you can expect the virtualization (WOW64) to slow the program down. but hopefully not that much on a local box, especially if it's in the OS's internal disk cache in RAM. first access will be slow, subsequent accesses should be much faster due to disk cache, but the virtualization thing is another hurdle, however it doesn't slow things down too much I should think.

[KrullBorg]

my case is slightly different; for me msys is slow on starting and on bash completion; but "ls" is fast

see http://sourceforge.net/p/msys2/discussion/general/thread/87925e6f/

i also tried to create passwd/group; "mkgroup -d" doesn't output group from domain

[KrullBorg]

i "solved" disabling db in nsswitch.conf

[Alexpux]

ok.

[mmozeiko]

This is great. Commenting out "db" from nsswitch.conf helps. Now all operations are fast for me.

[legends2k]

Disabling db from both passwd and group fixed the issue but led to my username being unrecognised (bash prompt showed me as Unknown+User). However, the "fix" was to just disable db from group and leave passwd as it was, and this sped up MSYS2 utilities like diff, which, ls and also the startup of MSYS2 significantly.

[KrullBorg]

unfortunately now some commands are slower than some time ago; for example: ls, git status/diff, configure (autoreconf), gitk

[KrullBorg]

re-enabling db in nsswitch.conf makes these commands again fast; but it slows bash completion (TAB); the start of msys2_shell.bat is quite normal

[legends2k]

@KrullKorg Bash completion being slow has got nothing to do with MSYS2 itself. See here for an explanation.

[KrullBorg]

as i wrote bash completion isn't slow with db disabled in nsswitch.conf and slow if db is enabled; tomorrow i'll try again, to be sure

of course I'm talking about a pc under ms (samba) domain; at home it works perfectly

and yes i know that msys2 derived from cygwin

[KrullBorg]

i just tried under latest cygwin (1.7.5) and it works perfectly

[elieux]

Notable differences between MSYS2 and Cygwin:

• the current MSYS2 runtime is based on Cygwin trunk from April 10th, not on Cygwin release
• MSYS2 doesn't include cygserver and other Cygwin system integration tools

Can you look at differences between your Cygwin and MSYS2 installs? Namely /etc/passwd, /etc/group, /etc/nsswitch.conf, /etc/fstab, output of mount, a cygserver process running. MSYS2 should take advantage the same caching mechanisms as Cygwin, at least within a process tree.

I'm sorry I'm dumping all the debugging on you, but I don't currently have any domain to join here.

[KrullBorg]

i don't have /etc/passwd, /etc/group and cygserver in both msys2 and cygwin

FSTAB - CYGWIN

# /etc/fstab
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.  For a description
#    see https://cygwin.com/cygwin-ug-net/using.html#mount-table

# This is default anyway:
none /cygdrive cygdrive binary,posix=0,user 0 0

FSTAB - MSYS2

# For a description of the file format, see the Users Guide
# http://cygwin.com/cygwin-ug-net/using.html#mount-table

# DO NOT REMOVE NEXT LINE. It remove cygdrive prefix from path
none / cygdrive binary,posix=0,noacl,user 0 0

NSSWITCH - CYGWIN

# /etc/nsswitch.conf
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.  For a description
#    see https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-nsswitch
#
# Defaults:
# passwd:   files db
# group:    files db
# db_home:  cygwin desc
# db_shell: cygwin desc
# db_gecos: cygwin desc

NSSWITCH - MSYS2

# Begin /etc/nsswitch.conf

passwd: files db
group: files db

db_enum: cache builtin

db_home: cygwin desc
db_shell: cygwin desc
db_gecos: cygwin desc

# End /etc/nsswitch.conf

MOUNT - CYGWIN

C:/cygwin64/bin on /usr/bin type ntfs (binary,auto)
C:/cygwin64/lib on /usr/lib type ntfs (binary,auto)
C:/cygwin64 on / type ntfs (binary,auto)
C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
I: on /cygdrive/i type smbfs (binary,posix=0,user,noumount,auto)
K: on /cygdrive/k type ntfs (binary,posix=0,user,noumount,auto)
O: on /cygdrive/o type smbfs (binary,posix=0,user,noumount,auto)
V: on /cygdrive/v type smbfs (binary,posix=0,user,noumount,auto)
X: on /cygdrive/x type smbfs (binary,posix=0,user,noumount,auto)
Z: on /cygdrive/z type sshvfs (binary,posix=0,user,noumount,auto)

MOUNT - MSYS2

C:/msys64 on / type ntfs (binary,noacl,auto)
C:/msys64/usr/bin on /bin type ntfs (binary,noacl,auto)
C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto)
I: on /i type smbfs (binary,noacl,posix=0,user,noumount,auto)
K: on /k type ntfs (binary,noacl,posix=0,user,noumount,auto)
O: on /o type smbfs (binary,noacl,posix=0,user,noumount,auto)
V: on /v type smbfs (binary,noacl,posix=0,user,noumount,auto)
X: on /x type smbfs (binary,noacl,posix=0,user,noumount,auto)
Z: on /z type sshvfs (binary,noacl,posix=0,user,noumount,auto)

PS: i just tried to remove db_enum from nsswitch.conf in msys2 PPS: don't worry about ask info; i'm happy to help msys2

[elieux]

Can you try to mount cygdrive in Cygwin with noacl? I don't see why noacl would make things slower, but it's worth a shot.

I'm assuming you're trying ls somewhere inside /c/ (/cygdrive/c/) and not inside /home/ or somewhere like that.

[KrullBorg]

i changed /etc/fstab in cygwin but i got the same behavior adding noacl: it's fast

and same behavior both in /home/ and in /c/

i also tried to remove noacl in msys2, but without improvements

[elieux]

I'm out of ideas at this point. I don't know enough about Cygwin internals to suggest a good test for you. You can try searching the Cygwin mailing list, maybe some of the information there will apply to your situation.

[KrullBorg]

i just tried on a relative new pc (virtual machine) with windows 10 tp (under the same domain) and it seems to work...

i installed msys2 from exe; then i upgraded with pacman -Syu

so i'll try to use a clean windows 8.1 vm... maybe the problem is the windows version... or my pc

[KrullBorg]

i really think it's the fault of my pc; i tried also with windows 8 and it works

i found on strace executed on my pc this error (that isn't present in virtual machines)

2388701 3040772 [main] ls 7032 pwdgrp::fetch_account_from_windows: LookupAccountSid(S-1-5-21-726227932-2052316878-829958588-513), Win32 error 1789

[elieux]

Interesting. KB976494 says the error is ERROR_TRUSTED_RELATIONSHIP_FAILURE "The trust relationship between this workstation and the primary domain failed.", so this could be a bug in Windows, or your domain connection is borked.

Can you try to leave and re-join the domain?

[KrullBorg]

yes i arrived at the same conclusion, and it worked

[k-takata]

I also faced the same issue with KrullKorg and legends2k's case. I found that expansion of ~* was very slow. I had to update /etc/passwd and /etc/group, then disable db in nsswitch.conf. More detail: https://gist.github.com/k-takata/9b8d143f0f3fef5abdab

[elieux]

Very nice summary, @k-takata.

[kjeremy]

@k-takata your workaround gist works great until I disconnect from the domain and hop onto another network... then it's slow again.

[virtuald]

I found that pacman installs are really slow until I applied @k-takata 's workaround. I wonder why it's so slow for it to do lookups on the domain?

[Globik]

And why no cygserver? Cannot run now postgresql. Error like function is not implemented.

[riban-bw]

Even after changing nsswitch.conf things are still a bit slow after dropping off an AD network. I found that disabling network and then re-enabling (using my laptop hardware button) made things responsive again.

[i2000s]

This may not be related to this issue, but Google led me here. My MSYS has been extremely slow (ran 12 hours not to finish a ./configure which could be finished in less than 1 min on Linux) using MinGW on Windows 10. All anti-virus software has been turned off, and I couldn't find the nsswitch.conf file on my installation directory. Any idea?

[legends2k]

@i2000s Are you using MSYS and not MSYS 2? They're very different! I'm pretty sure that every installation of MSYS 2 I've seen had /etc/nsswitch.conf.

[i2000s]

Yeah, eventually I found the file and commented out '#db' for the:'group'. But that doesn't help. Everything is extremely slow.

[i2000s]

@legends2k I think I have found the problem: my Anti-Virus software has been dragging every single one of my commands. Disabling it doesn't change the game. Once I completely removed the anti-virus software, everything started working again. Just to let you know. Thanks.

[legends2k]

@i2000s Glad it worked out. Thanks for letting us know!

[sskras]

@i2000s it would be useful for readers to know the name of the AV software you removed.

[i2000s]

I think the AV is called Avast. I have uninstalled it.

[Arnavion]

With a new msys2 installation today, I observed every process, the initial bash.exe spawned by mintty.exe as well as every process spawned by bash, taking ~1s to ~3s to start. strace bash 2>&1 | ts revealed all the time was spent before the first line out of output from strace, suggesting it was part of the process startup itself.

I tried the nsswitch.conf technique in the comments above but it did not change anything. (I believe Cygwin had got a patch to make the workaround unnecessary, and perhaps msys2 has had that same patch as well.)

So I traced bash.exe through procmon and observed it spending ~500ms on every startup enumerating ~8k files that match C:\Windows\System32\catroot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\*.cat, each with a callstack containing appid.sys (the AppLocker driver). That led me to this page, and indeed my machine has AppLocker policies enabled (). After code-signing all binaries under C:\msys64\usr\bin as that page suggests (`cd C:\msys64\usr\bin\; gci -re .exe | %{ &'C:\Program Files (x86)\Windows Kits\8.1\bin\x64\signtool.exe' sign /a /uw $_.FullName }`), all msys2 process start nearly instantaneously.

(*) secpol.msc actually says there are no AppLocker policies, but Get-AppLockerPolicy -Effective clearly lists several.

[legends2k]

Thanks for the info. Does this mean every time an executable under /usr/bin (, /mingw64/bin or /mingw32/bin) changes or new ones get added, the signtool has to be rerun?

[Arnavion]

Yes. Signing adds a signature to the executable, so if the executable changes it'll need to be signed again.

[RonaldinhoL]

@i2000s i try many ways and it has no help, then i try to close my AV, evething works! 3ks!

[StarWolf3000]

@i2000s i try many ways and it has no help, then i try to close my AV, evething works! 3ks!

@RonaldinhoL Yes, your AV scans most of the utilities every time, if they're not signed and in an internal database with checksums, to make sure they don't harm your system. You should be able to exclude your MSYS2 installation from scanning. AV software and similar security applications can be a nightmare for developers.

[jwiegley]

I've been encountering this exact same slowdown, but the nsswitch.conf changes have not helped. What else may be tried?

[jwiegley]

I was able to reproduce this in a Windows 7 virtual machine. After doing an upgrade with pacman -Syu followed by restarting the shell and running pacman -Su, solved the problem for me.

[jwiegley]

On my friend's Windows 10 machine, updating did not help.

[jmichae3]

unsubscribe

---

Jim Michaelsjmichae3@yahoo.comRenewal Computer Services, Vancouver, WA, USA (for now) https://www.RenewalComputerServices.com (Software, Specifications, fixes)1-360-209-5351 business

On Thursday, September 27, 2018, 8:28:28 AM PDT, John Wiegley <notifications@github.com> wrote:  

I was able to reproduce this in a Windows 7 virtual machine. After doing an upgrade with pacman -Syu followed by restarting the shell and running pacman -Su, solved the problem for me.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[smaudet]

I definitely think there's a culprit deep in some policies somewhere, between windows and linux, the same bash script runs ~50x faster on linux than it does on windows...I don't think it is I/O bound and I don't think it is a problem with AD lookups on files. AV was completely "off" although granted I have noticed in past "off" does not mean actually unloaded, sometimes that requires a reboot...

And yet I was able to forkbomb the system relatively slowly, meaning whichever system is ultimately a useless slowdown (as it is doing something un-necessary and unhelpful).

[smaudet]

@Arnavion I didn't have a signing cert, I tried signing (not sure I did it correctly), nevertheless bash shows a large, large number of registry keys being opened. Trying a couple more updates before I dive further....

[ghost]

I too experienced extreme slowness on Win 10. This fixed it for me:

mkpasswd > /etc/passwd mkgroup > /etc/group

Then comment out mentions of db in /etc/nsswitch.conf and restart msys2. Moving at a usable clip now.

[Tatsh]

I too experienced extreme slowness on Win 10. This fixed it for me:

mkpasswd > /etc/passwd mkgroup > /etc/group

Then comment out mentions of db in /etc/nsswitch.conf and restart msys2. Moving at a usable clip now.

Tried this and not sure if it made a difference yet. Need to run performance tests for comparison. I think most people could use this fix and it could be applied to the MSYS2 installer if it really does make a difference.

[orSolocate]

@Arnavion I don't have a C:\msys64\usr\bin path, I tried to run the SignTool at C:\SysGCC\mingw64\bin but got this error: SignTool Error: No certificates were found that met all the given criteria.

gci -re *.exe | %{ &'C:\Program Files (x86)\Windows Kits\8.1\bin\x64\signtool.exe' sign /a /uw $_.FullName }

I run your command on powershell on Windows 10. What's not right in the command here?

[eudoxos]

Tried this and not sure if it made a difference yet. Need to run performance tests for comparison. I think most people could use this fix and it could be applied to the MSYS2 installer if it really does make a difference.

For me, it was also necessary (besides modifying nssswitch.conf and adding /etc/passwd and /etc/group to switch to the offline (local) account in Windows 11 (had online because of joining the insider's program to test WSLg; not needed anymore). Still quite slow, but about 5x faster (no exaggeration) than before.