search

sublimehq / sublime_merge

Issue tracker for Sublime Merge
https://www.sublimemerge.com
276 stars 14 forks source link

Poor performance with a big project on Windows #83

Open PurplePowder opened 6 years ago

PurplePowder commented 6 years ago

Version info

Description

I'm working with a really large project (chromium fork) and current state in sublime merge is updated for a very long time. For example if I change file (or stage it, even using sublime merge interface), sublime merge takes a minute or two to discover this.

Steps to reproduce

Steps to reproduce the behavior:

  1. Open chromium repository
  2. Wait 1-2 minutes until it says "Loading changes"
  3. Change any file or add new one
  4. Wait 1-2 minutes until it find out this change
  5. Click "Stage" for a change from 3
  6. Wait 1-2 minutes until change became staged

Expected behavior

I'd expect that current changes will be discovered instantly.

jskinner commented 6 years ago

How long does git status take to run from the command line for you on this setup?

PurplePowder commented 6 years ago

How long does git status take to run from the command line for you on this setup?

About 2 seconds. I'm using git for windows, v2.19.1

jskinner commented 6 years ago

The Chromium repository on Windows for me takes 2-6s to load changes, approximately the same time as running git status from the command line.

Performance issues on Windows when doing file IO are frequently down to virus scanners.

PurplePowder commented 6 years ago

The Chromium repository on Windows for me takes 2-6s to load changes, approximately the same time as running git status from the command line.

I can collect other repo statistics, if you need.

Performance issues on Windows when doing file IO are frequently down to virus scanners.

But virus scanners should affect other applications too, even command line git client. I tried Atlassian SourceTree and it takes a couple of seconds to discover changes.

I also found out some strange behavior. When you stage some file in Sublime Merge, it's dissapear from interface, but according to CL git status this file will be unstaged for a minute or two.

ratcashdev commented 6 years ago

I had a similar experience (but not THAT bad) when opening the NetBeans project - the pack with Oracle's 2nd donation. The first commit is basically over 114 thousand files. When clicking on the initial commit, sublime-merge freezes for approx 10-15 sec until the changes are loaded and UI elements populated.

Setup: The ".git" folder is 1GB. The working folder more-less the same. Disk is SSD (~500MB/sec). Sublime-merge RAM consumption ~1GB. CPU utilization (while loading changes) 100% on a single core. The rest of the cores were idle.

Maybe paralellizing this task (if possible) could help.

Anyway, quite an achievement for sublime, given that SourceTree (my current daily helper) does it in 50 secs, same RAM, same CPU. And same UI freeze.

jskinner commented 6 years ago

@ratcashdev You're describing something quite different, it would be best to create a separate issue for it.

The original post is about what seems to be very slow file IO. Your issue is about a UI performance issue on very large commits.

jskinner commented 6 years ago

@PurplePowder virus scanners absolutely treat different applications differently, so I don't think you can draw conclusions that just because git.exe isn't having its file IO intercepted, sublime_merge.exe won't. I'm not saying that the issue you're seeing is certainly caused by a virus scanner, but that is where I'd put my money.

With regards to the files disappearing as soon as you hit Stage, that's by design: we predict some in-progress modifications to the Git repository, so that you can stage files quickly, irrespective for how long the underlying action is taking. If the modification fails (e.g., perhaps the index file has been made read only), then you'll see the file come back when the operation has finished.

PurplePowder commented 6 years ago

@jskinner I made a short research. Please take a look at this screenshot https://yadi.sk/i/Cu9gh0aFLOkvvw. This is File Summary of procmon capturing SourceTree (at the top) and Sublime Merge (at the bottom). As you can see, File Time for Sublime Merge is twice smaller, so this is not virus scanners impact.

I also noticed that SourceTree spawns git status and git.exe uses multiple treads, but Sublime Merge uses only one thread https://yadi.sk/i/HikVotK9hGEoXA. I guess this is the reason of my performance issue

jskinner commented 6 years ago

Git does use multiple threads, but in general only gains a modest speedup from doing so. You can test this fairly easily by running time git -c core.preloadindex=false status, and comparing that to the time a regular git status takes.

You should be seeing roughly the same performance from Sublime Merge and git when preloadindex is set to false.

I expect things would be somewhat different if you're using a repository over a networked file system though: any parallelism in the IO would be a significant help given the added latency.

However, you're seeing 100x worse performance from Sublime Merge, which is hard to explain with multi threading alone. git.exe creates a maximum of 20 threads (see preload-index.c) so even in the networked file system case you should be seeing performance no worse than 20x.

PurplePowder commented 6 years ago

Guess you right. git -c core.preloadindex=false status takes ~35 seconds, while regular git status takes only 2-3 seconds. Hovewer I measured Sublime Merge performance more accurately and it takes ~3.5 minutes to process all files when I click "Refresh".

I don't use repository over a networked file system. I keep my repository on a SSD (Intel 530 series)

jskinner commented 6 years ago

FWIW, with the extra data I've gotten from digging into this, it's very likely that multi-threaded fs walking will be coming to Sublime Merge, and while I'm sure that would be a help in your scenario, I don't think it'd be a significant one - there does seem to be something else going on that's causing really slow file IO for Sublime Merge for you.

PurplePowder commented 6 years ago

Sounds great! Thank you very much for the information, I'll wait for this feature.

Please tell me if you need any debug info of my problem.

tangmi commented 4 years ago

Performance issues on Windows when doing file IO are frequently down to virus scanners.

In case anyone is still running into this: I was recently observing the same slowdown as @PurplePowder while also working in a chromium fork on Windows. I added sublime_merge.exe to the Windows Defender exclusion list and (after restarting Sublime Merge just to be extra sure) things seem to be running significantly faster.

marcinsmialek commented 3 years ago

On my work machine, the wait time is really bad for big repos.

The raw git status executed from command line used to take around 14s without untracked exclusion, so it made sense. Excluding untracked files when querying status was fast from command line.

With git 2.32 and core.usebuiltinfsmonitor = true, the command line status is fast, but Sublime Merge still takes ages to refresh even when pointing it to git 2.32 executable.

dpjohnst commented 2 months ago

Hi all,

We've just released Sublime Merge build 2100 on the development channel which should speed things up considerably.

Thanks, - Dylan from Sublime HQ