Status buffer performance with borg

[jojojames]

Oh and you will notice that refreshing the status buffer gets rather slow if there are many modules. I started working on that today and discovered some low-hanging fruit. So this should be much less of a problem in a few days.

Originally posted by @tarsius in https://github.com/emacscollective/borg/issues/12#issuecomment-318936162

I was looking for more info on this but this seems to be sparse. @tarsius, do you just deal with this while using borg or is there some tricks you're using to get performance up. With around 100 submodules, refreshing the buffer is noticeably slower but still manageable.

[tarsius]

When I don't have to do anything with the submodules, then I keep the Modules section collapsed and everything is snappy. When I do have to deal with submodules, then I just accept the slowdown.

[jojojames]

Hmnn, I looked deeper into this since you've said things should still be snappy.

  before borg migration

  magit-insert-error-header                          3e-06
  magit-insert-diff-filter-header                    0.009832
  magit-insert-head-branch-header                    0.011721 !
  magit-insert-upstream-branch-header                7e-06
  magit-insert-push-branch-header                    4e-06
  magit-insert-tags-header                           0.012647 !
  magit-insert-status-headers                        0.04159 !!
  magit-insert-merge-log                             0.004533
  magit-insert-rebase-sequence                       0.000233
  magit-insert-am-sequence                           0.000106
  magit-insert-sequencer-sequence                    0.000203
  magit-insert-bisect-output                         0.000106
  magit-insert-bisect-rest                           2.6e-05
  magit-insert-bisect-log                            2.5e-05
  magit-insert-untracked-files                       0.126452 !!
  magit-insert-unstaged-changes                      0.124834 !!
  magit-insert-staged-changes                        0.012754 !
  magit-insert-stashes                               0.004464
  magit-insert-unpushed-to-pushremote                1.1e-05
  magit-insert-unpushed-to-upstream-or-recent        0.028529 !
  magit-insert-unpulled-from-pushremote              3e-06
  magit-insert-unpulled-from-upstream                1e-06

  after borg migration

  magit-insert-error-header                          2e-06
  magit-insert-diff-filter-header                    0.0091
  magit-insert-head-branch-header                    0.01059 !
  magit-insert-upstream-branch-header                0.010068 !
  magit-insert-push-branch-header                    0.004419
  magit-insert-tags-header                           0.012116 !
  magit-insert-status-headers                        0.051897 !!
  magit-insert-merge-log                             0.004361
  magit-insert-rebase-sequence                       0.000186
  magit-insert-am-sequence                           8.1e-05
  magit-insert-sequencer-sequence                    0.000176
  magit-insert-bisect-output                         8.1e-05
  magit-insert-bisect-rest                           1.8e-05
  magit-insert-bisect-log                            1.9e-05
  magit-insert-untracked-files                       1.05173 !!
  magit-insert-unstaged-changes                      1.063564 !!
  magit-insert-staged-changes                        0.009534
  magit-insert-stashes                               0.004324
  magit-insert-modules                               0.006176
  magit-insert-unpushed-to-pushremote                6.2e-05
  magit-insert-unpushed-to-upstream-or-recent        0.032899 !!
  magit-insert-unpulled-from-pushremote              5e-06
  magit-insert-unpulled-from-upstream                0.008047

It looks like you're right, that inserting modules is pretty fast but for some reason, the untracked/unstaged sections have grown up to 1 second each from .1 seconds.

Any tips or advice here appreciated, I took a look at those two functions but it wasn't immediately obvious where to begin looking or how to profile those two.

[tarsius]

I don't know what "borg migration" entails. For example did you use submodules before too?

Anyway, I don't think borg is responsible for that. Maybe the use of submodules contributes to it. I generally recommend against inserting untracked files as that tends to be slow unless you make sure there usually aren't any untracked files.

I don't have time right now to help users with things like this and kindly ask you to figure it out ourself. (If you come up with some actionable advice, then please let me know though.)

[jojojames]

I don't have time right now to help users with things like this and kindly ask you to figure it out ourself. (If you come up with some actionable advice, then please let me know though.)

That's reasonable, I tracked it down to this:

https://github.com/magit/magit/commit/a7699f868f6fd32aae8aa8e64a72ea989a5c25cf

Looks like, magit-insert-untracked-files passes a flag returned by 'magit-ignore-submodules-p to the eventual git shell command it makes, but by default, there are no git settings set by the user to set up the submodule-ignore which results in a slow experience after bootstrap.

e.g. It is running this command for me:

$ git status -z --porcelain #slow

vs

git status -z --porcelain --ignore-submodules=all #fast

I am guessing you already have some kind of submodule-ignore setting set in your own repos, whether global or local to those repos, that allows you to avoid seeing this problem.

[tarsius]

I don't think I have. I just make sure that there usually aren't many untracked files and unstaged changes, laying around. If there is an untracked file, then I either track it or I gitignore it. Performance isn't great but quite acceptable:

  magit-insert-untracked-files                       0.013418741 !
  magit-insert-unstaged-changes                      0.25856443 !!

But yeah, setting the variable that is mentioned in that commit should also help (but of course your would lose information).

[jojojames]

Yeah, setting up the submodule config cleared everything up for me.

My repository had 0 untracked/unstaged files so the 1 second to resolve the files was strange to me.

Now it looks like this:

  magit-insert-error-header                          3e-06 
  magit-insert-diff-filter-header                    0.009667 
  magit-insert-head-branch-header                    0.010138 !
  magit-insert-upstream-branch-header                0.010126 !
  magit-insert-push-branch-header                    0.004609 
  magit-insert-tags-header                           0.012483 !
  magit-insert-status-headers                        0.052818 !!
  magit-insert-merge-log                             0.004689 
  magit-insert-rebase-sequence                       0.000218 
  magit-insert-am-sequence                           0.0001 
  magit-insert-sequencer-sequence                    0.00019 
  magit-insert-bisect-output                         0.0001 
  magit-insert-bisect-rest                           2.4e-05 
  magit-insert-bisect-log                            2.3e-05 
  magit-insert-untracked-files                       0.041522 !!
  magit-insert-unstaged-changes                      0.037503 !!
  magit-insert-staged-changes                        0.010198 !
  magit-insert-stashes                               0.004354 
  magit-insert-modules-overview                      0.005184 
  magit-insert-modules-unpulled-from-upstream        3.3e-05 
  magit-insert-modules-unpulled-from-pushremote      1.3e-05 
  magit-insert-modules-unpushed-to-upstream          1.1e-05 
  magit-insert-modules-unpushed-to-pushremote        1e-05 
  magit-insert-modules                               0.010912 !
  magit-insert-unpushed-to-pushremote                4.2e-05 
  magit-insert-unpushed-to-upstream-or-recent        0.034282 !!
  magit-insert-unpulled-from-pushremote              3e-06 
  magit-insert-unpulled-from-upstream                0.008295 

For completeness sake and for future readers:

I found something like this:

$ git config --add diff.ignoreSubmodules dirty

fixed the issue for me but doesn't persist across repos when cloning so tweaked the Makefile's bootstrap.

.DEFAULT_GOAL := default

default: bootstrap-borg
    @ echo Calling git submodule update --init --recursive...
    @ git submodule update --init --recursive
    @ echo Calling git config --add diff.ignoreSubmodules dirty...
    @ git config --add diff.ignoreSubmodules dirty
    @ echo Call make bootstrap to finish bootstrap.

DRONES_DIR = $(shell git config "borg.drones-directory" || echo "lib")

-include $(DRONES_DIR)/borg/borg.mk

bootstrap-borg:
        @git submodule--helper clone --name borg --path $(DRONES_DIR)/borg \
        --url git@github.com:emacscollective/borg.git
        @cd $(DRONES_DIR)/borg; git symbolic-ref HEAD refs/heads/master
        @cd $(DRONES_DIR)/borg; git reset --hard HEAD

.PHONY: default