search

radian-software / prescient.el

☄️ Simple but effective sorting and filtering for Emacs.
MIT License
603 stars 25 forks source link

company-prescient performance in sh-mode #144

Closed ville-h closed 1 year ago

ville-h commented 1 year ago

I'm using emacs 28.2 with latest company and prescient packages from melpa on linux on a fairly-alright machine with 16 logical cores and 16g of memory and an ssd.

Whilst editing bash scripts getting completions can take considerable time. By my incredibly-accurate count-in-my-head method if I've company-prescient enabled in sh-mode emacs locks up for ~15 seconds trying to figure out the completions. If I don't have company-prescient enabled and just plain company it takes ~3 seconds. Calling completion-at-point directly is in the "near-instantaneous" region of maybe <0.5 seconds.

Is this the expected level of performance? What can I do to improve the performance?

okamsn commented 1 year ago

I think that is not the expected level of performance. I will try it on my machine.

Something othe things that might be helpful:

The profiler is documented here: https://www.gnu.org/software/emacs/manual/html_node/elisp/Profiling.html

ville-h commented 1 year ago

What settings do you use for Company for Bash?

I've no bash-specific settings, but these are the relevant company settings I presume. Rest are face attributes and keyboard shortcuts.

    (setq company-idle-delay 0)
    (setq company-minimum-prefix-length 1)
    (setq company-tooltip-limit 16)
    (setq company-tooltip-align-annotations t)
    (setq company-tooltip-annotation-padding 4)

What Company backends and CAPFs are you using for that mode?

The company-backends variable is:

(company-bbdb company-semantic company-cmake company-capf company-clang company-files
    (company-dabbrev-code company-gtags company-etags company-keywords)
    company-oddmuse company-dabbrev)

for sh-mode. Presumably the default.

Please share a stack trace from M-x toggle-debug-on-quit

Debugger entered--Lisp error: (quit)
  locate-file-completion-table(("/home/ville/bin" "/usr/local/bin" "/usr/bin" "/bin" "/usr/local/sbin" "/usr/bin/site_perl" "/usr/bin/vendor_perl" "/usr/bin/core_perl" "/usr/lib/emacs/28.2/x86_64-pc-linux-gnu") ("") #("r" 0 1 (fontified t)) nil t)
  sh--cmd-completion-table(#("r" 0 1 (fontified t)) nil metadata)
  completion-metadata(#("r" 0 1 (fontified t)) sh--cmd-completion-table nil)
  company-capf--annotation(#("iproxy" 0 2 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 2 3 (face (prescient-primary-highlight) prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 3 6 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|..."))))
  company-capf(annotation #("iproxy" 0 2 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 2 3 (face (prescient-primary-highlight) prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 3 6 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|..."))))
  apply(company-capf (annotation #("iproxy" 0 2 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 2 3 (face (prescient-primary-highlight) prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 3 6 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")))))
  company-call-backend-raw(annotation #("iproxy" 0 2 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 2 3 (face (prescient-primary-highlight) prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 3 6 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|..."))))
  company--force-sync(company-call-backend-raw (annotation #("iproxy" 0 2 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 2 3 (face (prescient-primary-highlight) prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 3 6 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")))) company-capf)
  company-call-backend(annotation #("iproxy" 0 2 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 2 3 (face (prescient-primary-highlight) prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|...")) 3 6 (prescient-ignore-case t prescient-regexps ("\\(?:r[̧̣̱́̇̌̏̑]\\|[rŕŗřȑȓʳᵣṙṛṝṟⓡｒ𝐫𝑟𝒓𝓇𝓻𝔯𝕣𝖗𝗋𝗿𝘳𝙧𝚛]\\)\\|..."))))
  company--strip-duplicates((#("*Rescan*" 0 1 ... 1 2 ... 2 8 ...) #("CreateDOMDocument" 0 1 ... 1 2 ... 2 17 ...) #("DOMPrint" 0 4 ... 4 5 ... 5 8 ...) #("DRAWEXE" 0 1 ... 1 2 ... 2 7 ...) #("DRAWEXE-7.6.3" 0 1 ... 1 2 ... 2 13 ...) #("FreeCAD" 0 1 ... 1 2 ... 2 7 ...) #("FreeCADCmd" 0 1 ... 1 2 ... 2 10 ...) #("GraphicsMagick++-con..." 0 1 ... 1 2 ... 2 23 ...) #("GraphicsMagick-confi..." 0 1 ... 1 2 ... 2 21 ...) #("GraphicsMagickWand-c..." 0 1 ... 1 2 ... 2 25 ...) #("JxrDecApp" 0 2 ... 2 3 ... 3 9 ...) #("JxrEncApp" 0 2 ... 2 3 ... 3 9 ...) #("MemParse" 0 5 ... 5 6 ... 6 8 ...) #("NetworkManager" 0 5 ... 5 6 ... 6 14 ...) #("PParse" 0 3 ... 3 4 ... 4 6 ...) #("PSVIWriter" 0 5 ... 5 6 ... 6 10 ...) #("Redirect" 0 1 ... 1 8 ...) #("SAX2Print" 0 5 ... 5 6 ... 6 9 ...) #("SAXPrint" 0 4 ... 4 5 ... 5 8 ...) #("SCMPrint" 0 4 ... 4 5 ... 5 8 ...) ...))
  company--preprocess-candidates((#("*Rescan*" 0 1 ... 1 2 ... 2 8 ...) #("CreateDOMDocument" 0 1 ... 1 2 ... 2 17 ...) #("DOMPrint" 0 4 ... 4 5 ... 5 8 ...) #("DRAWEXE" 0 1 ... 1 2 ... 2 7 ...) #("DRAWEXE-7.6.3" 0 1 ... 1 2 ... 2 13 ...) #("FreeCAD" 0 1 ... 1 2 ... 2 7 ...) #("FreeCADCmd" 0 1 ... 1 2 ... 2 10 ...) #("GraphicsMagick++-con..." 0 1 ... 1 2 ... 2 23 ...) #("GraphicsMagick-confi..." 0 1 ... 1 2 ... 2 21 ...) #("GraphicsMagickWand-c..." 0 1 ... 1 2 ... 2 25 ...) #("JxrDecApp" 0 2 ... 2 3 ... 3 9 ...) #("JxrEncApp" 0 2 ... 2 3 ... 3 9 ...) #("MemParse" 0 5 ... 5 6 ... 6 8 ...) #("NetworkManager" 0 5 ... 5 6 ... 6 14 ...) #("PParse" 0 3 ... 3 4 ... 4 6 ...) #("PSVIWriter" 0 5 ... 5 6 ... 6 10 ...) #("Redirect" 0 1 ... 1 8 ...) #("SAX2Print" 0 5 ... 5 6 ... 6 9 ...) #("SAXPrint" 0 4 ... 4 5 ... 5 8 ...) #("SCMPrint" 0 4 ... 4 5 ... 5 8 ...) ...))
  company-calculate-candidates("r" nil)
  company--begin-new()
  company--perform()
  company-auto-begin()
  company-idle-begin(#<buffer test.sh> #<window 3 on test.sh> 173 608)
  apply(company-idle-begin (#<buffer test.sh> #<window 3 on test.sh> 173 608))
  timer-event-handler([t 25674 34662 132256 nil company-idle-begin (#<buffer test.sh> #<window 3 on test.sh> 173 608) nil 34000 nil])

Maybe a profiler report would let use see where effort is being spent

  3,098,603,115  99% - timer-event-handler
  3,098,602,987  99%  - apply
  3,097,049,656  99%   - company-idle-begin
  3,079,904,815  99%    - company-auto-begin
  3,079,904,815  99%     - company--perform
  3,079,904,815  99%      - company--begin-new
  3,079,892,335  99%       - company-calculate-candidates
  3,070,047,081  98%        - company--preprocess-candidates
  3,070,047,081  98%         - company--strip-duplicates
  3,070,046,025  98%          - company-call-backend
  3,070,046,025  98%           - company--force-sync
  3,070,046,025  98%            - company-call-backend-raw
  3,070,046,025  98%             - apply
  3,070,046,025  98%              - company-capf
  3,069,827,625  98%               - company-capf--annotation
  3,069,827,625  98%                - completion-metadata
  3,069,827,625  98%                 - sh--cmd-completion-table
     11,969,313   0%                  - locate-file-completion-table
      2,389,134   0%                     stringp
        220,528   0%                     regexp-opt
         16,896   0%                   - completion-table-with-context
         16,896   0%                      complete-with-action
      1,820,248   0%                    sh--vars-before-point
        925,048   0%                    mapcar
      9,844,966   0%        - company--fetch-candidates
      9,844,966   0%         - company-call-backend-raw
      9,844,966   0%          - apply
      9,844,966   0%           - company-capf
      9,844,966   0%            - company-capf--candidates
      8,803,422   0%             - completion-all-completions
      8,803,422   0%              - completion--nth-completion
      8,803,422   0%               - completion--some
      8,803,422   0%                - #<compiled -0x745670ce0aee6a2>
      8,803,422   0%                 - prescient-all-completions
      3,973,292   0%                  - prescient-filter
      1,823,427   0%                   - sh--cmd-completion-table
        453,843   0%                    - locate-file-completion-table
          1,799   0%                       stringp
      1,628,092   0%                   - prescient--prefix-and-pattern
      1,628,092   0%                    - completion-boundaries
      1,628,092   0%                     - sh--cmd-completion-table
        704,828   0%                      - locate-file-completion-table
          1,504   0%                         stringp
          1,048   0%                        mapcar
         21,085   0%                   - prescient-filter-regexps
          9,773   0%                    - #<compiled -0x1741186dd95ce6dc>
          9,773   0%                     - mapcar
          9,773   0%                      - #<compiled -0xccf593833472b5>
          6,320   0%                       - prescient-literal-regexp
          6,320   0%                          prescient--char-fold-to-regexp
          3,072   0%                         prescient-regexp-regexp
            381   0%                       - prescient-initials-regexp
            381   0%                          #<compiled -0x10e9484a9e557ddf>
          8,240   0%                      prescient-split-query
      3,650,394   0%                  - prescient--highlight-matches
      3,639,339   0%                   - #<compiled -0x13fba3c728c8ea3b>
      1,978,248   0%                      font-lock-prepend-text-property
          4,098   0%                      string-match
          2,049   0%                      match-data
         11,055   0%                   - prescient-filter-regexps
          9,999   0%                    - #<compiled -0x17c7df4c195ce6dc>
            759   0%                     - mapcar
            759   0%                      - #<compiled -0x177fd33433472b5>
            759   0%                       - prescient-initials-regexp
            759   0%                          #<compiled -0x10e28a73f2557ddf>
          1,056   0%                      prescient-split-query
      1,058,152   0%                  - prescient--prefix-and-pattern
      1,058,152   0%                   - completion-boundaries
      1,058,152   0%                    - sh--cmd-completion-table
          9,504   0%                     - locate-file-completion-table
          8,448   0%                      - completion-table-with-context
          8,448   0%                         complete-with-action
          3,088   0%                       mapcar
      1,041,544   0%             - completion-metadata
      1,041,544   0%                sh--cmd-completion-table
            288   0%        - run-with-timer
            288   0%         - run-at-time
             72   0%            timer-set-time
             72   0%          - timer-activate
             72   0%             timer--activate
         12,480   0%       - company-call-backend
         12,480   0%        - company--force-sync
         12,480   0%         - company-call-backend-raw
         12,480   0%          - apply
         12,480   0%           - company-capf
         12,480   0%            - company--capf-data
         12,480   0%             - company--capf-data-real
         12,480   0%              - company--capf-wrapper
         12,480   0%               - completion--capf-wrapper
         12,480   0%                - sh-completion-at-point-function
         12,480   0%                 - sh-smie--keyword-p
         12,480   0%                  - sh-smie-sh-backward-token
          9,408   0%                   - sh-smie--newline-semi-p
          3,264   0%                      sh-smie-sh-backward-token
          3,072   0%                     sh-smie--looking-back-at-continuation-p
     17,144,841   0%    - company-post-command
     17,131,721   0%     - company-call-frontends
     17,131,193   0%      - company-pseudo-tooltip-unless-just-one-frontend
     17,131,193   0%       - company-pseudo-tooltip-frontend
     17,109,353   0%        - company-pseudo-tooltip-show-at-point
     16,948,979   0%         - company-pseudo-tooltip-show
     16,403,227   0%          - company--create-lines
     16,307,875   0%           - company-call-backend
     16,307,875   0%            - company--force-sync
     16,307,875   0%             - company-call-backend-raw
     16,307,875   0%              - apply
     16,307,875   0%               - company-capf
     16,306,827   0%                - company-capf--annotation
     16,306,827   0%                 - completion-metadata
     16,306,827   0%                  - sh--cmd-completion-table
         21,387   0%                   - locate-file-completion-table
         10,158   0%                      stringp
          1,048   0%                      regexp-opt
          5,240   0%                     mapcar
          2,096   0%                     sh--vars-before-point
         72,040   0%           - company-fill-propertize
         26,920   0%              company-safe-substring
         11,328   0%           - company-detect-icons-margin
         10,280   0%              company-vscode-dark-icons-margin
          8,696   0%           - company--clean-string
          4,192   0%              replace-regexp-in-string
        276,340   0%          - company--replacement-string
        255,028   0%           - company-modify-line
         77,278   0%              company-safe-substring
         11,032   0%             version<=
        102,668   0%          - company--pseudo-tooltip-height
        102,668   0%           - company--row
         66,560   0%            - company--col-row
         66,560   0%             - eval
         32,736   0%              + telephone-line-add-subseparators
         19,808   0%              + telephone-line-fill
         14,016   0%              + telephone-line-separator-render
         66,560   0%          - eval
         27,992   0%           + telephone-line-fill
         24,552   0%           + telephone-line-add-subseparators
         14,016   0%           + telephone-line-separator-render
         12,304   0%            company-plainify
         81,246   0%         - company--col-row
         81,246   0%          - eval
         44,432   0%           + telephone-line-add-subseparators
         22,798   0%           + telephone-line-fill
         14,016   0%           + telephone-line-separator-render
         21,840   0%          company-pseudo-tooltip-guard
            528   0%      - company-echo-metadata-frontend
            528   0%       - company-echo-show-soon
            528   0%        - run-with-timer
            528   0%         - run-at-time
            264   0%          - timer-activate
            264   0%             timer--activate
             72   0%            timer-set-time
         13,120   0%     - company--perform
         13,120   0%      - company--continue
         13,120   0%       - company-call-backend
         13,120   0%        - company--force-sync
         13,120   0%         - company-call-backend-raw
         13,120   0%          - apply
         13,120   0%           - company-capf
         13,120   0%            - company--capf-data
         13,120   0%             - company--capf-data-real
         13,120   0%              - company--capf-wrapper
         13,120   0%               - completion--capf-wrapper
         13,120   0%                - sh-completion-at-point-function
         13,120   0%                 - sh-smie--keyword-p
         13,120   0%                  - sh-smie-sh-backward-token
         10,048   0%                   - sh-smie--newline-semi-p
          3,904   0%                      sh-smie-sh-backward-token
          3,072   0%                     sh-smie--looking-back-at-continuation-p
      1,496,883   0%   - savehist-autosave
      1,494,762   0%      savehist-save
         56,448   0%   - show-paren-function
         56,448   0%    - #<compiled 0x1653d917d32a632c>
         56,448   0%     - apply
         56,448   0%      - smie--matching-block-data
         56,448   0%       - smie--opener/closer-at-point
         17,368   0%        - sh-smie-sh-backward-token
          9,216   0%         - sh-smie--newline-semi-p
          3,072   0%            sh-smie-sh-backward-token
          4,120   0%           sh-smie--looking-back-at-continuation-p
         14,712   0%        - sh-smie-sh-forward-token
          8,184   0%         - sh-smie--newline-semi-p
          8,184   0%            sh-smie-sh-backward-token
             80   0%    timer-inc-time
             24   0%  - timer-activate
             24   0%     timer--activate
     10,521,649   0% - command-execute
      6,858,616   0%  - byte-code
      6,858,616   0%   - read-extended-command
      6,858,616   0%    + completing-read-default
      3,663,033   0%  + funcall-interactively
        687,081   0% + redisplay_internal (C function)
         18,741   0% + jit-lock--antiblink-post-command
            264   0% + company-post-command
              0   0%   ...
okamsn commented 1 year ago
ville-h commented 1 year ago

If you remove sh-completion-at-point-function from completion-at-point-functions

How do I do that? I am not a lisp programmer.

Would you please provide an example

Create a buffer such as test.sh with the contents: 

#!/bin/bash

# at the beginning of the line below type letter 'r'

Different letters appear to take varying amounts of time, but none are particularly quick.

okamsn commented 1 year ago

Different letters appear to take varying amounts of time, but none are particularly quick.

For me, on Emacs 29 with native compilation enabled, the Company popup window shows in under a second. This is on an older laptop, so I think that it should be the typical performance.

How do I do that? I am not a lisp programmer.

After opening the file, paste 

(setq-local completion-at-point-functions (remove #'sh-completion-at-point-function completion-at-point-functions))

into it. Then, go to after the last parenthesis and press C-x C-e (eval-last-sexp).

If you do this

For me, sh-completion-at-point-function is the only CAPF that provides candidates in your example, so I don't see any completions once I remove it.

Also, are you using Prescient in the minibuffer, such as with Vertico Prescient? Some CAPFs will filter using completion styles.

ville-h commented 1 year ago

Are you using native compilation in your version of Emacs?

Yes it's the emacs-nativecomp arch package version 28.2 from the arch repos.

Do you have a lot of candidates? For the letter r

Based on executing completion-at-point after the letter r, selecting the region from line after "Possible completions are:" to the end of the buffer, using count-words gives me 375 lines and 4 candidates per line so 1500. So more than 70 but 1500 is nothing for a computer.

does the delay still occur?

No.

do you get any candidates?

Yes.

Also, are you using Prescient in the minibuffer, such as with Vertico Prescient?

Yes I am also using vertico-prescient. However if I comment out the use-package for it the performance is still the same. I don't know if that is sufficient to disable it or does some package try to be "helpful" and load it anyways if it can be found on the disk?

okamsn commented 1 year ago

I'm not sure, but I think that this performance issue was fixed on Emacs 29, if it's the same problem you're having: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59678

There, the reporter says

I found that the auto-completion on .sh files (with company-mode) was pretty slow (takes several seconds). It turns out that company-mode would call the capf function with 'metadata action for each candidate, where each function call would need to list all the cmds. In this patch, I modified the code to use completion-table-with-cache which fixes the problem for me.

So, if you try these newer definitions of the functions (and if these are the only things that changed), does the slowdown still occur?

(defun sh--cmd-completion-table-gen (string)
  (append (when (fboundp 'imenu--make-index-alist)
            (mapcar #'car
                    (condition-case nil
                        (imenu--make-index-alist)
                      (imenu-unavailable nil))))
          (mapcar (lambda (v) (concat v "="))
                  (sh--vars-before-point))
          (locate-file-completion-table
           exec-path exec-suffixes string nil t)
          sh--completion-keywords))

(defun sh-completion-at-point-function ()
  (save-excursion
    (skip-chars-forward "[:alnum:]_")
    (let ((end (point))
          (_ (skip-chars-backward "[:alnum:]_"))
          (start (point)))
      (cond
       ((eq (char-before) ?$)
        (list start end (sh--vars-before-point)
              :company-kind (lambda (_) 'variable)))
       ((sh-smie--keyword-p)
        (list start end
              (completion-table-with-cache #'sh--cmd-completion-table-gen)
              :company-kind
              (lambda (s)
                (cond
                 ((member s sh--completion-keywords) 'keyword)
                 ((string-suffix-p "=" s) 'variable)
                 (t 'function)))))))))
ville-h commented 1 year ago

So, if you try these newer definitions of the functions

I don't know how I'm supposed to try them. Do I paste them in my init.el? If so then there appears to be no difference.

okamsn commented 1 year ago

So, if you try these newer definitions of the functions

I don't know how I'm supposed to try them. Do I paste them in my init.el?

Yes. You might need to wrap them in (with-eval-after-load 'sh-mode ...).

If so then there appears to be no difference.

Darn. I was hoping that would be it. What if, after loading Company Prescient, you overwrite company-prescient-transformer to disable Prescient's sorting.

(with-eval-after-load 'company-prescient
  (defun company-prescient-transformer (candidates)
    "Overwritten `company-prescient-transformer'."
    (identity candidates)))

Also, what is the value of completion-styles in your SH Mode buffer? You can check using M-: completion-styles RET, or by using C-x C-e after typing the symbol in the buffer.

ville-h commented 1 year ago

Yes. You might need to wrap them in (with-eval-after-load 'sh-mode ...)

No improvement.

overwrite company-prescient-transformer

No improvement.

what is the value of completion-styles 

Its value is (prescient)
okamsn commented 1 year ago

Are you modifying completion-styles manually?

Company Prescient shouldn't modify it. It only does sorting. Vertico Prescient should only modify it in the Vertico buffer or minibuffer.

ville-h commented 1 year ago

Are you modifying completion-styles manually?

Yes.

okamsn commented 1 year ago

So, we already use Emacs's C functions for filtering, so I'm not sure if we can speed that part up further.

One contributing factor to the slowdown might be that I made the completion style unconditionally add information to all candidates for prescient. It seemed fine in my case (~30,000 candidates for describe-function), but might be slow for you.

If you disable that behavior using the below code, does it make an improvement?

(with-eval-after-load 'prescient
  (defun prescient--add-sort-info (cands _ _)
    cands))
ville-h commented 1 year ago

If you disable that behavior using the below code, does it make an improvement?

Can't say it makes a difference one way or the other.

okamsn commented 1 year ago

You said that calling completion-at-point does not have the slowdown. I'm not sure why that would be, if it's the same code that the produces the candidates.

Would you please try the following command using M-x after installing the Cape package? 

(defun my-interactive-sh-capf ()
  (interactive)
  (cape-interactive #'sh-completion-at-point-function))
ville-h commented 1 year ago

If you run this command with Company mode enabled, does the slowdown occur?

No.

Does it use the Company pop-up?

No.

If you run this command after disabling Company mode, does the slowdown occur?

No.

okamsn commented 1 year ago

So, this seems to be the situation:

@dgutov, do you have any information that you could share about CAPFs and completion styles slowing down Company?

I'm also pinging @minad and @raxod502, in case they have any ideas. I am all out of ideas at this point.

ville-h commented 1 year ago

I timed the performance with a stopwatch and what I counted to 15 is really ~29 seconds. If I set (setq gc-cons-threshold most-positive-fixnum), which maybe could take out some of the gc overhead out of the test, the result was ~14 seconds.

dgutov commented 1 year ago

FWIW, it's the same as https://debbugs.gnu.org/59678, and Emacs 29 shouldn't have this problem anymore.

okamsn commented 1 year ago

@ville-h, can you try installing Emacs 29 (including the native compilation enabled for good measure)?

ville-h commented 1 year ago

I tried, but the arch package for emacs-native-comp-git appears to be broken at the moment.

okamsn commented 1 year ago

@ville-h, have you had a chance to try installing Emacs 29.1 with native compilation?

ville-h commented 1 year ago

Yes, arch has updated their packages and so far 29.1 native compilation package seems to not exhibit the slow down. The popup isn't instant, but it's in the "interactive" region of probably 0.5-1.0 seconds. Which is fast enough.

okamsn commented 1 year ago

OK. Closing this issue.