Still quite sluggish?

[tigger04]

I really like ble.sh sho far, but my main complaint would be - it makes typing very sluggish. Sometimes I'll hit a keybinding for backwards-kill-word for example, and it freezes up for 2-3 seconds before allowing me to type again.

My HISTSIZE is I think on the large size (10,000) which I"m thinking might be a factor. How do other users find the performance?

(I'm running this on a brand new 2020 MacBook Pro 15" that my employer was kind enough to surprise me with recently, so I really don't think it's the machine. Oh, and running Bash 5.0.18 (down with zsh!)

[akinomyoga]

Thank you for the report! I use Linux and Windows and basically there are no performance issues for me. I'm sorry, but I don't have macOS, so I cannot directly test it with macOS, but it usually depends on the Bash settings largely. One possible cause is programmable completion setting. To pin down the problem, can you try complete -r and see if the behavior changes?

$ complete -r

Note: complete -r removes all the programmable completion settings.

[tigger04]

Oh my, that makes a huge difference. It's much snappier now.

I guess those look like the pre-existing completions I had configured (I think it was the scop/bash-completion package or something similar - of course bash's great weakness has always been inability to dynamically define these), and presumably ble.sh does not rely on them?

Seeing some other curious behaviour around prompt and spacing. I have a bit of complexity in my setup though, I'll raise another issue.

Nice work though, this may keep me in bash which I am reluctant to leave but frustrated with it's amnesia of the last 25 years of software and CLI progress :)

cheers

[beaugunderson]

I'm also on macOS; this is my .blerc, I turned off history completion because it is massive and ble.sh is still very slow:

bleopt complete_auto_history=

I also tried complete -r and see no difference in speed.

bash 5.0.17(1)-release + macOS 10.15.6.

[akinomyoga]

Oh my, that makes a huge difference. It's much snappier now.

@tigger04 Thank you! In my environments with bash-completion, the delay is still negligible. I guess some completion setting for specific commands causes the problem.

Which command do you see the delay with (when you don't disable the programmable completion)? For example, do you see the same delay in the arguments of echo command?

I'm going to take a look at the programmable completion settings this weekend. I think I will ask for your help after that again. The problem here is that the programmable completion is originally designed for TAB completion of the original Bash, so some of the user completion settings can contain computationally heavy calculations. ble.sh utilizes the user completion settings for both the TAB completion and auto-completion, and some settings can cause too much delay for auto-completion.

Seeing some other curious behaviour around prompt and spacing. I have a bit of complexity in my setup though, I'll raise another issue.

Thank you. I'm looking forward to seeing another issue!

---

I'm also on macOS; this is my .blerc, I turned off history completion because it is massive and ble.sh is still very slow:

I also tried complete -r and see no difference in speed.

@beaugunderson Thank you for the information! It seems your performance issue is different from the one reported by @tigger04. If you could help me with identifying your problem, please create another issue.

[akinomyoga]

@tigger04 I don't have macOS, so I can only guess the problem. My guess is that the delay is caused by complete -F setting or complete -C setting. I need to confirm it. Can you try the following settings?

Q1: When you set shopt -u progcomp instead of complete -r, how is the delay? Is it sluggish?

Q2: I would like to collect the timing information in the internal processing. To collect the information, can you try the following steps?

1. Add the following code in your .bashrc just after source ble.sh ...? To setup ble.sh in .bashrc, please follow the instruction in README.

# Add this code after "source ble.sh ..."

function debug/complete-load-hook {
  filename_debug_log=debug.txt
  function timestamp-args.advice {
    echo "$EPOCHREALTIME ${ADVICE_WORDS[*]}" >> "$filename_debug_log"
  }
  ble/function#advice \
    before ble-decode/.hook \
    timestamp-args.advice

  function timestamp-wrap.advice {
    echo "$EPOCHREALTIME ${ADVICE_WORDS[0]} start" >> "$filename_debug_log"
    ble/function#advice/do
    echo "$EPOCHREALTIME ${ADVICE_WORDS[0]} end" >> "$filename_debug_log"
  }
  ble/function#advice \
    around ble/complete/progcomp/.compgen-helper-prog \
    timestamp-wrap.advice
  ble/function#advice \
    around ble/complete/progcomp/.compgen-helper-func \
    timestamp-wrap.advice
}
blehook/eval-after-load complete debug/complete-load-hook

2. Start a new Bash session and type something (that experiences the delay).
3. End the Bash session. A file debug.txt should be created in the working directory of the session. Now you can remove the above code in .bashrc.

Then, can you attach the created file debug.txt here? To attach the file you can drag & drop the file to the textbox.

[tigger04]

Thanks so much akinomyoga for all of your effort in debugging this, and apologies I have not been able to get back to you sooner.

So when I saw the original response around running complete -r and that it seemed to make things snappier for me, I made some adjustments to my environment for when it's running in 'ble.sh' mode (and additionally to my bash prompt which was experiencing issues here, of which the other issue I mentioened and I will raise today with a video clip to show it a bit more clearly).

Right now, I am embarrassed to say that I don't think I can recreate the issue as originally described. I have reset my .bashrc to what it was at the time, but either 1) I have not recreated the environment faithfully to recreate the issue exactly, or 2) my initial impression was that it was a little sluggish but I have got used to it and my expectations have changed, or 3) it may have been affected by some other circumstance e.g. say if my company's anti-virus scan decided to start running in the background burning up CPU cycles (it does this occasionally with no indication that it is running unless you forensically inspect the processes ... or observe the fans going crazy)

In any case I'll add the debug statements above to my setup for a few days and see if it does recur, maybe the stats will be useful in the end.

[tigger04]

Q1: When you set shopt -u progcomp instead of complete -r, how is the delay? Is it sluggish?

I think it is noticeably slower, but given my previous comment, it is difficult to quantify - maybe the debug stats will help answer it.

[akinomyoga]

and additionally to my bash prompt which was experiencing issues here, of which the other issue I mentioened and I will raise today with a video clip to show it a bit more clearly

Oh, that's nice! I'm looking forward to it.

I don't think I can recreate the issue as originally described. I have reset my .bashrc to what it was at the time, but either [...]

OK, that's fine! That kind of debug troubles often occur.

In any case I'll add the debug statements above to my setup for a few days and see if it does recur, maybe the stats will be useful in the end.

The above debug codes print information into the current directory debug.txt, so if you change the directory, debug.txt will be created in each directory. If you run it for a few days, please adjust the second line so that the output file debug.txt is specified in an absolute path. Here is an example that saves the information in /path/to/debug.txt. Please replace the /path/to to your preferred place.

-  filename_debug_log=debug.txt
+  filename_debug_log=/path/to/debug.txt 

I think it is noticeably slower, but given my previous comment, it is difficult to quantify - maybe the debug stats will help answer it.

OK, let's look at the debug stats.

[tigger04]

great, script modified.

On Tue, Aug 4, 2020 at 2:47 PM Koichi Murase notifications@github.com wrote:

and additionally to my bash prompt which was experiencing issues here, of which the other issue I mentioened and I will raise today with a video clip to show it a bit more clearly

Oh, that's nice! I'm looking forward to it.

I don't think I can recreate the issue as originally described. I have reset my .bashrc to what it was at the time, but either [...]

OK, that's fine! That kind of debug troubles often occur.

In any case I'll add the debug statements above to my setup for a few days and see if it does recur, maybe the stats will be useful in the end.

The above debug codes print information into the current directory debug.txt, so if you change the directory, debug.txt will be created in each directory. If you run it for a few days, please adjust the second line so that the output file debug.txt is specified in an absolute path. Here is an example that saves the information in /path/to/debug.txt. Please replace the /path/to to your preferred place.

• filename_debug_log=debug.txt+ filename_debug_log=/path/to/debug.txt

I think it is noticeably slower, but given my previous comment, it is difficult to quantify - maybe the debug stats will help answer it.

OK, let's look at the debug stats.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/akinomyoga/ble.sh/issues/58#issuecomment-668606626, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKL7RSXR43FQ3DQOXUTRKPDR7AGQRANCNFSM4PJN4HYQ .

[akinomyoga]

In bashrc you provided at https://github.com/akinomyoga/ble.sh/issues/60#issuecomment-669926432, I noticed that you have actually both the debug/complete-load-hook setting and complete -r as follows.

    # Add this code after "source ble.sh ..."

    function debug/complete-load-hook {
    filename_debug_log=~/bin/scratch/ble-debug

# ...

    }
    blehook/eval-after-load complete debug/complete-load-hook

# ...

# Add this line at the end of .bashrc:
$_ble_is_enabled && ((_ble_bash)) && ble-attach && complete -r

I'm sorry, but my description above was not enough. I wanted the timing information when the user completion settings are enabled, i.e., complete -r is NOT performed.

[tigger04]

In bashrc you provided at #60 (comment), I noticed that you have actually both the debug/complete-load-hook setting and complete -r as follows.

    # Add this code after "source ble.sh ..."

    function debug/complete-load-hook {
    filename_debug_log=~/bin/scratch/ble-debug

# ...

    }
    blehook/eval-after-load complete debug/complete-load-hook

# ...

# Add this line at the end of .bashrc:
$_ble_is_enabled && ((_ble_bash)) && ble-attach && complete -r

I'm sorry, but my description above was not enough. I wanted the timing information when the user completion settings are enabled, i.e., complete -r is NOT performed.

I was actually going to give you both - I ran it for a day with complete -r in the code, and another day with this statement absent. I'll send you both .bashrc's

[akinomyoga]

I was actually going to give you both - I ran it for a day with complete -r in the code, and another day with this statement absent. I'll send you both .bashrc's

Oh, nice. Thank you!

[tigger04]

What is the best way of sending you these debug files?

[akinomyoga]

Can you attach these files here? You can drag and drop the file to the textarea.

Edit: The file extension needs to be .txt.

See the next comment.

[akinomyoga]

Sorry, forget about the previous reply. It is better to send it to my email address myoga.murase@gmail.com

[akinomyoga]

Thank you for the email! I have received the log file. I'm going to look in detail later.

[akinomyoga]

Hi, sorry for keeping you waiting for a long time. I have checked the results you have sent me, but it seems the programmable completion doesn't have performance problems. You have mentioned in https://github.com/akinomyoga/ble.sh/issues/58#issuecomment-668532259 that you cannot reproduce the original problem now.

May I ask whether you experienced the same problem after that or whether the problem didn't appear at all?

Today I fixed a performance bug (0770cda) where the prompt PS1 is evaluated for every key stroke. Actually this performance bug is unaffected by the setting complete -r, so I'm not sure if it is related to this problem. But if you have changed some prompt settings at the same time when you tried complete -r, that might explain the current situation where the original problem is not reproduced.

[tigger04]

Hi Koichi

Thanks so much for your response and for taking the time to look in to this.

I had been meaning to follow up with you as I have experienced it a few times since, thankfully it is not a common occurrence. It does seem that at the time when I've observed it, at the same time the fan on my computer was going full speed, the CPU temperature was high, and there was a process consuming most of the CPU cycles (usually AV but also I had some issue with a VS code plugin that was going haywire). So the cause of the issue is not ble.sh, but the effect, since I guess ble.sh is doing a lot more work to power the suggestions and autocompletion, is that it makes it difficult to type, as the cursor and suggestions are not able to keep up with my typing speed. The solution each time was to - either kill the offending process(es), or switch of blesh until the stress on my system subsided.

I tried to record this, but (in hindsight, should have been predictable), while the system was under stress it was not able to start a screen recording in a timely manner. And attempting to record this on my phone was just not possible as I need both hands to type at normal speed to demonstrate the effect.

If i get another chance to record or demonstrate this I will certainly send to you.

In the meantime, I don't know whether you want to consider it a performance but or a potential optimization, but I guess the key thing would be (if you can reproduce it) to allow for the system to degrade in performance without preventing the user from continuing to input text in to the command line, which I was still able to do (with ble.sh disabled) in plain bash with the system under stress as described.

I hope that's useful, or at least helps to clear up the root cause.

It's a great product, I really like what you have done with it, and I look forward to getting under the hood and playing around with it a bit more when I get some more time.

cheers

Tadhg

On Sat, Aug 22, 2020 at 8:59 AM Koichi Murase notifications@github.com wrote:

Hi, sorry for keeping you wait for a long time. I have checked the results you have sent me, but it seems the programmable completion doesn't have performance problems. You have mentioned in #58 (comment) https://github.com/akinomyoga/ble.sh/issues/58#issuecomment-668532259 that you cannot reproduce the original problem now.

May I ask whether you experienced the same problem after that or whether the problem didn't appear at all?

Today I fixed a performance bug (0770cda https://github.com/akinomyoga/ble.sh/commit/0770cdae67dbf13ba2aba89e92e564b5ee68c278) where the prompt PS1 is evaluated for every key stroke. Actually this performance bug is unaffected by the setting complete -r, so I'm not sure if it is related to this problem. But if you have changed some prompt settings at the same time when you tried complete -r, that might explain the current situation where the original problem is not reproduced.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/akinomyoga/ble.sh/issues/58#issuecomment-678610782, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKL7RSTOK4PHWSPQMVYYQC3SB53GJANCNFSM4PJN4HYQ .

[akinomyoga]

Hi Tadhg, thank you for your reply!

It does seem that at the time when I've observed it, [...] as the cursor and suggestions are not able to keep up with my typing speed.

Now it becomes more clear what happens there! Thank you for your detailed report!

The solution each time was to - either kill the offending process(es), or switch of blesh until the stress on my system subsided.

It is undocumented, but there are commands to switch on/off ble.sh within the shell session. You can turn off ble.sh by ble-detach and turn on ble.sh by ble-attach. I think they are useful in the above case.

$ ble-detach
[ble: detached]
Please run `stty sane' to recover the correct TTY state.
$ stty sane;
$
$ echo 'do something (without ble.sh)'
$
$ ble-attach

In the meantime, I don't know whether you want to consider it a performance but or a potential optimization, but I guess the key thing would be (if you can reproduce it) to allow for the system to degrade in performance without preventing the user from continuing to input text in to the command line, which I was still able to do (with ble.sh disabled) in plain bash with the system under stress as described.

Thank you for a nice suggestion! Actually ble.sh has already some mechanisms not to block user inputs; It tries to cancel auto-completion when unprocessed user inputs are detected. But sometimes the user-input detection itself can be blocked by other reasons when the disk is busy or ble.sh runs external programs. Another possible bottleneck might be the syntax highlighting based on the file type, which accesses the disk to expand the path and check the existence and type of file.

For further improvements, auto-complete and some part of syntax-highlighting may be automatically turned off when the system is busy. But I guess the appropriate criterion that the system is considered busy depends. For example, you can configure to automatically switch auto-complete by the following setting (where you need to replace is-system-busy with your preferred conditional command).

# blerc
function blerc/adjust-performance-settings {
  if is-system-busy; then
    bleopt complete_auto_complete=
  else
    bleopt complete_auto_complete=1
  fi
}
blehook PRECMD+=blerc/adjust-performance-settings

blehook PRECMD is a hook which will be invoked before evaluating PS1. bleopt complete_auto_complete is an option to enable/disable auto-complete.

Currently there is no option to (partially nor totally) turn off the syntax-highlighting, but I can easily add it if you are interested in this approach. [ Edit: There was a request for the option to turn of syntax-highlighting at #61, so I implemented. You can put bleopt highlight_syntax= / bleopt highlight_syntax=1 to enable/disable syntax-highlighting. ]

Cheers, Koichi

[aiotter]

Hi, I have a problem with some commands which blocks 3min~ at the first time. While it is blocking, any type of the user inputs including Ctrl-C have no effect and buffered so that it takes effect after blocking is released. I know this is a problem of programmatic autocompletion. After ble-detach the problem disappears. The blocking command is flake-related nix commands which need heavy computation on completion. The example is nix run nixpkgs#hello, which freezes when I type #. Can I help you solve the problem? I'm using macOS, but I don't know if this is specific to the OS.

[akinomyoga]

@aiotter Thank you for your report! I'll reply later, but the reply may delay (just as I explained in another thread). In short, this is one of the compatibility problems hard to cleanly solve. Currently, I patch the completions command by command, so I think I need the actual completion configuration by nix.