search

rkitover / vimpager

Use Vim as PAGER
http://www.vim.org/scripts/script.php?script_id=1723
Other
772 stars 72 forks source link

vimpager and vimcat freeze, no output (OS X, homebrew) #222

Open braham-snyder opened 6 years ago

braham-snyder commented 6 years ago

On OS X El Capitan, with homebrew vim, using vimpager also installed through homebrew, with and without --HEAD, from bash, in both Terminal.app and iTerm2:

bsmbp:~ bs$ brew info vimpager
vimpager: stable 2.06 (bottled), HEAD
Use ViM as PAGER
https://github.com/rkitover/vimpager
/usr/local/Cellar/vimpager/HEAD-4f1a077 (24 files, 399.9KB) *
  Built from source on 2017-10-03 at 20:34:12
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/vimpager.rb
==> Dependencies
Build: pandoc ✘
Optional: pandoc ✘
==> Options
--with-pandoc
    Use pandoc to build and install man pages
--HEAD
    Install HEAD version
==> Caveats
To use vimpager as your default pager, add `export PAGER=vimpager` to your
shell configuration.

bsmbp:~ bs$ cat ~/.vimpagerrc
set clipboard=unnamed

bsmbp:~ bs$ vim --version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Sep  1 2017 18:55:49)
MacOS X (unix) version
Included patches: 1-997
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl             +file_in_path    +mouse_sgr       +tag_old_static
+arabic          +find_in_path    -mouse_sysmouse  -tag_any_white
+autocmd         +float           +mouse_urxvt     -tcl
-balloon_eval    +folding         +mouse_xterm     +termguicolors
-browse          -footer          +multi_byte      +terminal
++builtin_terms  +fork()          +multi_lang      +terminfo
+byte_offset     -gettext         -mzscheme        +termresponse
+channel         -hangul_input    +netbeans_intg   +textobjects
+cindent         +iconv           +num64           +timers
+clientserver    +insert_expand   +packages        +title
+clipboard       +job             +path_extra      -toolbar
+cmdline_compl   +jumplist        +perl            +user_commands
+cmdline_hist    +keymap          +persistent_undo +vertsplit
+cmdline_info    +lambda          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con      -lua             +rightleft       +windows
+diff            +menu            +ruby            +writebackup
+digraphs        +mksession       +scrollbind      +X11
-dnd             +modify_fname    +signs           +xfontset
-ebcdic          +mouse           +smartindent     -xim
+emacs_tags      -mouseshape      +startuptime     -xpm
+eval            +mouse_dec       +statusline      +xsmp_interact
+ex_extra        -mouse_gpm       -sun_workshop    +xterm_clipboard
+extra_search    -mouse_jsbterm   +syntax          -xterm_save
+farsi           +mouse_netterm   +tag_binary
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X_UNIX  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang   -L. -fstack-protector -L/usr/local/lib -L/usr/local/opt/libyaml/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib  -L/usr/local/lib -o vim    -lXt -lX11 -lSM -lICE  -lncurses -liconv -framework Cocoa   -mmacosx-version-min=10.11 -fstack-protector-strong -L/usr/local/lib  -L/usr/local/Cellar/perl/5.26.0/lib/perl5/5.26.0/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc -F/usr/local/opt/python/Frameworks -framework Python   -lruby.2.4.1 -lobjc

bsmbp:~ bs$ vimpager -x ~/.vimpagerrc
+ [ 1 -gt 0 ]
+ break
+ find_tmp_directory
+ mkdir_options='-m 700'
+ [ -n '' ]
+ tmp=/tmp
+ tmp=/tmp/vimpager_81182
+ mkdir -m 700 /tmp/vimpager_81182
+ trap 'quit 1' PIPE HUP INT QUIT ILL TRAP KILL BUS TERM
+ detect_term_size
+ command -v tput
+ 1> /dev/null
+ tput cols
+ 1> /tmp/vimpager_81182/cols
+ tput lines
+ 1> /tmp/vimpager_81182/lines
+ cat /tmp/vimpager_81182/cols
+ cols=117
+ cat /tmp/vimpager_81182/lines
+ lines=32
+ rm -f -- /tmp/vimpager_81182/cols /tmp/vimpager_81182/lines
+ [ -z 117 ]
+ [ -z 117 ]
+ expand_config_vars
+ eval runtime='"/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager"'
+ runtime=/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager
+ eval vimcat='"/usr/local/Cellar/vimpager/HEAD-4f1a077/bin/vimcat"'
+ vimcat=/usr/local/Cellar/vimpager/HEAD-4f1a077/bin/vimcat
+ eval system_vimpagerrc='"/usr/local/Cellar/vimpager/HEAD-4f1a077/etc/vimpagerrc"'
+ system_vimpagerrc=/usr/local/Cellar/vimpager/HEAD-4f1a077/etc/vimpagerrc
+ find_vim_executable
+ [ -n '' ]
+ tvim=vim
+ [ -n vim -a -z '' ]
+ VIMPAGER_VIM=vim
+ export VIMPAGER_VIM
+ [ -n vim ]
+ tvim=vim
+ [ -n '' ]
+ vim_cmd=vim
+ [ ! -n '' -a -n '' ]
+ find_vimpagerrc_files
+ i=1
+ OLDIFS=$' \t\n'
+ IFS=$'\n'
+ IFS=$' \t\n'
+ vim -NEnR -i NONE '+call writefile(["", "VAL:" . $VIM, "VAL:" . $MYVIMRC], "/dev/stderr")' +q
+ 0< /dev/tty 2>& 1

At which point output stops, and all keys I press are simply appended to the output. Well, other than ^Z or ^\ -- the latter of which then shows (probably unremarkably):

+ quit 1
+ rm -f gvim.exe.stackdump
+ cd /tmp
+ 2> /dev/null
+ rm -rf /tmp/vimpager_98242
+ 2> /dev/null
+ exit 1

And same for vimcat, with this debugging output:

bsmbp:~ bs$ vimcat -x ~/.vimpagerrc
+ tmp_dir=/tmp
+ mkdir_options='-m 700'
+ uname -s
+ tmp_dir=/tmp/vimcat_90447
+ mkdir -m 700 /tmp/vimcat_90447
+ trap 'quit 1' PIPE HUP INT QUIT ILL TRAP KILL BUS TERM
+ tmp_file_in=/tmp/vimcat_90447/vimcat_in.txt
+ out_fifo=/tmp/vimcat_90447/vimcat_out.fifo
+ uname -s
+ mkfifo /tmp/vimcat_90447/vimcat_out.fifo
+ [ 2 -gt 0 ]
+ shift
+ [ 1 -gt 0 ]
+ break
+ [ -z '' ]
+ [ ! -t 1 ]
+ [ -z '' ]
+ [ -f /Users/bs/.vimcatrc ]
+ vimcatrc=''
+ [ 1 -eq 0 ]
+ [ -n '' -a 1 -gt 1 ]
+ chunks_dir=/tmp/vimcat_90447/chunks
+ mkdir /tmp/vimcat_90447/chunks
+ i=1
+ [ 1 -ge 2 ]
+ pipeline=''
+ pipeline_start=/Users/bs/.vimpagerrc
+ [ 0 -eq 1 ]
+ exit_code=0
+ [ /Users/bs/.vimpagerrc '!=' - ]
+ [ ! -r /Users/bs/.vimpagerrc ]
+ [ ! -s /Users/bs/.vimpagerrc ]
+ [ -z '' -o '' '=' - ]
+ dest_file=/tmp/vimcat_90447/vimcat_out.fifo
+ tail_pid=90453
+ start_highlight_job
+ set -- -NE -i NONE -n --cmd 'set runtimepath^=/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager' --cmd $'call vimcat#Init({ \'rc\': \'\' })' --cmd visual -c 'call vimcat#Run("/tmp/vimcat_90447/vimcat_out.fifo", 0, "/tmp/vimcat_90447/chunks", "/Users/bs/.vimpagerrc")'
+ [ -n '' ]
+ [ 0 -eq 0 ]
+ vim_pid=90454
+ start_pipeline
+ [ -n '' ]
+ pipeline=write_chunks
+ tail -f /tmp/vimcat_90447/vimcat_out.fifo
+ vim -NE -i NONE -n --cmd 'set runtimepath^=/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager' --cmd $'call vimcat#Init({ \'rc\': \'\' })' --cmd visual -c 'call vimcat#Run("/tmp/vimcat_90447/vimcat_out.fifo", 0, "/tmp/vimcat_90447/chunks", "/Users/bs/.vimpagerrc")'
+ 3<& 0
+ cat -- /Users/bs/.vimpagerrc
+ 0< /dev/tty 1> /dev/null 2>& 1
+ echo 90458
+ eval write_chunks
+ 1> /tmp/vimcat_90447/pipeline_pid
+ 0<& 3
+ write_chunks
+ cd /tmp/vimcat_90447/chunks
+ rm -f -- '*'
+ cat /tmp/vimcat_90447/pipeline_pid
+ split -b 4096 -
+ pipeline_pid=90458
+ [ ! -f /tmp/vimcat_90447/vim_done ]
+ do_sleep 50
+ _ms=50
+ vim -NEsnR -i NONE -u NONE '+sleep 50 m' +q
+ 1> /dev/null 2>& 1
+ touch PIPELINE_DONE

Thanks for vimpager!

ghost commented 6 years ago

I was experimenting with colorizing programs and found this ticket, and felt really sorry for you. So I'll help you out.

  1. Vim actually already has a built-in "vimpager" script, called less.sh. It's installed in a "macros" folder somewhere on your system. Find it with sudo find -x / -name less.sh. Although it isn't very good...

  2. Either way, the best pager is not vim. Not by a long shot. Vim is slow as hell as a pager (takes 2 seconds to start up before anything is shown). The best pager is less. But not by default. It needs some options to make it usable. Then you'll get full pagination, UTF-8, color highlighting, line numbers, text search via / just like in vim, and you can even edit files with vim directly from less by pressing v (for "visual editor"). And you press q to leave less.

Here's how to setup less properly to make it your pager and get syntax highlighted pagination of man-pages etc:

# COLORIZE THE OUTPUT OF "LESS" TO MAKE MANPAGES MORE READABLE
export LESS_TERMCAP_mb=$'\E[01;31m';
export LESS_TERMCAP_md=$'\E[01;31m';
export LESS_TERMCAP_me=$'\E[0m';
export LESS_TERMCAP_se=$'\E[0m';
export LESS_TERMCAP_so=$'\E[01;44;33m';
export LESS_TERMCAP_ue=$'\E[0m';
export LESS_TERMCAP_us=$'\E[01;32m';

# TWEAKING OTHER "LESS" PARAMETERS
export LESSCHARSET='utf-8';
export LESS='-i -N -w -z-4 -g -M -R -P%t?f%f :stdin .?pb%pb\%:?lbLine %lb:?bbByte %bb:-...';

# SETTING "LESS" AS DEFAULT PAGER AND REPLACING ALL OTHER PAGERS WITH IT
export PAGER=less;
alias more=$PAGER;
alias zless=$PAGER;

# SET UP "VIM" AS DEFAULT EDITOR
export VISUAL=vim;
export EDITOR=$VISUAL;

You need to put that in your ~/.bash_profile or ~/.bashrc.

Now all your man git and stuff will load with less in beautiful syntax highlighting, colorized mode. And it's super fast. Unlike vim/vimpager which takes 1-2 seconds to start, less is actually instant.

  1. You wanted vimcat to colorize code, right? Well, it's truly garbage. I'm sorry... I'm really sorry. I understand the work involved in making the vimscript that powers it. But it's garbage due to how bad vim's scripting language/limitations are.

For a 1600 line file, I ran vimcat <that file> and it took about 2 minutes to print all lines, line by line... as if I was running on a 300 baud modem in the 1980s. If I tried Ctrl-C to abort that tedious process, it hung and stopped accepting keyboard input and had to be killed via ps aux | grep vim and killing the process... (Probably because Ctrl-C just killed the "tail" pipe program but not actually vim? No idea...)

Why was it so slow? Because vimcat uses a temporary file named /tmp/vimcat_####/vimcat.fifo which is read by the shell 1 line at a time until it is empty.

Tedious...

  1. The best syntax highlighter by far is actually a Python program named Pygments. It supports over 300 languages and supports outputting the syntax highlighting to all kinds of targets such as HTML, Terminal (BINGO!), etc.

To install it, do the following:

First install Homebrew.

Then run these commands to install Python 2 and Python 3:

brew install python3
brew install python
brew unlink python && brew link --overwrite python # make sure python links to python2

pip2 install --upgrade pip setuptools # upgrade to latest pip2
pip3 install --upgrade pip setuptools # upgrade to latest pip3

# never use the generic “python” and “pip”.
# always use “python[2/3]” and “pip[2/3]”.

Then, to install the Python3-version of Pygments, I just ran:

pip3 install pygments

# to upgrade later:
pip3 install pygments --upgrade

Lastly, make an alias for running Pygmentize with output as colorized 256-color terminal, etc:

alias pcat='pygmentize -f terminal256 -O style=monokai -g'

Now just run pcat main.cpp or pcat index.php or whatever other language.

Enjoy the perfection on your screen! It syntax highlights the 1600 line file in ~1 second (pygmentize) compared to ~120 seconds (vimcat). And supports 300 programming languages.

You can change the style (theme). I use monokai, but others are listed here: pygmentize -L styles. (Another good one is native).

If you need to output to other formats (like BBCode for forums, etc), then just run this command to look at all of its available output formats. :-) pygmentize -L formatters.

Have fun.

Here's an example of the pcat output:

monokai

braham-snyder commented 6 years ago

I appreciate the thought, but I'm painfully aware of less, and I'm only incidentally interested in vimcat's behavior.

I'm looking for something more powerful. More precisely: I want to be able to search and select text in my pager as well as I can in emacs or vim (so -- ideally -- I'd like access to easymotion, sneak/snipe, swiper, evil-little-word, etc.).

Meanwhile, less can't even wrap its searches (which alone kills me) -- nor is its search incremental.

One use-case I'm particularly interested in is better terminal scrollback: I'd like to test https://github.com/kovidgoyal/kitty with my scrollback_pager set to vimpager (though I'd surely need to limit the number of lines sent to vimpager to keep startup times reasonable).

You need to put that in your ~/.bash_profile or ~/.bashrc

You might want those non-bash-specific exports in ~/.profile instead (which you can then source from ~/.bash_profile if the latter also exists): https://unix.stackexchange.com/a/45687/193985

Vim is slow as hell as a pager (takes 2 seconds to start up before anything is shown).

I see sub-one-second startup times with this, which I greatly prefer to less (though I also need to try emacs's woman):

export MANPAGER="/usr/local/bin/nvim -c 'set ft=man' -"

Which is what led me to want to try vimpager as well.

You wanted vimcat to colorize code, right?

No, actually -- I only included the vimcat -x output above to hopefully help the author (or help anyone) diagnose why I cannot get vimpager working.

For a 1600 line file, I ran vimcat and it took about 2 minutes to print all lines, line by line... as if I was running on a 300 baud modem in the 1980s. If I tried Ctrl-C to abort that tedious process, it hung and stopped accepting keyboard input and had to be killed via ps aux | grep vim and killing the process... (Probably because Ctrl-C just killed the "tail" pipe program but not actually vim? No idea...)

Why was it so slow? Because vimcat uses a temporary file named /tmp/vimcat_####/vimcat.fifo which is read by the shell 1 line at a time until it is empty.

While I would not at all be surprised if vimcat were relatively slow, a two-minute runtime for a 1600 line file must surely be a separate bug? Have you tried with a blank ~/.vimrc? (Though -- again -- I'm primarily interested in vimpager.)

edit: there's always tmux for scrollback paging and selection, but its vim-like keybindings are very limited

ghost commented 6 years ago

Haha, alright good luck with everything! :-) By the way, you can search incrementally forwards and backwards in less: /word, n, n, n, shift-N, shift-N.

And with my tweaks you get search result-highlighting etc. Works perfectly for my needs. But you've got other needs and vimpager seems more up your alley. Good luck.

braham-snyder commented 6 years ago

Haha, alright good luck with everything! :-)

Thank you!

By the way, you can search incrementally

I mean a different kind of incremental search (it's unfortunately quite ambiguously-named -- I might start calling it "real-time search" or "as-you-type-matching" or "incrementally-updated search" or something): https://en.wikipedia.org/wiki/Incremental_search

ghost commented 6 years ago

Ohh that kind of search. Sure that's nice. :)

ghost commented 6 years ago

Hmm just as a warning to you:

Remember the vimcat command I described taking 2 minutes, and which I hit Ctrl-C to abort, which froze vim and the terminal and required me to kill that process manually?

Uhm... well, that exact failed process has somehow continued running in the background since yesterday and is taking 60% CPU:

x

I know it's that process, because under the Open Files and Ports tab of Activity Monitor, it lists the folder of the file I was vimcating... so it belongs to that (long-ago closed) terminal.

I discovered this now because my fans are going nuts, and I wondered why my CPU was so hot... and why my battery drained in like 40 minutes.

Strange stuff...

Time to reboot my computer.

Edit: Rebooted and all is well and my battery estimate is back to 2-3 hours. So yeah turns out vimcat can really badly kill bash on macOS Sierra. I mention it because it may be related to the crashing you mentioned in this ticket.

wimstefan commented 6 years ago

Did you ever find a solution for your problem? I'm running against the same wall at the moment ;-)

braham-snyder commented 6 years ago

Regarding the vimpager freeze? Nope, but I haven't toyed with it further.
Or $PAGER more generally? I decided to settle -- for now at least -- with tmux's set-window-option -g mode-keys vi + set -g status-keys emacs (the vi status-keys are quite odd) + some minor modifications to make it slightly more vim-like (since, e.g., for some reason tmux only defaults the emacs-style search to incremental, and yanking does not cancel the visual selection by default).

All of the Emacs solutions print even medium-length outputs too slowly, due to the time it takes Emacs to deal with ANSI-coloring (which I suspect would be difficult to speed up).

https://github.com/neovim/neovim/issues/5054 could be quite relevant (with kitty for scrollback paging without re-running commands), but I'm not quite sure of the scope (is the idea to add ANSI-coloring capability to nvim itself?) -- plus it's "unplanned" at the moment.

lucc commented 6 years ago

@braham-snyder that issue is about the "internal pager" of (n)vim that is used for long output from vim commands (see :h more-prompt).

rkitover commented 6 years ago

I'm sorry, I haven't worked on this repo in a while because of various things in life, and am currently deeply engrossed in another project.

I'll try to take a break from the other project soon and take a look at stuff here.

ghost commented 6 years ago

@rkitover Relax and have fun with the other project. ;-) You're not under any obligation to do this. Have fun and come back when you feel like it.

wimstefan commented 6 years ago

@rkitover Take your time! My question was more out of curiosity since the error I'm getting is not exactly the same as described here and I was in doubt whether to create a separate issue or not ... no real life has caught me again and I'll have to see when I can dive into it a bit more ;-) @braham-snyder thanks for your quick response! I'll see how I can tackle the problem ...

wimstefan commented 6 years ago

@rkitover Sorry for reporting so late due to real life capture but my issue was fixed with 2.06-347-gf03db2e ... many thanks for that! 👏