`UnicodeDecodeError` when using `vim-peekaboo`

Hey there, I'm encountering the following error when peekaboo#peek from the plugin vim-peekaboo

Traceback (most recent call last):                                                                                                                              
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main  
    return _run_code(code, main_globals, None,                                                                                                                  
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code             
    exec(code, run_globals)                                                                                                                                     
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/coq/__main__.py", line 174, in <module>                                                 
    arun(main())                                                                                                                                                
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncio/runners.py", line 44, in run         
    return loop.run_until_complete(main)                                                                                                                        
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_unt
il_complete                                                                                                                                                     
    return future.result()                                                                                                                                      
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/coq/__main__.py", line 172, in main                                                     
    await init(args.socket)                                                                                                                                     
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/coq/client.py", line 80, in init                                                        
    await set_options(                                                                                                                                          
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 178, in __aexit__       
    await self.gen.__anext__()                                                                                                                                  
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.8/site-packages/pynvim_pp/nvim.py", line 257, in conn         
    yield rpc                                                                                                                                                   
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 178, in __aexit__       
    await self.gen.__anext__()                                                                                                                                  
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.8/site-packages/pynvim_pp/rpc.py", line 252, in client        
    await conn                                                                                                                                                  
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.8/site-packages/pynvim_pp/rpc.py", line 143, in _connect      
    await gather(rx(recv()), send())                                                                                                                            
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.8/site-packages/pynvim_pp/rpc.py", line 188, in rx            
    async for frame in rx:                                                                                                                                      
  File "/Users/shardy/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.8/site-packages/pynvim_pp/rpc.py", line 140, in recv          
    for frame in unpacker:                                                                                                                                      
  File "msgpack/_unpacker.pyx", line 540, in msgpack._cmsgpack.Unpacker.__next__                                                                                
  File "msgpack/_unpacker.pyx", line 463, in msgpack._cmsgpack.Unpacker._unpack                                                                                 
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 8: invalid start byte

Let me know what other information could be useful for debugging!

I couldn't figure out how to copy the contents of the peekaboo buffer that is opened, but here is a screenshot after the exception is cleared from the screen.

Also, after the above error is raised I get the following when I open/close buffers:

Error detected while processing InsertEnter Autocommands for "*":                                                                                                                     
E5108: Error executing lua Invalid channel: 6                                                                                                                                         
stack traceback:                                                                                                                                                                      
        [C]: in function 'Buf_enter'                                                                                                                                                  
        [string ":lua"]:1: in main chunk                                                                                                                                              
Press ENTER or type command to continue                                                                                                                                               
Error detected while processing InsertLeave Autocommands for "*":                                                                                                                     
E5108: Error executing lua Invalid channel: 6                                                                                                                                         
stack traceback:                                                                                                                                                                      
        [C]: in function 'Kill_win'                                                                                                                                                   
        [string ":lua"]:1: in main chunk

This error seems to occur when there is non-UTF8 data in a register.

It's easy to trigger with the clipboard: dd if=/dev/urandom bs=4 count=1 | xclip -i -sel clip. Then in Neovim, open the vim-peekaboo window to trigger the error. The peekaboo window itself is fine, it shows escaped/hex-encoded values for non-printable characters, but apparently COQ chokes on the raw bytes.

This can be worked-around by clearing the clipboard / registers (:call setreg('n', []) in Neovim to clear register n).

Here is a log of the messages sent/received by Coq leading up to the error:

[send]: (1, 7, None, None)
[recv]: [0, 8, 'Buf_enter', []]
[send]: (0, 57, 'nvim_get_current_win', ())
[recv]: [1, 57, None, <pynvim_pp.window.Window object at 0x7fc32c021360>]
[send]: (0, 58, 'nvim_win_get_buf', (<pynvim_pp.window.Window object at 0x7fc32c021360>,))
[recv]: [1, 58, None, <pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>]
[send]: (0, 59, 'nvim_buf_get_option', (<pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>, 'buflisted'))
[recv]: [1, 59, None, True]
[send]: (0, 60, 'nvim_buf_get_option', (<pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>, 'buftype'))
[recv]: [1, 60, None, '']
[send]: (0, 61, 'nvim_buf_attach', (<pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>, False, {}))
[recv]: [2, 'nvim_buf_changedtick_event', [<pynvim_pp.buffer.Buffer object at 0x7fc32c0210f0>, 2]]
[recv]: [1, 61, None, True]
[send]: (0, 62, 'nvim_buf_get_option', (<pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>, 'filetype'))
[recv]: [1, 62, None, '']
[send]: (0, 63, 'nvim_buf_get_name', (<pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>,))
[recv]: [1, 63, None, '']
[send]: (0, 64, 'nvim_win_get_cursor', (<pynvim_pp.window.Window object at 0x7fc32c021360>,))
[recv]: [1, 64, None, [1, 0]]
[send]: (0, 65, 'nvim_win_get_height', (<pynvim_pp.window.Window object at 0x7fc32c021360>,))
[recv]: [1, 65, None, 64]
[send]: (0, 66, 'nvim_buf_line_count', (<pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>,))
[recv]: [1, 66, None, 1]
[send]: (0, 67, 'nvim_buf_get_lines', (<pynvim_pp.buffer.Buffer object at 0x7fc32c021c60>, 0, 1, True))
[recv]: [1, 67, None, ['']]
[send]: (1, 8, None, None)
[recv]: [0, 9, 'Ft_changed', []]
[send]: (0, 68, 'nvim_get_current_buf', ())
[recv]: [1, 68, None, <pynvim_pp.buffer.Buffer object at 0x7fc32c023280>]
[send]: (0, 69, 'nvim_buf_get_option', (<pynvim_pp.buffer.Buffer object at 0x7fc32c023280>, 'filetype'))
[recv]: [1, 69, None, 'peekaboo']
[send]: (0, 70, 'nvim_buf_get_name', (<pynvim_pp.buffer.Buffer object at 0x7fc32c023280>,))
[recv]: [1, 70, None, '']
[send]: (1, 9, None, None)
[recv]: [2, 'nvim_buf_lines_event', [<pynvim_pp.buffer.Buffer object at 0x7fc32c0228c0>, 3, 1, 1, ['Special:'], False]]
[send]: (0, 71, 'nvim_call_atomic', ((('nvim_get_mode', ()), ('nvim_call_function', ('complete_info', (('mode',),)))),))
[recv]: [2, 'nvim_buf_lines_event', [<pynvim_pp.buffer.Buffer object at 0x7fc32c021930>, 4, 2, 2, [' ": #'], False]]
[send]: (0, 72, 'nvim_call_atomic', ((('nvim_get_mode', ()), ('nvim_call_function', ('complete_info', (('mode',),)))),))
[recv]: [2, 'nvim_buf_lines_event', [<pynvim_pp.buffer.Buffer object at 0x7fc32c0220e0>, 5, 3, 3, [' *: \x00'], False]]
[send]: (0, 73, 'nvim_call_atomic', ((('nvim_get_mode', ()), ('nvim_call_function', ('complete_info', (('mode',),)))),))
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/coq/__main__.py", line 173, in <module>
    arun(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/coq/__main__.py", line 171, in main
    await init(args.socket, ppid=args.ppid)
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/coq/client.py", line 70, in init
    async with conn(socket, default=_default) as client:
  File "/usr/lib/python3.10/contextlib.py", line 206, in __aexit__
    await anext(self.gen)
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.10/site-packages/pynvim_pp/nvim.py", line 249, in conn

    async with client(socket, default=default) as rpc:
  File "/usr/lib/python3.10/contextlib.py", line 206, in __aexit__
    await anext(self.gen)
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.10/site-packages/pynvim_pp/rpc.py", line 255, in client
    await conn
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.10/site-packages/pynvim_pp/rpc.py", line 145, in _connect
    await gather(rx(recv()), send())
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.10/site-packages/pynvim_pp/rpc.py", line 190, in rx
    async for frame in rx:
  File "/home/internet/.local/share/nvim/site/pack/packer/start/coq_nvim/.vars/runtime/lib/python3.10/site-packages/pynvim_pp/rpc.py", line 141, in recv
    for frame in unpacker:
  File "msgpack/_unpacker.pyx", line 540, in msgpack._cmsgpack.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 463, in msgpack._cmsgpack.Unpacker._unpack
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcc in position 4: invalid continuation byte

The last received nvim_buf_lines_event messages contain the lines preceding that of the + register, so I expect it's the binary data in that + register line that's indeed causing the exception.

The same data as used to trigger the log above (cc cc cc), but in a regular file buffer (nvim <(xxd -r -p <<<cccccc)), gives the following messages (after I add a c to trigger a nvim_buf_lines_event message):

[send]: (0, 51, 'nvim_buf_get_lines', (<pynvim_pp.buffer.Buffer object at 0x7fa89dee8ee0>, 0, 1, True))
[recv]: [1, 51, None, ['ÌÌÌ']]
[send]: (1, 6, None, None)
[recv]: [2, 'nvim_buf_lines_event', [<pynvim_pp.buffer.Buffer object at 0x7fa89c057070>, 3, 0, 1, ['cÌÌÌ'], False]]

No exception in that case.

Or with 91 92, which show up as <91><92> in a regular buffer (instead of cc which shows up as some weird-ass character, Ì):

[send]: (0, 51, 'nvim_buf_get_lines', (<pynvim_pp.buffer.Buffer object at 0x7f500133fca0>, 0, 1, True))
[recv]: [1, 51, None, ['\x91\x92']]
[send]: (1, 6, None, None)
[recv]: [2, 'nvim_buf_lines_event', [<pynvim_pp.buffer.Buffer object at 0x7f50004bf490>, 3, 0, 1, ['c\x91\x92'], False]]

No exception either.

Ok I can trigger without peekaboo by:

putting binary data into the clipboard: xxd -r -p <<<cccccc | xclip -i -sel clip
writing that data into an empty buffer (with Coq enabled) with eval like peekaboo does it: :put =eval('@+')

I think Neovim probably shouldn't pack buffer lines as strings (since apparently they could contain raw bytes).

But since it currently does, I tried:

using raw=True with the Unpacker object in _pynvimpp, but that breaks other messages that expect actual strings
recovering from the UnicodeDecodeError in _pynvimpp, skipping the broken message, but didn't manage to do so correctly

I'm giving up for now. :(

ms-jpq / coq_nvim

`UnicodeDecodeError` when using `vim-peekaboo` #539