search

wincent / command-t

⌨️ Fast file navigation for Neovim and Vim
BSD 2-Clause "Simplified" License
2.74k stars 317 forks source link

ArgumentError: invalid byte sequence in UTF-8 #359

Closed mgedmin closed 2 years ago

mgedmin commented 5 years ago

I'm trying to work on linkchecker and I get the following error, most likely because there's a file in the test scaffolding with a non-UTF-8 filename:

``` /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!' /usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run' /usr/lib/ruby/2.5.0/open3.rb:95:in `popen3' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches' (eval):3:in `list_matches' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear' (eval):3:in `clear' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!' /usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run' /usr/lib/ruby/2.5.0/open3.rb:95:in `popen3' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches' (eval):3:in `list_matches' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear' (eval):3:in `clear' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!' /usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run' /usr/lib/ruby/2.5.0/open3.rb:95:in `popen3' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches' (eval):3:in `list_matches' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear' (eval):3:in `clear' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder' Error detected while processing function commandt#FileFinder: line 2: ArgumentError: invalid byte sequence in UTF-8 /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!' /usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run' /usr/lib/ruby/2.5.0/open3.rb:95:in `popen3' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches' (eval):3:in `list_matches' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear' (eval):3:in `clear' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show' /home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder' (eval):3:in `show_file_finder' eval:1:in `
' Press ENTER or type command to continue ```

To reproduce:

The bad file is in tests/checker/data/dir/.

I'm fine with Command-T being unable to open or even list this file, but denying me access to the rest of the tree is a bit inconvenient.

wincent commented 5 years ago

Sadly, I can't repro this:

On cloning the repo, the file in question is hidden in a zip file:

Screen Shot 2019-09-10 at 21 13 24

And the unzip tool which comes with macOS refuses to open it. 😢

linkchecker___unzip

Do you know the full path (the raw bytes of it) that has the invalid sequence? I could try to fix this blindly, but I'd like to test it with a real test case before I do.

mgedmin commented 5 years ago

On cloning the repo, the file in question is hidden in a zip file:

Oh, interesting! It must've been extracted while running the test suite. A quick git clean -dfx and I can use command-t again!

So, from what I've heard about MacOS, its filesystem requires all filenames to be valid UTF-8 (and normalizes them into the NFD form, IIRC). I don't think it's possible for you to test it on your local system without using something like a Docker image or a Vagrant VM running Linux.

If you need an exact byte sequence, here's one that is enough to trigger the error in an empty directory:

but, again, I'm on Linux. I would expect the Python command that creates the file to fail on Mac OS.

wincent commented 2 years ago

Given the big rewrite for v6.0.x, I'm closing all older issues as there is unlikely to be anything significant happening on the 5-x-devel branch from here on[^patches]. Feedback issue for 6.0.x is here:

[^patches]: Patches and PRs would be welcome, but my personal efforts are going to be directed towards main.