Unicode characters in file names produce errors when using fzf integration

[abbottc]

ISSUE TYPE

• Bug report

RUNTIME ENVIRONMENT

• Operating system and version: Linux Mint 17.2
• Terminal emulator and version: tilda 1.1.11
• Python version: 2.7.6
• Ranger version/commit: 1.8.1
• Locale: en-us

EXPECTED BEHAVIOR

Integrating FZF with Ranger via this code and fuzzy-finding a file containing a unicode character should select that file in Ranger.

CURRENT BEHAVIOR

Fuzzy-finding a file containing a unicode character produces the following traceback (using the ranger --debug flag):

ranger version: 1.8.1, executed with python 2.7.6
Locale: en_US.UTF-8
Current file: /home/self/Anki
Traceback (most recent call last):
  File "/home/self/ranger-1.8.1/ranger/core/main.py", line 145, in main
    fm.loop()
  File "/home/self/ranger-1.8.1/ranger/core/fm.py", line 368, in loop
    ui.handle_input()
  File "/home/self/ranger-1.8.1/ranger/gui/ui.py", line 230, in handle_input
    self.handle_key(key)
  File "/home/self/ranger-1.8.1/ranger/gui/ui.py", line 166, in handle_key
    self.press(key)
  File "/home/self/ranger-1.8.1/ranger/gui/ui.py", line 181, in press
    quantifier=keybuffer.quantifier)
  File "/home/self/ranger-1.8.1/ranger/core/actions.py", line 221, in execute_console
    cmd_class(string, quantifier=quantifier).execute()
  File "/home/self/.config/ranger/commands.py", line 31, in execute
    self.fm.cd(fzf_file)
  File "/home/self/ranger-1.8.1/ranger/core/actions.py", line 532, in cd
    self.enter_dir(path, remember=remember)
  File "/home/self/ranger-1.8.1/ranger/core/actions.py", line 517, in enter_dir
    result = self.thistab.enter_dir(path, history=history)
  File "/home/self/ranger-1.8.1/ranger/core/tab.py", line 113, in enter_dir
    path = str(path)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xae' in position 68: ordinal not in range(128)

Also, fuzzy-finding a file even when it does not contain unicode characters can emit a Python warning when another item in the same directory contains a unicode character. This appears to be due to this line of code, which winds up comparing a unicode object (returned by fzf) in the variable good against a string object in the variable test. The Python warning is:

UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

POSSIBLE SOLUTIONS

Perhaps it would be more robust to treat paths as unicode objects here and here?

STEPS TO REPRODUCE

1. Create a file with a unicode character in the name (ex: foo®bar)
2. Follow the instructions for fzf integration
3. Open Ranger, and do a fuzzy-find for the file in step 1
4. After selecting the file in fzf and returning back to the Ranger screen, the error (mentioned above) appears at the bottom of the screen and the file is not selected in the Ranger interface.

[ghost]

Thanks for reporting this, as I encountered that problem as well.

Simply using fzf_file = os.path.abspath(stdout.rstrip('\n')) fixed it for me. Perhaps ranger does more Unicode work later on, making the UTF-8 decoding an extraneous precaution from the command's author.

In any case, I haven't had any trouble so far. I use ranger v1.7.1, with Python 2.7.13 on Debian 9.

[AmaruCoder]

helvetie, I am experiencing the same problem. Where did you change the code?

[toonn]

Does this same problem show up with python 3? I put helvetie's fix on the wiki. @AmaruCoder could you test this change?

[ghost]

@toonn Sorry, but my fix was really not a fix — it just swapped the problem.

The .decode method is needed for Python 3, but breaks with Python 2, and vice-versa.

Without touching ranger I was able to do two things:

1. Do as other commands on the wiki do and add the universal_newlines=True keyword to fm.execute_command, which returns the pipe output not as bytes but as str. It will use the system's preferred encoding (it might not always be UTF-8).
2. add a Python 3 check and decode if needed. This lets us choose the encoding to pass to ranger, which might be preferable:

diff --git a/config/ranger/commands.py b/config/ranger/commands.py
index dfea736..60ef166 100644
--- a/config/ranger/commands.py
+++ b/config/ranger/commands.py
@@ -6,6 +6,7 @@ class fzf_select(Command):
     def execute(self):
         import subprocess
         import os.path
+        import sys
         if self.quantifier:
             # match only directories
             command="find -L . \( -path '*/\.*' -o -fstype 'dev' -o -fstype 'proc' \) -prune \
@@ -16,8 +17,10 @@ class fzf_select(Command):
             -o -print 2> /dev/null | sed 1d | cut -b3- | fzf +m"
         fzf = self.fm.execute_command(command, stdout=subprocess.PIPE)
         stdout, stderr = fzf.communicate()
+        if sys.version_info[0] >= 3:
+            stdout = stdout.decode('utf-8')
         if fzf.returncode == 0:
-            fzf_file = os.path.abspath(stdout.decode('utf-8').rstrip('\n'))
+            fzf_file = os.path.abspath(stdout.rstrip('\n'))
             if os.path.isdir(fzf_file):
                 self.fm.cd(fzf_file)
             else:

Both solutions work for me, with both Python versions. I'm using the latter (this code) right now.

[toonn]

Thanks for reporting, I'll go with the former since solution since others have. Not sure what the advantage would be of decoding into a different encoding than the locale's.

[toonn]

Updated the code on the wiki again. @abbottc, @AmaruCoder, could you test this?

[AmaruCoder]

This solution works. Thank you.

[toonn]

Ok, good, then I'm closing this.

[AmaruCoder]

After upgrading to the newest ranger the functionally is not working again. Once fzf_select is invoked, it shows the fzf interface, but after selecting a file, nothing happens. I am using macOS Sierra (10.12.6) and ranger 1.9.2.

[vifon]

Are your config files up to date?

[AmaruCoder]

Yes, everything is up to date. The integration wigh locate works fine. See the code below:

# fzf_locate
class fzf_locate(Command):
    """
    :fzf_locate
    Find a file using fzf.
    With a prefix argument select only directories.
    See: https://github.com/junegunn/fzf
    """
    def execute(self):
        import subprocess
        if self.quantifier:
            command="locate ~ | fzf -e -i"
        else:
            command="locate ~ | fzf -e -i"
        fzf = self.fm.execute_command(command, stdout=subprocess.PIPE)
        stdout, stderr = fzf.communicate()
        if fzf.returncode == 0:
            fzf_file = os.path.abspath(stdout.decode('utf-8').rstrip('\n'))
            if os.path.isdir(fzf_file):
                self.fm.cd(fzf_file)
            else:
                self.fm.select_file(fzf_file)