emborg manifest fails with unicode error

[nidan841g]

I did an 'emborg manifest' on a raspberry pi running raspbian 10.3 (a derivative of debian 10.3) and saw the following traceback. The files were not listed by the manifest call.

  File "/usr/local/bin/emborg", line 10, in <module>  sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/emborg/main.py", line 96, in main cmd_name, args, settings, emborg_opts
  File "/usr/local/lib/python3.7/dist-packages/emborg/command.py", line 162, in execute  exit_status = cls.run(name, args if args else [], settings, options)
  File "/usr/local/lib/python3.7/dist-packages/emborg/command.py", line 1127, in run sep="\n",
  File "/usr/local/lib/python3.7/dist-packages/inform/inform.py", line 1452, in __call__   INFORMER._report(args, kwargs, self)
  File "/usr/local/lib/python3.7/dist-packages/inform/inform.py", line 1936, in _report multiline, continuing, options
  File "/usr/local/lib/python3.7/dist-packages/inform/inform.py", line 2028, in _show_msg
 print(': '.join(cull([header, culprit, message])), **options)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0151' in position 82: ordinal not in range(256)
root@raspberrypi:~# client_loop: send disconnect: Broken pipe

[KenKundert]

This can be a little difficult to debug, as I cannot replicate the situation here. However, I have added an encoding setting and updated the version in github. Can you try it and see if it fixes the issue?

It would require that you add:

    encoding = 'utf-8'

to your settings file, except you should replace 'utf-8' with the needed encoding. Hopefully you know what it needs to be.

[KenKundert]

I may have misread the issue. The fix might be as simple as specifying PYTHONIOENCODING. Can you try that? If you are using Bash, you would do so by running emborg as follows:

    PYTHONIOENCODING='utf-8' emborg manifest

Of course you would replace 'utf-8' with the needed encoding. Please let me know if this works.

[nidan841g]

That works! Thanks. Closing the ticket.

[KenKundert]

To avoid having to type that each time you run emborg, I recommend that you set the environment variable in your bash start up file:

export PYTHONIOENCODING='utf-8'

Can you tell us what encoding you needed, in case others run in to this issue? Thanks!

[nidan841g]

I grabbed a diff from github with your delta from 1.16.0 -> 1.16.1 and patched my emborg that was installed with "pip3 install emborg" on a raspbian 10.3 system (debian derivative). The 2 files that were modified are settings.py and preferences.py, which live in /usr/local/lib/python3.7/dist-packages/emborg.

I still see the problem if I do not set PYTHONIOENCODING='utf-8' in the shell that invokes emborg. I have inserted a line "encoding = 'utf-8'" in ~/.config/emborg/settings. If I do set PYTHONIOENCODING='utf-8' in the shell that invokes emborg, it runs without error and lists all the files in the borg backup.

[nidan841g]

Here's the diff I used to patch my raspbian 10.3 system.

diff --git a/emborg/preferences.py b/emborg/preferences.py
index 24cd663..24932fe 100644
--- a/emborg/preferences.py
+++ b/emborg/preferences.py
@@ -25,7 +25,6 @@
 # Constants {{{2
 PROGRAM_NAME = "emborg"
 DEFAULT_COMMAND = "create"
-ENCODING = "utf-8"
 INDENT = "    "
 BORG = "borg"

@@ -43,6 +42,7 @@
 CONFIGS_SETTING = "configurations"
 DEFAULT_CONFIG_SETTING = "default_configuration"
 INCLUDE_SETTING = "include"
+DEFAULT_ENCODING = "utf-8"

 # Emborg settings {{{2
 EMBORG_SETTINGS = dict(
@@ -57,6 +57,7 @@
     default_configuration="default Emborg configuration",
     default_mount_point="directory to use as mount point if one is not specified",
     do_not_expand="names of settings that must not undergo setting evaluation",
+    encoding="encoding when talking to borg",
     encryption="encryption method (see Borg documentation)",
     excludes="list of glob strings of files or directories to skip",
     exclude_from="file that contains exclude patterns",
diff --git a/emborg/settings.py b/emborg/settings.py
index 3924e2c..53a2d55 100644
--- a/emborg/settings.py
+++ b/emborg/settings.py
@@ -44,7 +44,10 @@
     render,
     warn,
 )
-from shlib import Run, cd, cwd, getmod, mv, render_command, rm, to_path
+from shlib import (
+    Run, cd, cwd, getmod, mv, render_command, rm, to_path,
+    set_prefs as set_shlib_prefs
+)

 from .collection import Collection, split_lines
 from .patterns import (
@@ -62,6 +65,7 @@
     DATA_DIR,
     DATE_FILE,
     DEFAULT_CONFIG_SETTING,
+    DEFAULT_ENCODING,
     INCLUDE_SETTING,
     INITIAL_HOME_CONFIG_FILE_CONTENTS,
     INITIAL_ROOT_CONFIG_FILE_CONTENTS,
@@ -173,6 +177,7 @@ def __init__(self, name=None, command=None, emborg_opts=()):
         self.config_dir = to_path(CONFIG_DIR)
         self.read(name)
         self.check()
+        set_shlib_prefs(encoding=self.encoding if self.encoding else DEFAULT_ENCODING)

     # read() {{{2
     def read(self, name=None, path=None):

[KenKundert]

Yes, that is expected. I was on the wrong track when I changed the code. I was thinking that the unicode error occurred when reading the output of the pipe from borg. But after I made the change I read your message more closely and realized that the issue occurred when using the print function. To fix the print function you need to tell Python itself what encoding to use. Hence the use of the environment variable. This is not really an emborg issue, rather there is an incompatibility between Python and your operating system. So, adding the environment variable should help with other Python programs as well.

Though adding the encoding setting did not help in your case, it does not hurt and it might help in other situations, so I intend to leave it in.

[nidan841g]

Closing this issue for good now. Thanks for writing emborg and sharing it.