UnicodeEncodeError on some Linux'es (ubuntu 16.04 and Slackware 14.2)

[qxlsz]

In [17]: from alive_progress import alive_bar

In [18]: def pretty_sleep(sleep_time: int, bar='filling'):
    ...:     """pretty sleep with progress bar"""
    ...:     with alive_bar(int(sleep_time), bar=bar) as bar:
    ...:         for i in range(int(sleep_time)):
    ...:             time.sleep(0.001)
    ...:             bar()
    ...:

In [19]: pretty_sleep(100)
Exception in thread Thread-1851:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rjosyula/x/lib/python3.5/site-packages/alive_progress/progress.py", line 100, in run
    alive_repr(next(player))
  File "/home/rjosyula/x/lib/python3.5/site-packages/alive_progress/progress.py", line 115, in alive_repr
    sys.__stdout__.write(line + (spin and '\r' or '\n'))
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 43-45: ordinal not in range(256)

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-19-5b2d8b0fde03> in <module>
----> 1 pretty_sleep(100)

<ipython-input-18-920a2b64dd16> in pretty_sleep(sleep_time, bar)
      4         for i in range(int(sleep_time)):
      5             time.sleep(0.001)
----> 6             bar()
      7

/usr/lib/python3.5/contextlib.py in __exit__(self, type, value, traceback)
     64         if type is None:
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:
     68                 return

~/x/lib/python3.5/site-packages/alive_progress/progress.py in alive_bar(total, title, calibrate, **options)
    257
    258     end, run.text, run.stats = True, '', stats_end
--> 259     alive_repr()

~/x/lib/python3.5/site-packages/alive_progress/progress.py in alive_repr(spin)
    113             if line_len < run.last_line_len:
    114                 clear_traces()
--> 115             sys.__stdout__.write(line + (spin and '\r' or '\n'))
    116             sys.__stdout__.flush()
    117

UnicodeEncodeError: 'latin-1' codec can't encode characters in position 1-40: ordinal not in range(256)

[qxlsz]

running on linux ubuntu 16.04 LTS

[rsalmei]

Hey @qxlsz, thanks.

But that error is really weird, I cannot reproduce it. I support all python versions in alive-progress.

I've even installed some python 3.5 (3.5.3 and 3.5.7 actually) to test your code, but it works like a charm.

 ~ $ pyenv shell 3.5.3
 ~ $ ipython
Python 3.5.3 (default, Oct 24 2019, 19:26:18) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from alive_progress import alive_bar                                                                                                                                             

In [2]: import time                                                                                                                                                                      

In [3]: def pretty_sleep(sleep_time: int, bar='filling'): 
   ...:     """pretty sleep with progress bar""" 
   ...:     with alive_bar(int(sleep_time), bar=bar) as bar: 
   ...:         for i in range(int(sleep_time)): 
   ...:             time.sleep(0.001) 
   ...:             bar() 
   ...:                                                                                                                                                                                  

In [4]: pretty_sleep(100)                                                                                                                                                                
|████████████████████████████████████████| 100/100 [100%] in 0.1s (878.94/s) 

In [5]:                                                                                                                                                                                  

Could you please help me debug that?

• try it in another python;
• try it in another shell;
• try it in another terminal app;
• try updating your ipython (which version is it?);
• try running print('\u2585')

Please report back, thanks!

[rogerio-geru]

Hello @qxlsz

Can you help me here please?

[MurdoMaclachlan]

Disappointing to see this issue closed without any further information, since I'm having the same issue.

Running on Slackware 14.2 with Python 3.7.3.

Running print('\u2585') gave the same error.

Worth noting it doesn't crash my program, but I do get a traceback where I should see a progress bar.

[rsalmei]

Hey @MurdoMaclachlan, don't be disappointed! I've closed because I couldn't reproduce it, and there was no activity in a month. If you could help me understand it, I may be able to help.

That's weird, it does work everywhere I've seen:

You could in the meantime try another spinner style, which do not use that character...

[rsalmei]

I need help to understand this problem.

• Is it because the linux boxes are too old?
• or some sort of locale misconfiguration?
• how to change latin-1 to utf-8?

[MurdoMaclachlan]

Sorry taking time to get back, been busy.

Although I was fairly certain changing latin-1 to utf-8 would fix it, I've also had little luck with actually finding out how to do that.

"Is it because the linux boxes are too old?" - not entirely sure what this means, but I'm kinda tired so maybe just having brain fart :P

As far as changing the spinners, I tried setting the theme to classic with with aliveBar(gvars.config["limit"], theme='classic') as progress: (I imported it as aliveBar because the rest of the project uses camelCase), and got the following error? Might just be because I'm a newbie with this module.

raceback (most recent call last):
  File "oscr", line 59, in <module>
    from oscrmodules import main
  File "/home/murdo/Silphaer/StuffFolder/ProgrammingStuff/PRAW/oscr/oscrmodules/main.py", line 98, in <module>
    with aliveBar(gvars.config["limit"], theme='classic') as progress:
  File "/usr/lib64/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/murdo/.local/lib/python3.7/site-packages/alive_progress/core/progress.py", line 95, in alive_bar
    config = config_handler(**options)
  File "/home/murdo/.local/lib/python3.7/site-packages/alive_progress/core/configuration.py", line 82, in create_context
    local_config.update(_parse(theme, options))
  File "/home/murdo/.local/lib/python3.7/site-packages/alive_progress/core/configuration.py", line 102, in _parse
    raise ValueError('invalid theme name={}'.format(repr(theme)))
ValueError: invalid theme name='classic'

[rsalmei]

Hey @MurdoMaclachlan, sorry I took so long, haven't seen your reply!

Yeah, it's just a newbie mistake, there's a spinner named classic, but not a theme. Just change word theme -> spinner.

Could you please test something? Run this in your shell and try again: PYTHONIOENCODING=utf8

There're some other tricks here if that doesn't work: https://stackoverflow.com/questions/2276200/changing-default-encoding-of-python

What does your system report to this?

import sys
sys.getdefaultencoding()

[MurdoMaclachlan]

No worries about the wait!

Changing the spinner to classic gets me as far as the following:

But as soon as it gets enough progress to display on the bar, I get the same error.

I tried the PYTHONIOENCODING=utf8 and it didn't work.

import sys
sys.getdefaultencoding()

reports utf-8.

I tried a few of the methods from StackOverflow, but none of them seemed to fix it.

[MurdoMaclachlan]

I just added bar='classic' as well, after realising that these are two different configurations. Using both of these, the progress bar works perfectly as the classic settings.

[rsalmei]

Well, going to close this one as stale. Please let me know of anything new.

[mborsalino]

I recently stumbled upon the same issue. This is the code snippet to reproduce the issue.

import time
from alive_progress import alive_bar

with alive_bar(force_tty=True,
               title='foo',
               monitor=False,
               spinner='twirls',
               unknown='fish2') as bar:
    for i in range(4):
        bar()
        time.sleep(0.5)

If I run the above code in a terminal it will work as expected. But if I run it inside a terminal within a VNC session then the issue show up:

UnicodeEncodeError                        Traceback (most recent call last)
/home/mborsalino/tmp/test_748.py in <module>
     10             bar()
     11             time.sleep(0.5)
---> 12         print("All good")

/cad/tools/common/python/3.7.4/lib/python3.7/contextlib.py in __exit__(self, type, value, traceback)
    117         if type is None:
    118             try:
--> 119                 next(self.gen)
    120             except StopIteration:
    121                 return False

/cad/tools/common/python/3.7.4/env/default/lib/python3.7/site-packages/alive_progress/core/progress.py in __alive_bar(config, total, calibrate, _cond)
    273         if not config.receipt_text:
    274             run.text = ''
--> 275         alive_repr()
    276         term.write('\n')
    277     else:

/cad/tools/common/python/3.7.4/env/default/lib/python3.7/site-packages/alive_progress/core/progress.py in alive_repr(spin)
    122                      elapsed(), stats(), run.text)
    123 
--> 124         run.last_len = print_cells(fragments, term.cols(), run.last_len, _term=term)
    125         term.flush()
    126 

/cad/tools/common/python/3.7.4/env/default/lib/python3.7/site-packages/alive_progress/utils/cells.py in print_cells(fragments, cols, last_line_len, _term)
     86             available, fragment = 0, fix_cells(fragment[:available])
     87 
---> 88         _term.write(join_cells(fragment))
     89         if available == 0:
     90             break

UnicodeEncodeError: 'latin-1' codec can't encode characters in position 1-40: ordinal not in range(256)

Hopefully this will help you reproduce the issue.

• python 3.7.4
• Centos 7.9
• alive_progress 2.2.0

[rsalmei]

Hello @mborsalino,

Humm, only inside a VNC session, huh? That is new to me... It seems the terminal is not being set up correctly.

Could you try to run this before running your script?

export PYTHONIOENCODING=UTF-8

[rsalmei]

Also, searching for "python LC_CTYPE" has given some interesting tips... Please try with the tips in here too: https://stackoverflow.com/questions/21251272/why-lc-ctype-needs-to-be-set-manually-for-python-in-some-cases

[mborsalino]

Could you try to run this before running your script?

export PYTHONIOENCODING=UTF-8

This makes it work as expected

[mborsalino]

Also, searching for "python LC_CTYPE" has given some interesting tips... Please try with the tips in here too: https://stackoverflow.com/questions/21251272/why-lc-ctype-needs-to-be-set-manually-for-python-in-some-cases

I compared the environment of the VNC terminal Vs. non-VNC.

VNC:
  LC_CTYPE: not set
  LANG=en_US

non VNC:
  LC_CTYPE: not set
  LANG=en_US.UTF-8

Thanks for looking into this. As far as I am concerned I can live with the PYTHONIOENCONDING workaround. I guess it will be up to you to see whether you want to perform some checks and eventually revert to the "classic" bar workaround automatically.

Thanks again for your help, please let me know if there's anything else you'd like me to check given that I have an environment where I can easily reproduce the issue.

[TheTechRobo]

@mborsalino BTW, you can automatically run that command in your shell. Add it to your ~/.bashrc (or ~/.zshrc if you use zsh) and it will run every time the shell starts.

[mborsalino]

@TheTechRobo: the problem is that when code is distributed there's no guarantee regarding the environment that is going to be used.

This is not an issue for my particular case because users of my code never start Python directly, but rather via a wrapper script. So I have the freedom to set the environment in the wrapper script and be done with it.

Others might want to write their own applications/modules using alive-progress. When they distribute the module they have no control over the environment where it's going to be used.

A simple warning regarding this potential issue in alive_progress documentation could suffice. A better approach would be to try to catch the non-UTF-8 encoding and default to a "classic" progress bar appearance. But this is up to @rsalmei since he's the maintainer.

[qxlsz]

@TheTechRobo: the problem is that when code is distributed there's no guarantee regarding the environment that is going to be used.

This is not an issue for my particular case because users of my code never start Python directly, but rather via a wrapper script. So I have the freedom to set the environment in the wrapper script and be done with it.

Others might want to write their own applications/modules using alive-progress. When they distribute the module they have no control over the environment where it's going to be used.

A simple warning regarding this potential issue in alive_progress documentation could suffice. A better approach would be to try to catch the non-UTF-8 encoding and default to a "classic" progress bar appearance. But this is up to @rsalmei since he's the maintainer.

well said

[TheTechRobo]

@mborsalino I thought you were just running your script on your personal computer and that was it. Sorry

[rsalmei]

Thanks, @mborsalino, I'm glad it has worked! And the find about LANG is also relevant, perhaps that would also make it work, even without PYTHONIOENCONDING...

Anyway, I'd love to "make some checks" and revert to classic if errors are foreseen, but unfortunately, it's not clear which checks would be needed. @MurdoMaclachlan has stated above that this has not worked for him:

I tried the PYTHONIOENCODING=utf8 and it didn't work.

Also, surprisingly, his default encoding in python returned utf-8........

I'm still clueless here, I fail to see how I could automatize anything just yet 😓

[rsalmei]

Well, I'm going to close this one again. Python can be cryptic and convoluted, especially when combined with cryptic and convoluted terminals... Since I can't even test if the PYTHONIOENCODING is really UTF-8, I'm not sure what more could be done. Please post if anything new arises.

[raphaelsetin]

The only solution that worked from within my Python code was the following:

import sys

sys.stdin.reconfigure(encoding='utf-8')
sys.stdout.reconfigure(encoding='utf-8')

For me the problem was using alive-progress through Gooey inside a packaged binary from PyInstaller. My OS is Windows 10, btw.