search

opensouls / terminal-copilot

A smart terminal assistant that helps you find the right command.
Apache License 2.0
573 stars 43 forks source link

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 3444: invalid continuation byte #11

Open kirxkirx opened 1 year ago

kirxkirx commented 1 year ago

The program does not run with a non-UTF8 locale crashing with the error message:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 3444: invalid continuation byte

It does not seem to be related to the terminal emulator I'm using: reproduced with konsole and aterm. Here is the complete output of my attempt to install and try out terminal-copilot on Gentoo Linux:

kirx@eridan ~ $ pip install terminal-copilot --user
Collecting terminal-copilot
  Downloading terminal_copilot-1.0.7-py3-none-any.whl (8.2 kB)
Requirement already satisfied: charset-normalizer==2.1.1 in /usr/lib64/python3.8/site-packages (from terminal-copilot) (2.1.1)
Collecting simple-term-menu==1.5.2
  Downloading simple_term_menu-1.5.2-py3-none-any.whl (26 kB)
Collecting typing-extensions==4.4.0
  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Requirement already satisfied: idna==3.4 in /usr/lib64/python3.8/site-packages (from terminal-copilot) (3.4)
Collecting numpy==1.23.5
  Downloading numpy-1.23.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ---------------------------------------- 17.1/17.1 MB 2.4 MB/s eta 0:00:00
Collecting openai==0.25.0
  Downloading openai-0.25.0.tar.gz (44 kB)
     --------------------------------------- 44.9/44.9 kB 825.8 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting et-xmlfile==1.1.0
  Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
Requirement already satisfied: python-dateutil==2.8.2 in /usr/lib64/python3.8/site-packages (from terminal-copilot) (2.8.2)
Collecting pytz==2022.6
  Downloading pytz-2022.6-py2.py3-none-any.whl (498 kB)
     ------------------------------------- 498.1/498.1 kB 622.6 kB/s eta 0:00:00
Requirement already satisfied: requests==2.28.1 in /usr/lib64/python3.8/site-packages (from terminal-copilot) (2.28.1)
Collecting certifi==2022.12.7
  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
     --------------------------------------- 155.3/155.3 kB 1.7 MB/s eta 0:00:00
Collecting types-pytz==2022.6.0.1
  Downloading types_pytz-2022.6.0.1-py3-none-any.whl (4.7 kB)
Collecting pandas-stubs==1.5.2.221124
  Downloading pandas_stubs-1.5.2.221124-py3-none-any.whl (146 kB)
     --------------------------------------- 146.4/146.4 kB 1.8 MB/s eta 0:00:00
Requirement already satisfied: six==1.16.0 in /usr/lib64/python3.8/site-packages (from terminal-copilot) (1.16.0)
Collecting openpyxl==3.0.10
  Downloading openpyxl-3.0.10-py2.py3-none-any.whl (242 kB)
     ------------------------------------- 242.1/242.1 kB 895.9 kB/s eta 0:00:00
Collecting pandas==1.5.2
  Downloading pandas-1.5.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
     ---------------------------------------- 12.2/12.2 MB 1.7 MB/s eta 0:00:00
Collecting tqdm==4.64.1
  Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
     ---------------------------------------- 78.5/78.5 kB 1.0 MB/s eta 0:00:00
Collecting urllib3==1.26.13
  Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
     --------------------------------------- 140.6/140.6 kB 1.1 MB/s eta 0:00:00
Building wheels for collected packages: openai
  Building wheel for openai (pyproject.toml) ... done
  Created wheel for openai: filename=openai-0.25.0-py3-none-any.whl size=55859 sha256=cb542f4654240372aefa0c06bf92b541a89d6a3c7a4e9446fc528ff7593fafb6
  Stored in directory: /home/kirx/.cache/pip/wheels/9b/ec/6b/a7a72f2bcb08749a0b4be510b397fda58c364b7216fc114e5e
Successfully built openai
Installing collected packages: types-pytz, pytz, urllib3, typing-extensions, tqdm, simple-term-menu, pandas-stubs, numpy, et-xmlfile, certifi, pandas, openpyxl, openai, terminal-copilot
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.60.0
    Uninstalling tqdm-4.60.0:
      Successfully uninstalled tqdm-4.60.0
Successfully installed certifi-2022.12.7 et-xmlfile-1.1.0 numpy-1.23.5 openai-0.25.0 openpyxl-3.0.10 pandas-1.5.2 pandas-stubs-1.5.2.221124 pytz-2022.6 simple-term-menu-1.5.2 terminal-copilot-1.0.7 tqdm-4.64.1 types-pytz-2022.6.0.1 typing-extensions-4.4.0 urllib3-1.26.13
kirx@eridan ~ $ copilot list all files
Traceback (most recent call last):
  File "/home/kirx/.local/bin/copilot", line 8, in <module>
    sys.exit(main())
  File "/home/kirx/.local/lib/python3.8/site-packages/copilot/copilot.py", line 48, in main
    {subprocess.run(["ls"], capture_output=True).stdout.decode("utf-8")}
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 4671: invalid continuation byte
kirx@eridan ~ $ export LANG=C
kirx@eridan ~ $ copilot list all files
Traceback (most recent call last):
  File "/home/kirx/.local/bin/copilot", line 8, in <module>
    sys.exit(main())
  File "/home/kirx/.local/lib/python3.8/site-packages/copilot/copilot.py", line 48, in main
    {subprocess.run(["ls"], capture_output=True).stdout.decode("utf-8")}
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 3444: invalid continuation byte
kirx@eridan ~ $ copilot how to get local time
Traceback (most recent call last):
  File "/home/kirx/.local/bin/copilot", line 8, in <module>
    sys.exit(main())
  File "/home/kirx/.local/lib/python3.8/site-packages/copilot/copilot.py", line 48, in main
    {subprocess.run(["ls"], capture_output=True).stdout.decode("utf-8")}
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 3444: invalid continuation byte
kirx@eridan ~ $ locale                        
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=
kirx@eridan ~ $ uname -a
Linux eridan 5.11.8-gentoo-r1-x86_64 #1 SMP Sun Mar 28 03:47:57 UTC 2021 x86_64 Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz GenuineIntel GNU/Linux
kirx@eridan ~ $ env
SHELL=/bin/bash
WINDOWID=37748738
COLORTERM=rxvt
XDG_CONFIG_DIRS=/etc/xdg
LESS=-R -M --shift 5
JDK_HOME=/etc/java-config-2/current-system-vm
CONFIG_PROTECT_MASK=/etc/sandbox.d /etc/php/cli-php7.4/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/apache2-php7.4/ext-active/ /etc/php/cli-php8.0/ext-active/ /etc/php/cgi-php8.0/ext-active/ /etc/php/apache2-php8.0/ext-active/ /etc/php/cli-php8.1/ext-active/ /etc/php/cgi-php8.1/ext-active/ /etc/php/apache2-php8.1/ext-active/ /etc/php/fpm-php8.1/ext-active/ /etc/php/phpdbg-php8.1/ext-active/ /etc/fonts/fonts.conf /etc/gentoo-release /etc/terminfo /etc/dconf /etc/ca-certificates.conf /etc/texmf/web2c /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/revdep-rebuild
HISTSIZE=
PGPLOT_DIR=/usr/lib64/pgplot/
JAVA_HOME=/etc/java-config-2/current-system-vm
iraf=/mnt/usb/iraf/iraf/
CALDBCONFIG=/mnt/usb/caldb//software/tools/caldb.config
HISTTIMEFORMAT=[%F %T] 
ANT_HOME=/usr/share/ant
ADS_API_TOKEN=lVivFKGO6bcxurjfKeN25SFswFtg7EOuT0vggcQp
EDITOR=/usr/bin/joe
PWD=/home/kirx
CONFIG_PROTECT=/usr/share/gnupg/qualified.txt /usr/share/config /usr/lib64/libreoffice/program/sofficerc
LOGNAME=kirx
PGPLOT_FONT=/usr/lib64/pgplot//grfont.dat
MANPATH=/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/11.2.1/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.38/man:/etc/java-config-2/current-system-vm/man/:/usr/lib64/php7.4/man/:/usr/lib64/php8.0/man/:/usr/lib64/php8.1/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/15/share/man:/usr/lib/llvm/14/share/man:/usr/lib/llvm/13/share/man
IRAFARCH=
XAUTHORITY=/home/kirx/.Xauthority
OPENCL_PROFILE=beignet
SCHED=/home/kirx/sched
WINDOWPATH=7
HOME=/home/kirx
LANG=C
HISTFILE=/home/kirx/.bash_eternal_history
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
FLTK_DOCDIR=/usr/share/doc/fltk-1.3.5-r4/html
DIFMAP_LOGIN=/home/kirx/.difmap
OPENGL_PROFILE=xorg-x11
QT_GRAPHICSSYSTEM=raster
HEADAS=/mnt/usb/heasoft-6.30.1/x86_64-pc-linux-gnu-libc2.32
CSPICE_PLANETDATA_FILE=/home/kirx/.vartools/pck00010.tpc
INFOPATH=/usr/share/gcc-data/x86_64-pc-linux-gnu/11.2.1/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.38/info:/usr/share/info:/usr/share/info/emacs-27
MOZ_GMP_PATH=/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed
JAVAC=/etc/java-config-2/current-system-vm/bin/javac
XEHELPURL=/usr/share/doc/xephem-4.1.0/html/xephem.html
TERMINFO=/usr/share/terminfo
TERM=rxvt
LESSOPEN=|lesspipe %s
USER=kirx
CSPICE_EPHEM_FILE=/home/kirx/.vartools/de432s.bsp
COLORFGBG=0;15
MANPAGER=manpager
DISPLAY=:0.0
CALDB=/mnt/usb/caldb/
SHLVL=3
PAGER=/usr/bin/less
LD_LIBRARY_PATH=/usr/local/lib::.
CALDBALIAS=/mnt/usb/caldb//software/tools/alias_config.fits
GCC_SPECS=
GSETTINGS_BACKEND=dconf
XDG_DATA_DIRS=/usr/local/share:/usr/share
PATH=/home/kirx/mybin:/home/kirx/.local/bin:/home/kirx/current_work/vast/util/ccd:/home/kirx/cod/wcstools-3.9.4/bin:/opt/sun-jre-bin-1.6.0.02/bin/:/home/kirx/.iraf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/15/bin:/usr/lib/llvm/14/bin:/usr/lib/llvm/13/bin:/usr/bin/cdsclient:/home/kirx/.gem/ruby/1.8/bin/:/usr/games/bin/
VBOX_APP_HOME=/usr/lib64/virtualbox
CSPICE_LEAPSEC_FILE=/home/kirx/.vartools/naif0011.tls
HISTFILESIZE=
OLDPWD=/home/kirx/video
_=/usr/bin/env

Previously, I had issues with various python software mysteriously crashing after seeing my non-UTF8 locale, but all the previous problems could be fixed by exporting LANG=C.

github-actions[bot] commented 1 year ago

Message that will be displayed on users' first issue

Shu-Ji commented 1 year ago 
$ copilot find a txt file
Traceback (most recent call last):
  File "/opt/homebrew/bin/copilot", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.10/site-packages/copilot/copilot.py", line 83, in main
    {history.get_history() if args.history and operating_system.lower().startswith("lin") or operating_system.lower().startswith("dar") else ""}
  File "/opt/homebrew/lib/python3.10/site-packages/copilot/history.py", line 46, in get_history
    return _zsh_history(history_context_size)
  File "/opt/homebrew/lib/python3.10/site-packages/copilot/history.py", line 29, in _zsh_history
    lines = history_file.zsh_history_file_lines()
  File "/opt/homebrew/lib/python3.10/site-packages/copilot/history_file.py", line 41, in zsh_history_file_lines
    lines = history.read().splitlines()
  File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 552: invalid start byte

There are some unicode chars in my ~/.zsh_history file:

image