search

manjaro / pacman-mirrors

This repo has been archived. Our code is now hosted at
https://gitlab.manjaro.org
GNU General Public License v3.0
25 stars 16 forks source link

UnicodeEncodeError: 'ascii' codec can't encode character '\u0144' in position 26: ordinal not in range(128) #89

Closed FadeMind closed 7 years ago

FadeMind commented 7 years ago

Interactive mode is damaged:

[fademind@manjaro ~]$ sudo pacman-mirrors -i         
.: Info Downloading mirrors from repo.manjaro.org
.: Info Using default mirror file
.: Info Querying mirrors - This may take some time
   1.223 Australia      : http://mirror.ventraip.net.au/Manjaro/
   0.957 Australia      : http://manjaro.uberglobalmirror.com/
   0.946 Australia      : http://manjaro.mirror.serversaustralia.com.au/
   0.385 Austria        : http://mirror.inode.at/manjaro/
   0.285 Belarus        : http://mirror.datacenter.by/pub/mirrors/manjaro/
   0.324 Belgium        : http://ftp.belnet.be/manjaro/
   0.478 Belgium        : https://manjaro.cu.be/
   0.730 Brasil         : http://linorg.usp.br/manjaro/
   0.625 Bulgaria       : https://mirrors.netix.net/manjaro/
   0.726 Bulgaria       : http://manjaro.ipacct.com/manjaro/
   0.455 Bulgaria       : http://manjaro.telecoms.bg/
   0.463 Canada         : http://manjaro.cybr.ca/
   0.804 Canada         : https://ca.mirror.babylon.network/manjaro/
   0.971 Chile          : http://manjaro.dcc.uchile.cl/
   ..... Chile          : http://doge.ing.puc.cl/Mirrors/Manjaro/
.: Error timed out 'http://doge.ing.puc.cl/Mirrors/Manjaro/'

   1.283 China          : http://mirrors.ustc.edu.cn/manjaro/
   1.101 China          : http://ftp.cuhk.edu.hk/pub/Linux/manjaro/
   0.887 China          : http://mirrors.tuna.tsinghua.edu.cn/manjaro/
   ..... China          : https://mirrors.zju.edu.cn/manjaro/
.: Error _ssl.c:733: The handshake operation timed out 'https://mirrors.zju.edu.cn/manjaro/'

   0.787 Colombia       : http://mirror.edatel.net.co/manjaro/
   0.793 Costa_Rica     : http://mirrors.ucr.ac.cr/manjaro/
   0.498 Czech          : https://mirror.dkm.cz/manjaro/
   0.487 Denmark        : https://mirrors.dotsrc.org/manjaro/
   0.527 Denmark        : https://www.uex.dk/repos/manjaro/
   1.217 Ecuador        : https://mirror.cedia.org.ec/manjaro/
   ..... Ecuador        : https://mirror.uta.edu.ec/manjaro/
.: Error timed out 'https://mirror.uta.edu.ec/manjaro/'

   1.161 Ecuador        : https://mirror.espoch.edu.ec/manjaro/
   0.328 France         : http://manjarolinux.polymorf.fr/
   0.327 France         : http://mirror.lignux.com/manjaro/
   0.330 France         : http://ftp.free.org/mirrors/repo.manjaro.org/repos/
   0.518 France         : https://fr.mirror.babylon.network/manjaro/
   0.454 Germany        : http://mirror.ragenetwork.de/manjaro/
   0.311 Germany        : http://ftp.halifax.rwth-aachen.de/manjaro/
   0.340 Germany        : http://ftp.tu-chemnitz.de/pub/linux/manjaro/
   0.529 Germany        : https://mirror.netzspielplatz.de/manjaro/packages/
   0.496 Germany        : https://mirror.netcologne.de/manjaro/
   0.454 Germany        : https://repo.rhindon.net/manjaro/
   0.453 Germany        : https://repo.stdout.net/manjaro/
   0.493 Germany        : https://manjaro-mirror-a.alpix.eu/
   0.343 Germany        : http://ftp.rz.tu-bs.de/pub/mirror/manjaro.org/repos/
   0.412 Greece         : http://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.406 Hungary        : http://mirror.infotronik.hu/mirrors/pub/manjaro/
   1.028 Indonesia      : http://kambing.ui.ac.id/manjaro/
   ..... Indonesia      : http://kartolo.sby.datautama.net.id/manjaro/
.: Error [Errno 97] Address family not supported by protocol 'http://kartolo.sby.datautama.net.id/manjaro/'

   1.480 Indonesia      : http://jaran.undip.ac.id/manjaro/
   0.570 Ireland        : https://ftp.heanet.ie/mirrors/manjaro/
   0.362 Italy          : http://manjaro.mirror.garr.it/mirrors/manjaro/
   0.361 Italy          : http://ba.mirror.garr.it/mirrors/manjaro/
   0.362 Italy          : http://ct.mirror.garr.it/mirrors/manjaro/
   1.375 Japan          : http://ftp.tsukuba.wide.ad.jp/Linux/manjaro/
   0.810 Japan          : http://ftp.riken.jp/Linux/manjaro/
   0.337 Netherlands    : http://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
   0.310 Netherlands    : http://ftp.snt.utwente.nl/pub/linux/manjaro/
   0.513 Netherlands    : https://nl.mirror.babylon.network/manjaro/
   0.865 Philippines    : http://mirror.rise.ph/manjaro/
   0.373 Poland         : http://mirror.chmuri.net/manjaro/
   0.393 Portugal       : http://ftp.dei.uc.pt/pub/linux/manjaro/
   0.380 Portugal       : http://manjaro.barata.pt/
   0.456 Romania        : http://mirrors.serverhost.ro/manjaro/packages/
   0.367 Romania        : http://ftp.lug.ro/manjaro/
   0.476 Russia         : https://mirror.yandex.ru/mirrors/manjaro/
   1.313 Singapore      : http://download.nus.edu.sg/mirror/manjaro/
   0.878 South_Africa   : http://mirror.is.co.za/mirrors/manjaro.org/
   0.491 Sweden         : https://ftp.lysator.liu.se/pub/manjaro/
   0.467 Spain          : http://ftp.caliu.cat/manjaro/
   0.340 Sweden         : http://mirror.zetup.net/manjaro/
   1.156 Taiwan         : http://free.nchc.org.tw/manjaro/
   0.420 Turkey         : http://ftp.linux.org.tr/manjaro/
   0.310 United_Kingdom : http://repo.manjaro.org.uk/
   0.467 United_Kingdom : https://www.mirrorservice.org/sites/repo.manjaro.org/repos/
   0.345 United_Kingdom : http://mirror.catn.com/pub/manjaro/
   0.306 United_Kingdom : http://manjaro.mirrors.uk2.net/
   0.502 United_States  : http://mirror.dacentec.com/manjaro/
   0.495 United_States  : http://mirror.jmu.edu/manjaro/
   0.549 United_States  : http://mirror.solarvps.com/manjaro/
   0.614 United_States  : http://mirror.nexcess.net/manjaro/
   0.609 United_States  : http://distro.ibiblio.org/manjaro/
   0.548 United_States  : http://mirror.clarkson.edu/manjaro/
   0.598 United_States  : http://mirrors.ocf.berkeley.edu/manjaro/
   1.974 Vietnam        : https://mirror.freedif.org/Manjaro/

(pacman-mirrors:3371): Gtk-WARNING **: Theme parsing error: gtk.css:68:35: The style property GtkButton:child-displacement-x is deprecated and shouldn't be used anymore. It will be removed in a future version

(pacman-mirrors:3371): Gtk-WARNING **: Theme parsing error: gtk.css:69:35: The style property GtkButton:child-displacement-y is deprecated and shouldn't be used anymore. It will be removed in a future version

(pacman-mirrors:3371): Gtk-WARNING **: Theme parsing error: gtk.css:73:46: The style property GtkScrolledWindow:scrollbars-within-bevel is deprecated and shouldn't be used anymore. It will be removed in a future version

.: Info User generated mirror list
--------------------------
.: Info Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json
.: Info Writing mirror list
Traceback (most recent call last):
  File "/usr/bin/pacman-mirrors", line 55, in <module>
    pm.run()
  File "/usr/lib/python3.6/site-packages/pacman_mirrors/pacman_mirrors.py", line 419, in run
    self.build_interactive_mirror_list()
  File "/usr/lib/python3.6/site-packages/pacman_mirrors/pacman_mirrors.py", line 313, in build_interactive_mirror_list
    filefn.output_mirror_list(self.config, selected, custom=True, quiet=self.quiet)
  File "/usr/lib/python3.6/site-packages/pacman_mirrors/filefn.py", line 78, in output_mirror_list
    write_mirrorlist_header(outfile, custom=custom)
  File "/usr/lib/python3.6/site-packages/pacman_mirrors/filefn.py", line 127, in write_mirrorlist_header
    datetime.datetime.now().strftime("%d %B %Y %H:%M")))
UnicodeEncodeError: 'ascii' codec can't encode character '\u0144' in position 26: ordinal not in range(128)
[fademind@manjaro ~]$
fhdk commented 7 years ago

@FadeMind Could you please tell me the output of pacman-mirrors -v?

output-pacman-mirrors-4.0.3-2017-04-10.1948.txt

Did you change your symlink /usr/bin/python from python3 to python2?

UnicodeEncodeError: 'ascii' codec can't encode character '\u0144' in position 26: ordinal not in range(128)

The error indicates Python 2 is used instead of Python 3.6 Python 3 uses unicode string as default whereas Python 2 uses ascii str

Also check that these packages are installed and current

gtk3
python-gobject

Please report back the result of the above checks.

FadeMind commented 7 years ago

Issue regarding with lastest python 3.6.x changes see: https://github.com/StevenBlack/hosts/issues/258#issuecomment-284878414

Switch to Python2 (via symlink will WORKAROUND issue cause Py2 don't use encoding arg in open() ). Currently I don't have access to my manjaro setup, so I cannot attach any logs.

Regards

fhdk commented 7 years ago

@Huluti This issue is a fine example of why we should have this in file headers for pacman-mirrors.

#!/usr/bin/env python3

How else to make sure the script is executed with python3 when user has changed the default python env?

fhdk commented 7 years ago

@FadeMind I have changed the fileheaders for pacman-mirrors to point to python3

Nevermind - it does not help.

Seems you will have to find another way because it is not pacman-mirrors - at least what I make of it.

FadeMind commented 7 years ago

Solutions are two ways: switch to python2 or add proper code for handle CODECS in Py script.

philmmanjaro commented 7 years ago

Which version of pacman-mirrors is affected by this?

FadeMind commented 7 years ago

All lastest (since Manjaro switched to Python JSON mode scripting for generate mirrorlist).

This issue only happened AFTER select custom mirrors via interactive mode via -i switch.

fhdk commented 7 years ago

So you're saying that some handling of CODECS in pacman-mirrors is causing this?

Pacman-mirrors does not use any type of CODECS so what are you're referring to?

If so - it should be a problem for everyone using --interactive yet I have not heard about it - not even on the forum - which if it was a problem would have been boiling over the issue by now.

Could you please tell me excact steps to reproduce what is in your OP?

Could you please try install pacman-mirrors-dev from unstable and report back if it is also an issue there?

FadeMind commented 7 years ago

Reproduce:

sudo pacman-mirrors -i -c all

Select random mirrors and click on OK for apply changes. Custom mirrorlist will be generated and saved but at least error about ASCII encoding will happen.

Py 3.6 open() has an encoding argument: https://docs.python.org/3.6/library/functions.html#open https://docs.python.org/3.6/library/codecs.html

fhdk commented 7 years ago

I can see the Gtk-warning - but that I have never seen before - so something must have changed in the GTK implementation.

Just realized why I haven't seen it before, I am using Xfce on another rig and I am usually on Gnome - there must be some significance

at least error about ASCII encoding will happen

No encoding errors. By your name I guess Poland? Did you or any other just recently translated pacman-mirrors to Polish? If so I suspect that polish characters are written to the mirrorlist according to the last error

It could be that the %d %B %Y %H:%M contains regional chars which is outside ascii.

File "/usr/lib/python3.6/site-packages/pacman_mirrors/filefn.py", line 127, in write_mirrorlist_header
    datetime.datetime.now().strftime("%d %B %Y %H:%M")))
UnicodeEncodeError: 'ascii' codec can't encode character '\u0144' in position 26: ordinal not in range(128)

What are your output of pacman-mirrors -v

The links you are referring to has nothing to do with the mirrorlist generation so it is not that the fetched file is decoded because that happens when pacman-mirrors load.

/media/.../manjaro/pamac >>> sudo pacman-mirrors -v               ±[master]
pacman-mirrors 4.0.3
/media/.../manjaro/pamac >>> sudo pacman-mirrors -i               ±[master]
.: Info Downloading mirrors from repo.manjaro.org
.: Info Using custom mirror file
.: Info Querying mirrors - This may take some time
   0.073 Denmark        : https://www.uex.dk/repos/manjaro/
   0.077 Denmark        : https://mirrors.dotsrc.org/manjaro/

(pacman-mirrors:15094): Gtk-WARNING **: Theme parsing error: gtk.css:68:35: The style property GtkButton:child-displacement-x is deprecated and shouldn't be used anymore. It will be removed in a future version

(pacman-mirrors:15094): Gtk-WARNING **: Theme parsing error: gtk.css:69:35: The style property GtkButton:child-displacement-y is deprecated and shouldn't be used anymore. It will be removed in a future version

(pacman-mirrors:15094): Gtk-WARNING **: Theme parsing error: gtk.css:73:46: The style property GtkScrolledWindow:scrollbars-within-bevel is deprecated and shouldn't be used anymore. It will be removed in a future version

.: Info User generated mirror list
--------------------------
.: Info Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json
.: Info Writing mirror list
   Denmark         : http://www.uex.dk/repos/manjaro/stable
   Denmark         : https://www.uex.dk/repos/manjaro/stable
   Denmark         : http://mirrors.dotsrc.org/manjaro/stable
   Denmark         : https://mirrors.dotsrc.org/manjaro/stable
.: Info Mirrorlist generated and saved to: /etc/pacman.d/mirrorlist
.: Info To reset custom config run  pacman-mirrors -c all
FadeMind commented 7 years ago

Like I mention in PM, I don't have now access to my Manjaro setup disk, but I was on Unstable branch and 9 days ago this happens.

fhdk commented 7 years ago

I have found that it maybe is a bug in the internal strftime() The bug is mentioned here: http://stackoverflow.com/questions/16034060/python3-datetime-datetime-strftime-failed-to-accept-utf-8-string-format

I have changed the datetime output so it goes like this

dd-mm-yyyy hh:mm

Instead of

dd monthlocalename yyyy hh:mm

It should remove the bug and I know its the easy way - but it is just a convenience timestamp so the user can see when the mirrorlist was generated.

fhdk commented 7 years ago

@philmmanjaro I have pushed a commit to master branch Will you build v4.1.1-dev and upload a new package

fhdk commented 7 years ago

@Huluti Do you know what to do with this message which recently has emerged?

(pacman-mirrors:3371): Gtk-WARNING **: Theme parsing error: gtk.css:68:35: The style property GtkButton:child-displacement-x is deprecated and shouldn't be used anymore. It will be removed in a future version

(pacman-mirrors:3371): Gtk-WARNING **: Theme parsing error: gtk.css:69:35: The style property GtkButton:child-displacement-y is deprecated and shouldn't be used anymore. It will be removed in a future version

(pacman-mirrors:3371): Gtk-WARNING **: Theme parsing error: gtk.css:73:46: The style property GtkScrolledWindow:scrollbars-within-bevel is deprecated and shouldn't be used anymore. It will be removed in a future version

Just realized why I haven't seen it before - I am usually on Gnome - there must be some significance

philmmanjaro commented 7 years ago

@FadeMind: I've released pacman-mirrors-dev-4.1.0-2 to our unstable branch. Please give us feedback. @fhdk: Is v4.1.0 ready to be released?

fhdk commented 7 years ago

@philmmanjaro

Is v4.1.0 ready to be released?

Indeed

fhdk commented 7 years ago

@Huluti @philmmanjaro The above comment regarding the gtk-warning about theme parsing error - I have no idea what to do with it. I have seen it - on xfce edition - now that I am back on gnome edition is does not exist. Otherwise - it is good to go :smile: I have testet the additions very thoroughly - but of course - you'll never fully know until it is in the hand of the user.

philmmanjaro commented 7 years ago

@FadeMind: should be also fixed in v4.0.4. Please confirm.

FadeMind commented 7 years ago

@philmmanjaro FIXED.