Build fails with UnicodeDecodeError on german locale

[black-silence]

log:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
Creating 'mypaint-config.h'
gcc -o brushmodes.os -c -std=c99 -D_POSIX_C_SOURCE=200809L -fopenmp -O3 -Wall -Wstrict-prototypes -Werror -g -fPIC -DHAVE_JSON_C -DHAVE_GETTEXT -I. -I/usr/include/json-c brushmodes.c
generate_cheaders(["mypaint-brush-settings-gen.h", "brushsettings-gen.h"], ["generate.py", "brushsettings.py", "brushsettings.json"])
python2.7 generate.py mypaint-brush-settings-gen.h brushsettings-gen.h
Writing mypaint-brush-settings-gen.h
Traceback (most recent call last):
  File "generate.py", line 137, in <module>
    writefile(args[1], generate_internal_settings_code())
  File "generate.py", line 105, in generate_internal_settings_code
    [settings_info_struct(i) for i in brushsettings.settings])
  File "generate.py", line 69, in generate_static_struct_array
    entries.append(indent + "{%s}" % (", ".join(entry)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)
scons: *** Error 1

state of the repository is master/commit 28eb0bd315c6

locale -v:

LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

locale -a:

C
de_DE
de_DE@euro
de_DE.iso88591
de_DE.iso885915@euro
de_DE.utf8
deutsch
en_US
en_US.iso88591
en_US.utf8
german
POSIX

After adding a print(entry) before the line that throws, I noticed the "u" marker missing in front of the translated strings but not the strings that have no translation (second element of the tuple). The last line is the one that throws, "H\xc3\xa4rte" is the word "Härte".

(u'"opaque"', 'N_("Deckkraft")', 'FALSE', '0.0', '1.0', '2.0', u'N_("0 means brush is transparent, 1 fully visible\\n(also known as alpha or opacity)")')
(u'"opaque_multiply"', u'N_("Opacity multiply")', 'FALSE', '0.0', '0.0', '2.0', u'N_("This gets multiplied with opaque. You should only change the pressure input of this setting. Use \'opaque\' instead to make opacity depend on speed.\\nThis setting is responsible to stop painting when there is zero pressure. This is just a convention, the behaviour is identical to \'opaque\'.")')
...
(u'"hardness"', 'N_("H\xc3\xa4rte")', 'FALSE', '0.0', '0.8', '1.0', u'N_("Hard brush-circle borders (setting to zero will draw nothing). To reach the maximum hardness, you need to disable Pixel feather.")')

[kaerhon]

similar issue here too

log : scons: Reading SConscript files ... building for 'python2.7' (use scons python_binary=xxx to change) using 'python2.7-config' (use scons python_config=xxx to change) Delete(["libmypaint-tests.so", "libmypaint-tests.so", "libmypaint.so", "libmypaintlib.so", "libmypaint.a", "libmypaint-tests.a", "lib/_mypaintlib.so"]) Enabling i18n for brushlib in full application build set umask to 0022 (was 0002) scons: done reading SConscript files. scons: Building targets ... Creating 'brushlib/mypaint-config.h' gcc -o brushlib/brushmodes.os -c -std=c99 -D_POSIX_C_SOURCE=200809L -fopenmp -O3 -Wall -g -fPIC -DHAVE_JSON_C -DHAVE_GETTEXT -Ibrushlib -I/usr/include/json-c brushlib/brushmodes.c gcc -o brushlib/fifo.os -c -std=c99 -D_POSIX_C_SOURCE=200809L -fopenmp -O3 -Wall -g -fPIC -DHAVE_JSON_C -DHAVE_GETTEXT -Ibrushlib -I/usr/include/json-c brushlib/fifo.c gcc -o brushlib/helpers.os -c -std=c99 -D_POSIX_C_SOURCE=200809L -fopenmp -O3 -Wall -g -fPIC -DHAVE_JSON_C -DHAVE_GETTEXT -Ibrushlib -I/usr/include/json-c brushlib/helpers.c gcc -o brushlib/mapping.os -c -std=c99 -D_POSIX_C_SOURCE=200809L -fopenmp -O3 -Wall -g -fPIC -DHAVE_JSON_C -DHAVE_GETTEXT -Ibrushlib -I/usr/include/json-c brushlib/mapping.c generate_cheaders(["brushlib/mypaint-brush-settings-gen.h", "brushlib/brushsettings-gen.h"], ["brushlib/generate.py", "brushlib/brushsettings.py", "brushlib/brushsettings.json"]) python2.7 brushlib/generate.py brushlib/mypaint-brush-settings-gen.h brushlib/brushsettings-gen.h Writing brushlib/mypaint-brush-settings-gen.h Traceback (most recent call last): File "brushlib/generate.py", line 137, in writefile(args[1], generate_internal_settings_code()) File "brushlib/generate.py", line 105, in generate_internal_settings_code [settings_info_struct(i) for i in brushsettings.settings]) File "brushlib/generate.py", line 69, in generate_static_struct_array entries.append(indent + "{%s}" % (", ".join(entry))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) scons: *\ Error 1 gcc -o brushlib/mypaint-brush-settings.os -c -std=c99 -D_POSIX_C_SOURCE=200809L -fopenmp -O3 -Wall -g -fPIC -DHAVE_JSON_C -DHAVE_GETTEXT -Ibrushlib -I/usr/include/json-c brushlib/mypaint-brush-settings.c brushlib/mypaint-brush-settings.c:36:31: fatal error: brushsettings-gen.h: Aucun fichier ou dossier de ce type

include "brushsettings-gen.h"

                           ^

compilation terminated. scons: *\ [brushlib/mypaint-brush-settings.os] Error 1 scons: building terminated because of errors.

and for locale :

loula@loula:~$ locale -v LANG=fr_FR.UTF-8 LANGUAGE=fr_FR LC_CTYPE="fr_FR.UTF-8" LC_NUMERIC="fr_FR.UTF-8" LC_TIME="fr_FR.UTF-8" LC_COLLATE="fr_FR.UTF-8" LC_MONETARY="fr_FR.UTF-8" LC_MESSAGES="fr_FR.UTF-8" LC_PAPER="fr_FR.UTF-8" LC_NAME="fr_FR.UTF-8" LC_ADDRESS="fr_FR.UTF-8" LC_TELEPHONE="fr_FR.UTF-8" LC_MEASUREMENT="fr_FR.UTF-8" LC_IDENTIFICATION="fr_FR.UTF-8" LC_ALL=

loula@loula:~$ locale -a C C.UTF-8 en_AG en_AG.utf8 en_AU.utf8 en_BW.utf8 en_CA.utf8 en_DK.utf8 en_GB.utf8 en_HK.utf8 en_IE.utf8 en_IN en_IN.utf8 en_NG en_NG.utf8 en_NZ.utf8 en_PH.utf8 en_SG.utf8 en_US.utf8 en_ZA.utf8 en_ZM en_ZM.utf8 en_ZW.utf8 fr_BE.utf8 fr_CA.utf8 fr_CH.utf8 fr_FR.utf8 fr_LU.utf8 POSIX

[jy2wong]

Managed to reproduce this with python 2.7.9.

1. scons -c
2. find a non-english locale from locale -a. I picked fr_CA.UTF-8.
3. LANG=fr_CA.UTF-8 scons

I stuck in print entry[1], type(entry[1]) inside the for loop at line 68.

N_("Opacité") <type 'str'>

Oddly enough, when I ran LANG=en_CA.UTF-8, I instead got

N_("Opacity") <type 'unicode'>

[jy2wong]

diff --git a/generate.py b/generate.py
index 8cfb880..ed32082 100644
--- a/generate.py
+++ b/generate.py
@@ -21,6 +21,7 @@ if os.path.exists("brushlib"):
     sys.path.append("brushlib")
 import brushsettings
 from os.path import basename
+import io

 # workaround for https://gna.org/bugs/index.php?20281
@@ -38,7 +39,7 @@ def writefile(filename, s):
         print 'Checked', filename
     else:
         print 'Writing', filename
-        open(filename, 'w').write(s)
+        io.open(filename, 'w', encoding='utf-8').write(s)

 def generate_enum(enum_name, enum_prefix, count_name, name_list, value_list):
@@ -87,7 +88,7 @@ def generate_internal_settings_code():
         return str(value)

     def gettextify(value):
-        return "N_(%s)" % stringify(value)
+        return "N_(%s)" % stringify(value.decode('utf-8'))

     def boolify(value):
         return str("TRUE") if value else str("FALSE")

works for me.

[achadwick]

Huh it runs with the patch but I'm getting N_("Opacité") in the output, which would be a double-translation.

// [...]
static MyPaintBrushSettingInfo settings_info_array[] = {
    {"opaque", N_("Opacité"), FALSE, 0.0, 1.0, 2.0, N_("0 means brush is transparent, 1 fully visible\n(also known as alpha or opacity)")}, 
    {"opaque_multiply", N_(" Multiplier par l'opacité"), FALSE, 0.0, 0.0, 2.0, N_("This gets multiplied with opaque. You should only change the pressure input of this setting. Use 'opaque' instead to make opacity depend on speed.\nThis setting is responsible to stop painting when there is zero pressure. This is just a convention, the behaviour is identical to 'opaque'.")}, 
// [...]

It's inconsistent with the other strings in the struct initializer.

There's a reference to https://gna.org/bugs/index.php?20281 in the code, so maybe this was fixed in the past. Bisecting to see if it got reintroduced recently.

[achadwick]

Looks like it got accidentally reintroduced by an import reordering in 618edddffb639fc97f2fc04eed3746b5435fd39d. Fixing...

[achadwick]

@jy2wong @kaerhon @black-silence Can you confirm that this is fixed on your systems with the locales you were using before?

[kaerhon]

yes, it works again! thanks!