Spanish accents

[dsejas]

Description: When using SageTeX with Spanish accents, there is an error message:

Traceback (most recent call last): File "test.sagetex.sage.py", line 21, in st.endofdoc() File "/SageMath/local/lib/python2.7/site-packages/sagetex.py", line 276, in endofdoc m.update(bytearray(line,'utf8')) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 15: ordinal not in range(128)

Version: SageTeX v3.2, dated 2019/01/09; SageMath v8.7; TeXLive 2018. How to reproduce: My MWE with the Spanish word 'opinión' follows; if this word is replaced with a non-accented version ('opinion'), the problem disappears. `\documentclass{article}

\usepackage{amsmath} \usepackage[utf8]{inputenc} \usepackage{sagetex}

\begin{document} \begin{sagesilent} ans = 'opinión' \end{sagesilent} $\sage{ans}$ \end{document}`

Other info: I have reduced the problem to commit b98bd6d, specifically the change of line 1535 of file "py-and-sty.dtx" from m.update(line) to the new m.update(bytearray(line,'utf8')). If the old line is restored, the problem disappears. Conjecture: This seems to be a double encoding problem. My LaTeX document is already using UTF-8 encoding, so the line m.update(bytearray(line,'utf8')) is trying to double encode. Just a theory... Related reports: https://trac.sagemath.org/ticket/27598

[jhpalmieri]

I think this change will fix it:

diff --git a/py-and-sty.dtx b/py-and-sty.dtx
index a27dbc5..49bf5cd 100644
--- a/py-and-sty.dtx
+++ b/py-and-sty.dtx
@@ -859,6 +859,7 @@ again}}
 pyversion = ' '.join(__version__.strip('[').split()[0:2])
 from sage.misc.latex import latex
 from sage.repl.preparse import preparse
+from six import PY3
 import sys
 import os
 import os.path
@@ -1532,7 +1533,10 @@ SAGE_ROOT/local/share/doc/sagetex.""")
                               "print('SageT",
                               "_st_.current_tex_line",
                               " _st_.current_tex_line")):
-          m.update(bytearray(line,'utf8'))
+          if PY3:
+              m.update(bytearray(line,'utf8'))
+          else:
+              m.update(bytearray(line))
 %    \end{macrocode}
 % (The |current_tex_line| thing appears twice because it may appear
 % indented one space or not, depending on whether it's used before

Not tested, though.

[jhpalmieri]

One reason this is untested: sagetex's Makefile doesn't work on OS X because of the non-posix use of sed:

sed -e 's/usage|hyperpage/usagehyperpage/g' -i sagetex.idx
sed: -i may not be used with stdin
make: *** [sagetex.ind] Error 1

I'm trying to work around this.

[dsejas]

Well, it works perfectly, @jhpalmieri , except for a small point. When I try to run the generated .sagetex.sage file, I get the following error message:

NameError: global name 'PY3' is not defined

Let me "reprise" your suggestion: if PY3: m.update(bytearray(line,'utf8')) else: m.update(bytearray(line))

May I suggest to change it for: if sys.version_info[0] == 2: m.update(bytearray(line)) else: m.update(bytearray(line,'utf8'))

Another option would be (although I am not completely sure of this): if sys.hexversion < 0x03000000: m.update(bytearray(line)) else: m.update(bytearray(line,'utf8'))

Both solutions work fine for me. Maybe somebody can make the corresponding modifications of the code. Sorry I don't make them myself, but Git is still one of the mysteries of life for me, and this is my first day on Github.

Oh, by the way, thank you very much, @jhpalmieri !

[jhpalmieri]

Is the line from six import PY3 included at the beginning of the file sagetex.py?

[dsejas]

Is the line from six import PY3 included at the beginning of the file sagetex.py?

No, not by default. I didn't notice that. In order to be included in the sagetex.py file, it should be added between lines 860 and 871 of py-and-sty.dtx

[dsejas]

I can confirm that it works now. I added from six import PY3 as line number 860 in py-and-sty.dtx. Works beautifully. Thank you again, @jhpalmieri ! Should I close the issue? Or should I left it for revision?

[jhpalmieri]

I'm glad it works, but as I said in the pull request (#31), I am not sure if I'm dealing with unicode issues correctly, especially so that it works with both Python 2 and 3. So let's leave this open until more people can look at it.

[kcrisman]

As a workaround you could do \'el, I think, which I need to use for things like Erd\H{o}s in my non-UTF-8 documents (guess I need to get in the millennium!).

[dimpase]

fixed in #31 and subsequent commit