encoding hell

[ischurov]

It seems that doconce needs some refactoring to get out from the encoding hell we have now. Current status (as far as I can see after using doconce for a week) is as follows:

• Most of the basic features of doconce format work with utf-8 sources if --encoding='utf-8' option is present.
  • This is done by manual prepending codecs.open fallback to any open invoсation if encoding variable is set. Sometimes the user will get a proper error message, other times they will get just a backtrace with 'ascii' codec can't decode… message.
• A little bit less known features (e.g. quizzes or doconce helper commands) do not work with utf-8 (and presumably any other encodings) and should be fixed.
  • Some of them were fixed recently with the same manual strategy, see e.g. 2f38b4cee936a233adc0313d3bb89785a7e6d1b0.
• I didn't check any other encodings except 'utf-8' but I believe that they will go in the same way.

I believe that we need some general approach to this stuff to avoid copy-pasting of the code. The following issues should be addressed:

1. wrapper function to open that respects encoding parameter transparently should exists.
2. there should be one handler of 'ascii' codec can't decode… exception that suggests adding '--encoding=utf-8' option at the command line.

The other ways we can consider are:

• drop --encoding option at all and assume that the source is always utf-8. I'm not sure, are there huge use of non-utf-8 sources in the project? (Note that ASCII is also valid UTF-8.) Then we'll get rid of all that if encoding stuff and solve at least issue 1 above.
• switch to python3 finally which handles unicode in a better way (anyway, I believe, we have to switch to 'utf-8-only' in this case).

I'd like to volunteer on fixing this (as time permits) as we choose a strategy.

[hplgit]

I totally agree with the description - doconce emerged from pure ascii (pure English texts) and utf-8 support has been just a series of ugly hacks.

I would vote for opening all files as utf-8 as I think that is the best long-term solution. There is significant interest in using doconce for non-English texts, so we should also think about a dictionary for generated expressions such as Table of Contents, Figure, Movie, etc.

We have to keep --encoding=... for backward compatibility, but we can issue a warning that it is no longer necessary.

Regarding Python 3, a common futurized code is on the todo list, but a previous attempt (see issue #38) failed due to future's handling of strings for Python 2.

Any attempt from ischurov to fix the encoding hell is much appreciated!